Depending on Bootstrapping the Future

Most of the code changes made for today are code cleanup and refactorings to get things tidy, which are not really work talking about in the general sense. I have to imagine that “I took some code that I used in two places and made it a function” is even less exciting than the things I would normally discuss in this devlog (if such is even possible).

So instead I’m going to take this opportunity to talk about some future changes that are going to happen in hyperhelp which may not get done during this Devember phase of the project.

Packages in Sublime are collections of resources very reminiscent of Python modules (for good reason). If you’re familiar with Python at all, the Packages folder is automatically a part of the embedded interpreters sys.path setting. When Sublime loads packages it will load all normal resources inside regardless of their location in the package, but it will only load python source files from the top level automatically.

The reason for that is so that package authors have more control over how their packages are loaded by Sublime. A common use case for this would be having multiple versions of a submodule that are OS specific and only loading the one that’s appropriate for the current platform, for example.

One of the most important packages for Sublime is Package Control, which makes it incredibly easy for end users to install packages to customize Sublime Text to the way that they want to work. I highly suspect that the large majority of people creating extensions for Sublime work towards getting their packages added to Package Control (hereafter referred to as PC) to make them easy to install.

PC maintains a repository of all known packages so that they can be easily discovered, downloaded and installed. Along with the list of packages are a list of dependencies. These are essentially just regular Python packages that are specially bundled and set up to work within the Sublime Text environment.

PC allows package authors to specify a list of dependencies for their packages, which it will make sure are installed whenever a package that requires them is installed. This removes a huge headache for end users, who can easily use packages that require external packages without having to manually take steps to install them.

Dependencies are really just a special case of a Sublime package. They have a directory structure that keeps all of the Python code out of the top level, and instead have special folders that tell PC what version of the code to install for the current operating system and version of Sublime (if it matters). Since dependencies are packages and packages are on sys.path, this means that regular Sublime packages can easily import modules from dependencies to use their code.

Except that it’s more tricky than that, because of that whole “different subfolders for different architectures and versions of Sublime” part. Imagine if as a package author every import of a dependency required an if-else chain to determine which thing to import based on the current operating system?

PC handles this by installing it’s own sub-package called 0_package_control_loader. The name is meant to ensure that it’s the first user-space package loaded because Sublime loads things in lexical order. Each file in the package adds the appropriate sub-folder of a dependency to the sys.path based on all of the required criteria, so that the ultimate package can just import and get what it wants.

This tends to cause a large number of paths to be crammed into sys.path, and has other issues since dependencies might have relative load orders if they depend on each other, which requires everyone to cooperate when publishing their dependencies to keep everything working as expected.

As of the official version 3.0 of Sublime Text, there is now a dedicated lib folder that is also automatically added to sys.path at startup. It’s generally empty because nothing is taking advantage of it yet, but the idea is that PC is going to alter how it works with dependencies to install them into that folder instead of as regular packages.

The benefits of this include a more clean package directory and structure, a much simplified sys.path and the resolution to potential ordering problems with dependencies.

What does all of this babble have to do with hyperhelp? The idea is for it to be a dependency package so that any package can depend on it in order to add help instead of requiring every package to try and bundle a version themselves. This is possible (and the original version of it works that way), but there are also issues that goes along with it.

For example, along with library code for help it also includes resources like the help syntax, menus, key bindings and so on. The way things currently work, Sublime sees those resources even if they’re not in the top level of the dependency. When PC is altered to install dependencies outside of the Packages folder, that will no longer work.

Another is that since it’s a package that transparently works it’s currently set up to assume that it’s a regular package, which means that it has to include it’s own bootstrapping code in order to be loaded. This is somewhat of a hack currently because the custom PC loader is not meant to try and introduce those sorts of symbols to the global environment.

In order to solve both problems, hyperhelp has to support being installed outside of the Packages folder, which means that it also requires external packages that use it to make some sort of bootstrap call in order to set it up. The general idea is to design a single bootstrap call that not only sets things up and imports the appropriate Sublime commands, but which would also check and see if the bundled resources are available and install them if not.

I have some ideas on how to do that, but I’m not sure if it’s going to happen during Devember with everything else that’s going on. If it doesn’t happen in the new few days, you can look for that in the regular hyperhelp repository some time in the new year.