Ship pip as a standalone application

[pfmoore]

Actually, it occurred to me that we may even be able to do this right now. I put together a very simple proof of concept and it seems to work. If you put the following script alongside a "lib" directory with pip installed into it (pip install pip --target lib) but with the bin and pip*.dist-info directory removed (so the bundled pip isn't visible in pip list) then it can be run from any Python interpreter to effectively act as a copy of pip in that environment.

#!/usr/bin/env python

import runpy
import sys
import os

lib = os.path.join(os.path.dirname(__file__), "lib")
sys.path.insert(0, lib)

runpy.run_module("pip", run_name="__main__")

I don't think it would take much to turn this into a viable "standalone pip" application (I'd mostly just want to set up an executable wrapper for Windows). I've done some very basic testing - this would need a lot more real-world testing to make sure there aren't any problem edge cases, but it basically seems to work.

Originally posted by @pfmoore in #11223 (comment)

For now, this is just a placeholder to discuss whether we want to do this at all, or how we'd distribute it. The main point here is that with a script like this, there would no longer be a need to install pip in every virtual environment.

One thing we'd have to work out is what tools assume that pip is available in every environment. I'm thinking of environment managers and IDEs, like nox, or VS Code. The ecosystem implications here are likely to be more complicated than the technical issues. Maybe we need to start with a heads-up discussion on Discourse? But before we do that I'd like to make sure the pip committers are all on board with the idea...

[sbidoul]

Very interesting approach.

[sbidoul]

There will be teaching implications too (undoing years of python -m pip).

[sbidoul]

The script will need to check python version compatibility.

[pfmoore]

There will be teaching implications too

At least initially, this can be an alternative, rather than a replacement. But absolutely, this is a significant change in approach. Which is why I think it needs to be flagged in advance. I'd post a topic on the Packaging discourse right now, but I'm frankly scared of the controversy it'll probably cause 😨

[pradyunsg]

I mean, we can also ship pip as a zipapp. IIUC, that should still not be visible on pip list and, it's literally a python pip.pyz ... which would be equivalent to python -m pip.

That's easy to communicate as well. :)

[pradyunsg]

More broadly though, I'm on board. :)

[pfmoore]

I mean, we can also ship pip as a zipapp.

This is true. I'm not sure we can simply zip up the pip directory and call it a zipapp, but we can certainly ship a zipapp containing the script I posted above plus a copy of pip.

Do we know if all of our dependencies work when shipped as a zipapp (I believe requests didn't like the certificate file being in a zip at one stage, but IIRC that's fixed now)? Also, does the mechanism we use for injecting pip into a build environment work from a zipapp?

Shiv gets round this by creating zipapps that extract themselves on first use. I don't know if we want to go that far.

Otherwise, the main things that annoy me about zipapps are (1) python pip.pyz doesn't search PATH, and (2) .pyz files aren't registered on Windows to run from the command line by default (they need to be added to PATHEXT), and even when they are we have the old problem that nothing but an exe file is a "first class citizen" 🙁

As an initial step in this direction, though, we could ship a .pyz - virtualenv does it, and I'm pretty sure a couple of other tools do as well, so it's not an unfamiliar model to people. We could then promote the idea as "if you don't want to install pip in all of your environments, you can use the zipapp version (and use --no-pip when creating virtualenvs").

That's something I'd be comfortable announcing as a plan on Discourse...

[uranusjr]

I think the main hurdle toward shipping a standalone application (versus a zipapp) is source build. If someone needs to build something from source, it's likely they'll want to build against an existing Python installation, instead of the interpreter bundled in the standalone executable, and that'll need some additional mechanism.

Wheel-only installations should be more or less plausible. The only reason thing we need to deal with (that I can think of) is console script shebangs.

[pfmoore]

The key here is that the standalone executable doesn't bundle an interpreter¹. That's basically what the /usr/bin/env python shebang achieves. It runs the included pip in the environment's own Python.

Footnotes

1. Or if it does, it executes pip with the installation interpreter, not the bundled one. But that's harder (not impossible, but a bit more fiddly). ↩

[sbidoul]

How to upgrade pip is going to be a topic. pip install --upgrade pip is not going to do what people expect.

If we want to be fancy, the script could have a mechanism to download the latest pip for the corresponding python version.

[pfmoore]

Initially, I'd prefer to just publish a zipapp at https://bootstrap.pypa.io, like virtualenv does. Users can download that to get the latest version. Maybe we could also also publish it as a github release for people who want a specific version. I'd leave installers and upgraders to the community to provide, if they want (on Windows, for example, scoop and chocolatey can handle this, and on Linux distro packagers fulfil that role, I guess).

Agreed that pip install --upgrade pip will be confusing, but I'm not sure there's much we can do about that, apart from have a gradual transition. Maybe we could add a warning to pip so that if it detects that it's not running from the location that it will upgrade, we let the user know? That might be useful in any case, not just for this situation.

[pradyunsg]

Noting this here, so that we don't forget -- we'd want to update the upgrade prompt, to be aware of the zipapp based workflow and behave differently. What that different behaviour should be is something I don't have an opinion on, and I don't intend to think about that until we get somewhere in the discussion. :)