Create a supported "high level" programmatic API for pip

[pfmoore]

This has come up a few times recently now that pip is available in a standard Python install.

Maybe we should formally support a programmatic API for pip that allows the high-level command line operations pip supports to be run from within a Python interpreter? That may simply mean blessing (and documenting) pip.main() as a supported API.

There may be some odd corner cases to take care with, if we're considering people running pip.main() from within a persistent interpreter (e.g. IPython or Idle). I'm thinking of cases where sys.modules caches something you're upgrading, for example. But that may just need documentation saying that those things need an interpreter restart, at least as a starting point.

[Ivoz]

I'd be hesistating to suggest an api for anything that needs an interpreter restart, because therefore it seems to me hardly more useful to run inline code but have to exit and restart your interpreter anyway.

If we were going to do something anyway, (perhaps, even, start off with operations that wouldn't warrant any interpreter restart, like package information querying), I'd suggest putting them under a fascade under a pip.api module or similar.

Also you'd want to be careful about not unconditionally relying on setuptools functionality for anything (without huge reason) and even when so, possibly not duplicating what api setupools already provides.

[RonnyPfannschmidt]

If the only blessed api is pip.main, then there is no good reason to do it in-process some it won't have a advantage over just calling pip as subprocess

[dstufft]

I'm not sure if pip should have an API or not, but certainly parts of what you'd use an API for will be shuffled over to packaging as we get new PEPs written.

[pfmoore]

I think there's enough people talking about using pip from within Python (either prompts or scripts) that we should provide something. Even if it's very limited, at least that will get people thinking about what exactly they need, and we'll be able to firm up on details as we go along.

@dstufft for lower-level API's based around the standards PEPs, we should absolutely be pointing people at packaging.

@RonnyPfannschmidt you have a good point. It's not at all unreasonable to tell people that to call pip they should be using

subprocess.check_call([sys.executable, '-m', 'pip', ...])

If that's not sufficient for them (for example, the format of stdout isn't useful for parsing) we can discuss options (for example, adding a --json argument to informational commands).

On the other hand, I think the big issue is that people expect to be able to do

pip.main(['install', 'foo'])
import foo

from within (say) an Idle session. This is actually no different in principle from running pip in a separate command window and not restarting your Idle session (and indeed it's exactly the same if you use subprocess) but maybe just documenting that is sufficient - after all, it's not really a pip issue that you have to take care if you change the contents of sys.path while your Python process is running.

I think I'll write up a new section for the pip documentation on the basis of subprocess being pip's supported API. That can be a starting point for people wanting to integrate pip into IDEs, and we can then address any issues as people find them.

[Rosuav]

I was just tinkering with this kind of thing, and the API that I reached for was:

import pip
pip.install("psycopg2")

If there's a dedicated entrypoint like that, it could even choose whether to use subprocess or to do the work itself. Using pip.main(["install","psycopg2"]) does work (on Python 3.4.3, at least - on 3.5.1, there's no psycopg2 wheel, so my poor Windows VM got confuzzled and gave up), but given the concerns about subprocesses, I'm not sure that's sufficient.

Is there somewhere a summary of why it's important to call pip via subprocess rather than directly?

[dstufft]

@Rosuav The short version is that Python, setuptools, and pip all have various, process level caches for different pieces of information that is affected by installing, upgrading, and uninstalling projects. Forcing pip to be called with a fresh process each time ensures that pip always sees a consistent view of the world instead of a possibly partially cached view of the world where it might get confused about the state of the system. It is likely possible to go through and make sure that these caches get invalidated or otherwise removed so that pip will always see a consistent state and can live longer than 1 process per invocation.

The other issue is that pip sort of purposely has no real public API that can be easily called from Python code. That is again because of the various caches that make it easy for things to go wrong. For example:

import foo.bar  # Assume foo 1.0 is installed, and Python will load and cache "foo" and "foo.bar" from 1.0 into sys.modules
import pip
pip.install("foo", upgrade=True)  # Pretend foo is now upgraded on disk to foo 2.0
import foo.other  # Python will return "foo" 1.0 from the sys.modules cache, and then will load and import "foo.other" from foo 2.0 and cache that in sys.modules

You now have a process that is in what I think most people would call a broken state. You have different modules from different parts of the system in the same process coming from different versions of foo. This situation is of course completely possible today, you just replace the pip.install with a call to subprocess, but one worry is that by making the programatic API easier, we're paving a cowpath that will incentivize people to do something like that which appears easy to do on the surface but which is very fragile.

[pfmoore]

@Rosuav To simplify even further, you should use subprocess because there's no documented or supported in-process pip API. (For why we don't provide a supported API, see @dstufft's answer).

And note that even calling pip via subprocess has risks that may bite you unless you're sure you know what you're doing (again as @dstufft explained)

[dstufft]

To be more explicit, I meant to say (emphases on addition):

It is likely possible to go through and make sure that these caches get invalidated or otherwise removed so that pip will always see a consistent state and can live longer than 1 process per invocation, _however that has not been a priority of anyone working on pip, largely due to the fragility of trying to modify the installed packages at runtime_.

[Rosuav]

The problem you describe would be the same regardless of how pip is invoked, though, wouldn't it? Whether you call pip.main as it currently is, or fire up a completely independent process in a separate terminal, upgrading and then importing can create exactly this problem.

But if pip has its own internal caching, then sure, let it fire off a subprocess. Even on Windows, the cost of starting a process won't be as much as the cost of downloading new software. It'd still be nice to be able to say pip.install(pkg, pkg, pkg) instead of subprocess.call([sys.executable, '-m', 'pip', 'install', pkg, pkg. pkg]) - which could be exactly how the install entrypoint is then implemented.

[Rosuav]

@dstufft Fair enough re changing installed packages. TBH the only use-case I'm really looking at here is:

>>> import foo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'foo'
>>> pip.install("foo")
# ... chug chug chug ...
>>> import foo
>>>

That would be the least risky operation. If anything beyond that is declared to be unsupported, fine; and if it's implemented by spawning a subprocess, no big deal. It would still be extremely convenient.

[dstufft]

The problem you describe would be the same regardless of how pip is invoked, though, wouldn't it?

Yes, but by making an official top level API we signal to people that this is a supported thing they can do programmatically from within their own code. I'm of the opinion (and other pip developers may feel differently) that we should not go out of our way to provide APIs that are hard to use in a way that it is actually "safe" (e.g. not broken) to use. Especially when safe usage requires knowing if/how every single module you're importing uses that same API.

An easy thing to do would be for someone (doesn't even have to be a pip developer!) to provide a pip-api package on PyPI that did basically this. It'd be slightly worse since you wouldn't be able to do import pip and you'd need to instead do import pip_api as pip or something. It'd allow experimentation to see if A) many people would use it at all and B) when people do use it, are they able to grok how to use it safely or does it end up being a footgun. If it shows that I am wrong and my fears are unjustified, then it becomes much easier to justify adding it.

As always, I'm just a single pip developer and the other developers may feel differently and if they wanted to do this anyways, I wouldn't block them.

[pfmoore]

You could of course write a helper function for that yourself.

# Warning, untested code ahead!
def conditional_install(name, project_name=None):
    try:
        mod = importlib.import_module(name)
    except ImportError:
        subprocess.call([sys.executable, '-m', 'pip', 'install', project_name or name])
        mod = importlib.import_module(name)
    return name

# and call it like this
foo = conditional_install('foo')

Whoops, just saw @dstufft's response, I basically agree with him we don't want to deal with people getting into trouble using such an API, so supporting it within pip isn't something I'm keen on (in spite of me having raised this issue - I've since been persuaded it isn't a good idea). But a 3rd party module for this would be OK with me, it'd help us get a feel for what the support costs are like ;-)

[Rosuav]

More and more I'm thinking that I need to provide all students with a little utilities module that loads stuff up on startup :) It's all very well to slap a personal function into a personal installation, but when I'm trying to walk someone through something, s/he won't have my toolbox handy. (Anyway, if I want to pip-install something, I'll just hit Ctrl-Alt-T to open up a new terminal, and run pip from there. Job done.)

[edmorley]

Can't the students do something like...

git clone <tutorial_project>
pip install -r requirements.txt
python
<start tutorial from your slides/whiteboard/...>