Skip to content

Document the way pip chooses what version to install more explicitly #8117

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Document the way pip chooses what version to install more explicitly #8117

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 44 additions & 1 deletion docs/html/reference/pip_install.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,50 @@ explicitly state the project name.
Satisfying Requirements
-----------------------

Once pip has the set of requirements to satisfy, it chooses which version of
In order to satisfy the requirements specified on the command line, pip
calculates a set of packages and versions that meets the given requirements. The
details of this process are as follows:

1. Pip considers all packages named in requirements specified on the command
line, and their dependencies, recursively.
2. No other packages are considered when determining what to install. In
particular, installed packages not part of the dependency tree constructed
in step 1 will not be considered.
3. When looking for solutions to the various version constraints, pip will
normally prioritise the installed version of a given package over any other.
However, note that this does not mean that pip will certainly retain that
version - other constraints may influence this.
4. The ``--upgrade`` flag alters this behaviour by *not* prioritising installed
versions for packages named on the command line. This is the default,
"only-if-needed" upgrade strategy. It is also possible to specify
``--upgrade-strategy=eager``, which doesn't prioritise installed versions
for *any* installed package - directly specified or dependency.
5. Apart from the above exceptions around installed versions, pip will
prioritise later versions over earlier versions.

Once pip has satisfied all constraints, it will install the selected versions,
uninstalling existing installed versions if necessary.

It should be noted that in order to satisfy the given requirements, pip will
upgrade (or even downgrade) installed packages as needed, regardless of whether
the ``--upgrade`` flag is present. The ``--upgrade`` flag only influences the
*choice* of what version to install, not whether to allow upgrades.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence feels so confusing. I can instinctively understand the implication, but feel it is not explaining it right, but can’t point my finger to where nor suggestion something better 😞

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, it's pretty bad. I need to rethink this (at least part of the problem is the fact that --upgrade doesn't do what I (and I suspect most people) think it should.

I'll have another go at the text here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH as an user after reading this I have no idea what --upgrade is supposed to do anymore. With the legacy resolver, upgrading everything as a whole was not possible and I think the flag was to enable users to resolve dependencies by themselves and decide which package to upgrade (i.e. pip install -U this is equivalent to upgrade this in other package managers). With the new resolver, it might be less mind-boggling for users if we have pip upgrade, and just let the resolver tries its best during installation.

Since --upgrade doesn't really preserve backward-compatibility (it would if --upgrade-strategy=eager is provided IIUC), IMHO if when the new resolver becomes default and upgrade subcommand is yet to be ready, we may want to flag it as deprecation (e.g. *--upgrade may not do what you want, since the dep-resolve process influences the choice of which package to be upgraded/downgraded). If one wants some sort of a new version, perse can always specify it explicitly, which also complies with the Zen of Python.

Copy link
Member

@uranusjr uranusjr Apr 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pip install -U this is equivalent to upgrade this in other package managers

I think this is the correct mental model both with the legacy and new resolvers. But the problem is in the details. What does upgrade this actually mean? Does it only upgeade this? Does it upgrade this and all its dependencies? What do “dependencies” even mean? This fries the brain if you really try to think it through.

The operation “upgrade” actually works in a different abstraction level from package installation, and most package managers implement it as such: in a different abstraction level with the manifest/lock mechanism. Having an upgrade operation in the abstraction level pip operates in is quirky by itself, so the definition has to be quirky as a result.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH as an user after reading this I have no idea what --upgrade is supposed to do anymore.

TBH after reading the code I lost all sense of what --upgrade does, so I'm glad that I captured my understanding so accurately 🙂

More seriously, yes, once you start digging pip's upgrade operation is a very weird beast and it's not at all what youthink it is at first glance. One thing that became obvious to me, for example, while writing tests for the new upgrade strategy code, is that upgrade doesn't even have any meaning unless you factor in the concept of "a new version has been released" and that is an idea that's never really relevant anywhere else in pip.

My feeling is that there's a fairly significant restructuring that needs to be done here before it'll be possible to describe things really accurately. What I'm not sure about is whether we'll be able to get a better description before we do that, or whether we'd be better ignoring the user documentation problem for now and moving this description into the internals documentation (where it can afford to stay a bit confusing...).

But thanks for all the thoughtful responses, I'll let the discussion continue for a while and then read it all up at once, rather than keep responding bit-by-bit like this.


Other flags that will influence the versions installed are:

* ``--ignore-installed``. This will prevent the resolver from considering
what is already installed as part of the calculation. (It will also mean
that the installation phase will not uninstall an existing version before
installing the selected version, so has the potential to leave the package
in an invalid state - use with care).
* ``--force-reinstall``. This will not affect the calculation, but will mean
that if pip determines that the currently installed version is the one we
want, it will reinstall it.

**TODO**: Clarify the distinction better here!
**TODO**: How to discuss options that affect the *finder* like ``--pre``.

.. Once pip has the set of requirements to satisfy, it chooses which version of
each requirement to install using the simple rule that the latest version that
satisfies the given constraints will be installed (but see :ref:`here <Pre Release Versions>`
for an exception regarding pre-release versions). Where more than one source of
Expand Down