Document the way pip chooses what version to install more explicitly

[pfmoore]

Initial attempt, for review.

This will be refined in parallele with writing the corresponding code for the new resolver. I wanted to get a first version up to act as a starting spec for the behaviour, though.

[uranusjr]

This sentence feels so confusing. I can instinctively understand the implication, but feel it is not explaining it right, but can’t point my finger to where nor suggestion something better 😞

[pfmoore]

Agreed, it's pretty bad. I need to rethink this (at least part of the problem is the fact that --upgrade doesn't do what I (and I suspect most people) think it should.

I'll have another go at the text here.

[McSinyx]

TBH as an user after reading this I have no idea what --upgrade is supposed to do anymore. With the legacy resolver, upgrading everything as a whole was not possible and I think the flag was to enable users to resolve dependencies by themselves and decide which package to upgrade (i.e. pip install -U this is equivalent to upgrade this in other package managers). With the new resolver, it might be less mind-boggling for users if we have pip upgrade, and just let the resolver tries its best during installation.

Since --upgrade doesn't really preserve backward-compatibility (it would if --upgrade-strategy=eager is provided IIUC), IMHO if when the new resolver becomes default and upgrade subcommand is yet to be ready, we may want to flag it as deprecation (e.g. *--upgrade may not do what you want, since the dep-resolve process influences the choice of which package to be upgraded/downgraded). If one wants some sort of a new version, perse can always specify it explicitly, which also complies with the Zen of Python.

[uranusjr]

pip install -U this is equivalent to upgrade this in other package managers

I think this is the correct mental model both with the legacy and new resolvers. But the problem is in the details. What does upgrade this actually mean? Does it only upgeade this? Does it upgrade this and all its dependencies? What do “dependencies” even mean? This fries the brain if you really try to think it through.

The operation “upgrade” actually works in a different abstraction level from package installation, and most package managers implement it as such: in a different abstraction level with the manifest/lock mechanism. Having an upgrade operation in the abstraction level pip operates in is quirky by itself, so the definition has to be quirky as a result.

[pfmoore]

TBH as an user after reading this I have no idea what --upgrade is supposed to do anymore.

TBH after reading the code I lost all sense of what --upgrade does, so I'm glad that I captured my understanding so accurately 🙂

More seriously, yes, once you start digging pip's upgrade operation is a very weird beast and it's not at all what youthink it is at first glance. One thing that became obvious to me, for example, while writing tests for the new upgrade strategy code, is that upgrade doesn't even have any meaning unless you factor in the concept of "a new version has been released" and that is an idea that's never really relevant anywhere else in pip.

My feeling is that there's a fairly significant restructuring that needs to be done here before it'll be possible to describe things really accurately. What I'm not sure about is whether we'll be able to get a better description before we do that, or whether we'd be better ignoring the user documentation problem for now and moving this description into the internals documentation (where it can afford to stay a bit confusing...).

But thanks for all the thoughtful responses, I'll let the discussion continue for a while and then read it all up at once, rather than keep responding bit-by-bit like this.

[PythonCoderAS]

I would also like to know how would pip act if two different conflicting packages were installed, either directly specified, or as dependents of an package.

For example, if package a is dependent on x<5, but package b is dependent on x>6, how would pip act if I ran pip install a b?

Then, how would pip act if package a is dependent on package m, which is dependent on package x==4, but package a is also dependent on package n, which is dependent on package x==5?

Finally, what if package a is dependent on x<5, but I ran pip install a x>5?

[pfmoore]

I would also like to know how would pip act if two different conflicting packages were installed, either directly specified, or as dependents of an package.

It's a conflict, so it's an error (the legacy resolver is very bad at handling such conflicts, so current behaviour is much more likely to be "get in a mess and install something that's broken", but this PR doesn't intend to address the legacy behaviour).

[pfmoore]

Note: I might move this explanation to the "internal architecture" section of the document, as it's likely that it's too confusing to be exposed as user-facing docs.

Fixing the user-facing docs would then be part of the larger exercise of making the user-facing behaviour less confusing 😉

[pradyunsg]

Fixing the user-facing docs would then be part of the larger exercise of making the user-facing behaviour less confusing 😉

Or... something we could apply for https://developers.google.com/season-of-docs/. :)

[brainwane]

@pfmoore Now that your contract is ending, is this something that we should be moving over to Pradyun's plate for fixup revisions?

[pfmoore]

Sounds reasonable. This got bogged down in trying to describe the non-intuitive behaviour of --upgrade, and that's probably the biggest thing that needs addressing before this can be merged.