Skip to content

Sub-dependencies are not upgraded with the only-if-needed strategy #5274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jbeezley opened this issue Apr 17, 2018 · 3 comments
Closed

Sub-dependencies are not upgraded with the only-if-needed strategy #5274

jbeezley opened this issue Apr 17, 2018 · 3 comments
Labels
auto-locked Outdated issues that have been locked by automation type: support User Support

Comments

@jbeezley
Copy link

  • Pip version: 10.0.0
  • Python version: 2.7.12
  • Operating system: Ubuntu 16.04

Description:

pip install -U with the default only-if-needed upgrade strategy does not upgrade sub-dependencies required by the installed package.

What I've run:

In a new virtualenv, I install an older version of boto3 and then try to install moto. After this completes, the direct dependency (boto3) is updated, but the sub-dependency botocore (a direct dependency of boto3) is not.

$ pip install boto3==1.6.8

Collecting boto3==1.6.8
# ... unrelated lines cropped for brevity ...
Installing collected packages: jmespath, six, python-dateutil, docutils, botocore, futures, s3transfer, boto3
Successfully installed boto3-1.6.8 botocore-1.9.23 docutils-0.14 futures-3.2.0 jmespath-0.9.3 python-dateutil-2.6.1 s3transfer-0.1.13 six-1.11.0

$ pip install -U moto
# ... unrelated lines cropped for brevity ...
 ./env/lib/python2.7/site-packages (from moto) (2.6.1)
Requirement not upgraded as not directly required: botocore>=1.9.16 in ./env/lib/python2.7/site-packages (from moto) (1.9.23)
Requirement not upgraded as not directly required: six>1.9 in ./env/lib/python2.7/site-packages (from moto) (1.11.0)
Requirement not upgraded as not directly required: s3transfer<0.2.0,>=0.1.10 in ./env/lib/python2.7/site-packages (from boto3>=1.6.16->moto) (0.1.13)
Requirement not upgraded as not directly required: jmespath<1.0.0,>=0.7.1 in ./env/lib/python2.7/site-packages (from boto3>=1.6.16->moto) (0.9.3)
Requirement not upgraded as not directly required: docutils>=0.10 in ./env/lib/python2.7/site-packages (from botocore>=1.9.16->moto) (0.14)
Requirement not upgraded as not directly required: futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" in ./env/lib/python2.7/site-packages (from s3transfer<0.2.0,>=0.1.10->boto3>=1.6.16->moto) (3.2.0)
boto3 1.7.4 has requirement botocore<1.11.0,>=1.10.4, but you'll have botocore 1.9.23 which is incompatible.
Installing collected packages: jsondiff, werkzeug, boto, PyYAML, pyaml, MarkupSafe, Jinja2, xmltodict, pbr, funcsigs, mock, certifi, chardet, idna, urllib3, requests, cookies, responses, boto3, backports.weakref, backports.tempfile, backports.ssl-match-hostname, ipaddress, docker-pycreds, websocket-client, docker, pytz, pycparser, cffi, enum34, asn1crypto, cryptography, wrapt, jsonpickle, aws-xray-sdk, moto
  Found existing installation: boto3 1.6.8
    Uninstalling boto3-1.6.8:
      Successfully uninstalled boto3-1.6.8
Successfully installed Jinja2-2.10 MarkupSafe-1.0 PyYAML-3.12 asn1crypto-0.24.0 aws-xray-sdk-0.95 backports.ssl-match-hostname-3.5.0.1 backports.tempfile-1.0 backports.weakref-1.0.post1 boto-2.48.0 boto3-1.7.4 certifi-2018.4.16 cffi-1.11.5 chardet-3.0.4 cookies-2.2.1 cryptography-2.2.2 docker-3.2.1 docker-pycreds-0.2.2 enum34-1.1.6 funcsigs-1.0.2 idna-2.6 ipaddress-1.0.22 jsondiff-1.1.1 jsonpickle-0.9.6 mock-2.0.0 moto-1.3.2 pbr-4.0.2 pyaml-17.12.1 pycparser-2.18 pytz-2018.4 requests-2.18.4 responses-0.9.0 urllib3-1.22 websocket-client-0.47.0 werkzeug-0.14.1 wrapt-1.10.11 xmltodict-0.11.0

This is a contrived example, but it came up when installing our package into a cached virtual environment inside our CI testing.

Behavioral question

I understand the new default only-if-needed upgrade strategy is causing the issue. (Running with the eager strategy updates botocore as expected.) Is it the intention that the new default behavior of pip does not upgrade sub-dependencies even when required by something being installed?

I will note this is similar to #5137 except that here I am using the -U flag. In light of #988, I can understand why pip tries to make minimal changes without the upgrade flag. However, the new default behavior if this is not a bug seems non-intuitive. I'm having a hard time understanding a situation where the eager strategy wouldn't be the most appropriate default.
@pradyunsg
Copy link
Member

Is it the intention that the new default behavior of pip does not upgrade sub-dependencies even when required by something being installed?

It's intentional.

In light of #988, I can understand why pip tries to make minimal changes without the upgrade flag.

That's exactly what it is.

There's a lot of discussion for this, ending in #3871. Feel free to close this issue or ask more questions here. :)

@pradyunsg pradyunsg added the type: support User Support label Apr 17, 2018
@jbeezley
Copy link
Author

In light of #3871 (comment), it sounds like I'm in this camp:

This will get us a lot of real world use by forcing a change on everyone, but gives an escape hatch for people who, for whatever reason are broken by the change.

I'm happy to close this as intended behavior; however, I would argue against removing the eager strategy until #988 is resolved. Thanks for your response!

jbeezley added a commit to girder/girder that referenced this issue Apr 17, 2018
@jbeezley
Add --upgrade-strategy eager to all uses of pip install -U
c2931fc 
The default behavior of `pip install -U` has changed as of version 10.
The default only-if-needed strategy only upgrades direct dependencies of
the packages being installed.  This is likely to break automated
upgrades such as ansible and CI deployments that use a previously
generated virtual environment.

Both upgrade strategies can lead to invalid virtual environment states.
One could argue that the new behavior is superior because it makes
minimal changes to the environment; however, my guess is that the eager
strategy will cause less confusion and is identical to what pip had done
prior to version 10.

See also:

pypa/pip#5274
jbeezley added a commit to girder/girder that referenced this issue Apr 17, 2018
@jbeezley
Add --upgrade-strategy eager to all uses of pip install -U
939027a 
The default behavior of `pip install -U` has changed as of version 10.
The default only-if-needed strategy only upgrades direct dependencies of
the packages being installed.  This is likely to break automated
upgrades such as ansible and CI deployments that use a previously
generated virtual environment.

Both upgrade strategies can lead to invalid virtual environment states
because resolving version conflicts is not something pip claims to
solve.  One could argue that the new behavior is superior because it
makes minimal changes to the environment; however, my guess is that the
eager strategy will cause less confusion and is identical to what pip
had done prior to version 10.

See also:

pypa/pip#5274
@jbeezley jbeezley mentioned this issue Apr 17, 2018
Merged
paradoxxxzero added a commit to Kozea/unrest that referenced this issue Jun 18, 2018
@paradoxxxzero
pypa/pip#5274
f5f1b49
@lock
Copy link

lock bot commented Jun 2, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 2, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Jun 2, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
auto-locked Outdated issues that have been locked by automation type: support User Support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jbeezley @pradyunsg