pip install -U upgrades already satisfied dependencies

[ejucovy]

If I pip install -U foo, I would expect that the latest version of foo will be installed, and foo's dependencies will only be reinstalled if they're not already satisfied. But in fact the dependencies all get reinstalled even if I already have identical versions installed:

$ pip install -U django-supervisor
Downloading/unpacking django-supervisor
  Downloading django-supervisor-0.2.0.tar.gz
  Running setup.py egg_info for package django-supervisor
Downloading/unpacking supervisor (from django-supervisor)
  Downloading supervisor-3.0a10.tar.gz (438Kb): 438Kb downloaded
  Running setup.py egg_info for package supervisor
    no previously-included directories found matching 'docs/*.pyc'
    no previously-included directories found matching 'docs/.build'
Downloading/unpacking meld3>=0.6.5 (from supervisor->django-supervisor)
  Downloading meld3-0.6.7.tar.gz
  Running setup.py egg_info for package meld3
Installing collected packages: django-supervisor, supervisor, meld3
  Found existing installation: django-supervisor 0.1.1
    Uninstalling django-supervisor:
      Successfully uninstalled django-supervisor
  Running setup.py install for django-supervisor
  Found existing installation: supervisor 3.0a10
    Uninstalling supervisor:
      Successfully uninstalled supervisor
  Running setup.py install for supervisor
    no previously-included directories found matching 'docs/*.pyc'
    no previously-included directories found matching 'docs/.build'
    Skipping installation of /usr/local/ejucovy/django/lib/python2.6/site-packages/supervisor/__init__.py (namespace package)
    Installing /usr/local/ejucovy/django/lib/python2.6/site-packages/supervisor-3.0a10-py2.6-nspkg.pth
    Installing echo_supervisord_conf script to /usr/local/ejucovy/django/bin
    Installing pidproxy script to /usr/local/ejucovy/django/bin
    Installing supervisorctl script to /usr/local/ejucovy/django/bin
    Installing supervisord script to /usr/local/ejucovy/django/bin
  Found existing installation: meld3 0.6.7
    Uninstalling meld3:
      Successfully uninstalled meld3
  Running setup.py install for meld3
Successfully installed django-supervisor supervisor meld3
Cleaning up...

My "existing installations" of supervisor-3.0a10 and meld3-0.6.7 are both "successfully uninstalled", and then identical versions are installed.

[hltbra]

In my judgement it is a known behavior, don't know if it is really a bug - I guess easy_install has the same behavior.

I would like to see other's opinions.

PS.: There was a question in StackOverflow related to this: http://stackoverflow.com/questions/5937756/why-is-pip-looking-for-download-cache-if-the-same-exact-package-is-already-instal

[hltbra]

Related to issue #49

[carljm]

It is a bug, and it is a duplicate of #49.

[ejucovy]

I don't think it's a duplicate of #49. I read #49 as saying that install -U foo shouldn't reinstall foo itself if it's already at the latest version -- which is different from whether or not it should reinstall foo's already-satisfied dependencies.

This distinction matters for hard-to-build libraries which have frequent releases but fairly stable APIs -- for the most part, one installation is good enough -- I'd only want to reinstall it if my dependencies start to use newer features from those nested dependencies (i.e. if the requirement is no longer satisfied) -- for example:

• foo 0.1 depends on lxml>=2.3.0
• foo 0.2 is released, and depends on lxml>=2.3.0 (same dependency)
• lxml 2.4.0 is released

If I've already installed foo 0.1 and lxml 2.3.0, and I pip install -U foo, I wouldn't want it to install lxml 2.4.0. It should only install lxml 2.4.0 when foo starts to depend on lxml>=2.4.0.

[carljm]

Ah, yes, that is a bit different. Even if #49 is fixed, there would be some additional code needed to achieve the behavior you're wanting when the dependency is not at its latest version, but still satisfies the dependency requirements.

I think if we made this change, some people would find it surprising. I can see why it's desirable in certain cases -- but I can also see cases where the current behavior (minus #49) is preferable. So I'm on the fence on this one.

[ejucovy]

Would an extra command-line option be appropriate? pip install foo --upgrade vs pip install foo --upgrade-recursive? (Or --upgrade-nonrecursive if preserving the current behavior backwards-compatibly is important)

[nisavid]

making upgrades non-recursive by default would provide consistency with other package managers, such as APT and Portage. and i think there is a good reason for such behavior, which is that it avoids unintended side effects--if i want to upgrade a package P, then i would like to enter a command along the lines of upgrade P, not upgrade P --but-not-other-things.

on the other hand, i think that an "upgrade all" command (see #59) should be recursive by default, since "all" should really mean "all" by default. in this case, non-recursive behavior would mean "upgrade all packages installed directly, but not any dependencies that weren't installed directly" (like Portage's emerge --update @world without --deep).

[cls]

A related issue is that if the package being upgraded has a dependency on another package which is already installed but unavailable from a repository, Pip will fail and is unable to upgrade the requested package, despite its dependencies being fulfilled.

[ex-nerd]

This appears to happen with -I as well as -U.

[carljm]

Since -I stands for --ignore-installed, and is intended to make pip operate as if nothing were currently installed, reinstalling everything is the correct behavior for -I. Thus the behavior here is only a bug for -U, not for -I.

[jvanasco]

easy_install does not have the same behavior. easy_install will not re-install dependencies if they are already met.

this isn't a feature or behavior, this is a bug.

this is also really annoying -- testing PIP distribution for a framework's package , or updating a single framework add-on, means needing to re-install the entire framework and all of it's dependencies. these unnecessary downloads and installs are a waste of time and resources.

[qwcode]

for those who just want a way to do this now, short of a behavior/code change to "-U"

I think this achieves the desired result, right?

upgrade top-level requirements only:

• pip install -U --no-deps REQS //upgrades only top-level
• pip install REQS //this 2nd pass will install any new dependencies from the upgrade

[fdintino]

For the simplest case that suffices, but I don't think that works for a requirements.txt file, or if there are dependencies which have updates to their install_requires. We have a complicated script that does a diff of our requirements.txt and more-or-less does what you describe, but it doesn't handle upgrade depth > 1.

What complicates matters to an extent is that many django packages comment out or remove their install_requires lines because of this bug; otherwise they end up with some alpha version of django installed (that has been our experience, and I've seen it in the github issues of many prominent django packages.

[qwcode]

Hey @fdintino , I rigged up a basic test with requirements, -U, and --no-deps and it seemed to work, i.e. the items in the requirements file were upgraded, but no dependencies. that's consistent with my understanding of what the code is doing. it's tedious in there though, so there could be a failing with this idea.

for the case where a top-level requirement has a new "install_requires" dependency, that's why I mentioned the 2nd pass command to install those.

for the case where there are "are dependencies which have updates to their install_requires", I'm not sure I follow you there. you would only discover and want to act on that, if you were wanting to do a full recursive upgrade, right?

[fdintino]

Only if you no longer met the requirements. So, for example, if I updated django-sentry, and django-sentry required django-celery>=2.5.4,django-celery<3.0where it previously required django-celery>=2.5.3,django-celery<3.0 (perhaps because of a bug fix or a new feature), and I currently have django-celery==2.5.3, then I would expect it to update django-celery to satisfy the requirement, the way my patch does. However, if I happened to have django-celery==2.5.4 I would not expect it to update celery. In the case of celery it's not a big deal, but when packages have Django>1.2,Django<=1.4, and projects are often written to target Django 1.2, 1.3, or 1.4, an unexpected upgrade of django can be a huge headache.

[qwcode]

thanks @fdintino . I follow now. you don't want to cut off all recursive behavior (that will do necessary upgrades to fulfill requirements), just want to stop recursive "forced" upgrades.

btw, your comment is cropped due to formatting. might want to fix that for others.

[qwcode]

I posted a gist that breakdowns achieving the desired behavior using the 2 commands in sequence mentioned above.
Not that it invalidates the desire for this ticket or the pull, but it was helpful to me to confirm how
it works currently and what's possible currently.
(hint: in the example, "b" is the parallel to django that was mentioned as a concern)

https://gist.github.com/3088149

comment appreciated on the gist if this is not the scenario.

[mverleg]

I notice this has been open a long time.

I found that my version of pip now has the option to do

pip install --upgrade --upgrade-strategy=only-if-needed package

This seems to be the desired behaviour, although rather verbose. Personally I think it'd have been great if this were the default, but maybe it's too late to change now.

If it's too late to change the default, then I think this can be closed?

[dstufft]

In #3871 (comment) I mentioned what I believe to be the way we're going to move forward on this, reproducing here:

Circling back to this now. Here's what I think we should do:

1. Add --upgrade-strategy=[eager/non-eager] in pip vX which defaults to eager, and allow people to opt-in to the non-eager strategy. This will allow us to get some real world testing from people without forcing a change on everyone at once.
2. Once we've addressed any feedback from 1., switch the default of --upgrade-strategy to non-eager in pip vX+1. This will get us a lot of real world use by forcing a change on everyone, but gives an escape hatch for people who, for whatever reason are broken by the change.
3. Once we've addressed any feedback from 2., look at deprecating --upgrade-strategy in pip vx+2 (to be removed following our normal deprecation policy).

This will have a longer lead time before non-eager is the default, but it allows us to phase it in slower to make sure there's not some use case that's going to go kaboom from it. Ideally I think we'd want to eventually get rid of the --upgrade-strategy flag. That kind of flag feels like it's just asking for trouble with things breaking because it essentially requires us to duplicate our upgrade testing to actually fully test things. I do think it's a good flag to add for now though, to allow the phase in and dealing with any breakages.

[mverleg]

Okay sounds good, I'll wait for step 2 to complete. Seems that other issue is a duplicate, but already closed.

[pradyunsg]

@xavfernandez @dstufft Can this be closed or will we wait till the default switches?

[dstufft]

We'll probably wait till the default switches.

[xavfernandez]

Closing, since #4500 was merged