New Resolver unable to handle multiple versions of git dependencies

[Tbilgere-td]

What did you want to do?
New Resolver is unable to handle cases where private Git Dependencies are shared but do not match "version" (i.e. Branch, Tag, Commit). This not only makes repo management much more tedious but also makes feature testing in repos much harder or impossible.

For example if we have:

• a repo with common shared code called common
• a repo with logic to operate a particular service called service
• a repo for a app called app

Requirements.txt in service calls:

commoncode @ git+git://github.com/repoOrg/common.git@v1.0.0

While requirements.txt in app calls:

commoncode @ git+git://github.com/repoOrg/common.git@v1.0.1
service @ git+git://github.com/repoOrg/service.get@v2.1.0

Output

ERROR: Cannot install -r requirements.txt (line 70), -r requirements.txt (line 71) and common 1.0.1 (from git+git://github.com/repoOrg/common@v1.0.1) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested common 1.0.1 (from git+git://github.com/repoOrg/common@v1.0.1)
    service 2.1.0 depends on common 1.0.0 (from git+git://github.com/repoOrg/common@v1.0.0)

Additional information
I have spent a considerable amount of time reading though available documentation and have not been able to identify a solution for this use case (such as version ranging). Am I missing something? Ideally there would be some way to specify compatible range in service, with concrete versions specified in app.

[pradyunsg]

Thanks for filing an issue!

I'm unsure what you're asking for tho. pip is correctly rejecting to install 1.0.0 and 1.0.1, which (from pip's resolvers' perspectives) are two different URLs that don't hold the same artifact. This is exactly what the error message says: that there's a conflict.

Could you elaborate on what you are trying to do, and what kind of solution you are looking for?

[Tbilgere-td]

What you are trying to do

Trying to maintain our current (seemingly common) workflow with the new resolver.

What kind of solution you are looking for?

Ideally one that doesn't disrupt our ability to share common code across multiple repos. This doesn't seem like it should not be a covered use case by pip. If that requires specifying compatibility ranges as I stated before that would be fine but currently there appears to be no way to do so.

[andylamp]

Hello all,

Actually @pradyunsg, it seems I've stumbled across the same issue albeit this happens with the same artifacts.

The conflict is caused by:
    The user requested myorg-logger 0.0.7 (from git+https://****@github.com/myorg/myorg-logger.git#egg=myorg-logger)
    myorg-utilities 0.0.6 depends on myorg-logger 0.0.7 (from git+https://****@github.com/myorg/myorg-logger.git@main)

The workflow is the same as described above, however all of the packages try to pull the same common package myorg-logger.git@main and still results in the same issue; there is no issue with the dependencies as it pulls the same versions for both instances.

This, obviously, started happening after the upgrade to 20.3.3 😄 -- any clues on how to fix this?

[sbidoul]

@andylamp in your case it's probably because one URL explicitly mentions the @main branch and the other not.

[andylamp]

@sbidoul it happens regardless if I have the main branch there or not, unfortunately 😢

edit, @sbidoul here is a log that has the branch removed:

Collecting myorg-logger@ git+https://84acb367756ff1d77f68e4a5f923cc85b66a159e@github.com/myorg/myorg-logger.git
  Cloning https://****@github.com/myorg/myorg-logger.git to /tmp/pip-install-sc3bdyo9/myorg-logger_c5ccc5f317fa4e5ab51b4bb315bdd340
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done

The conflict is caused by:
    The user requested myorg-logger 0.0.7 (from git+https://****@github.com/myorg/myorg-logger.git#egg=myorg-logger)
    myorg-utilities 0.0.7 depends on myorg-logger 0.0.7 (from git+https://****@github.com/myorg/myorg-logger.git)

The issue is still there, as I said...

[aaraney]

I am also facing a related problem. I think that @sbidoul is potentially hinting at it. In my case, I have two packages A and B, B depends on A. Both of the packages are strictly on GitHub and not published on pypi. But both have setup.py etc. so they are installable via GitHub using https, git, or ssh protocols.

Since B depends on A, in B's setup.py, A is listed in the install_requires list using the following (notice protocol used https):

[
"A@ git+https://github.com/someuser/A.git",
]

The B package is installable like so, pip install git+https://github.com/someuser/B. Simple enough.

However, if you place both of the packages in a requirements.txt file:

git+git://github.com/someuser/A
git+git://github.com/someuser/B

and change the protocol that is used to install (git or ssh), a conflict is raised.

Naively, it seems that pip is comparing the string literal to determine conflicts. I am probably wrong about this, but if this is the case, functionally equivalent dependencies that use different protocols will have a conflict. This is also inclusive of if as @andylamp pointed out, @main is included. From my working hypothesis, the string literal comparison would fail because, you know the @main part.

[andylamp]

@aaraney in my case all packages strictly use git+https to fetch them in all instances, just to provide some clarity on my end 😄

[Tbilgere-td]

From my working hypothesis, the string literal comparison would fail because, you know the @main part.

or in @andylamp 's case I think the issue is the #egg=myorg-logger and/or @main

[goodspark]

I have a similar problem. pip 20.3.0 and 20.3.3

Obfuscated the names, but hopefully it's clear enough:

The conflict is caused by:
    The user requested thing 1.2.3 (from git+https://github.com/fake/thing.git@my-branch-v1.2.3#egg=thing)
    blah 4.5.6 depends on thing 1.2.3 (from git+https://github.com/fake/thing.git@my-branch-v1.2.3)

It looks like the egg is being ignored from blah. I wonder if that's making pip fail? In the requirements for blah, it's definitely there though. Again, obfuscated:

pytz
thing @ git+https://github.com/fake/thing.git@my-branch-v1.2.3#egg=thing

Does not happen on pip 20.2.4.

[andylamp]

@Tbilgere-td just to add to the discussion, reverting back to pip 20.2.4 everything works as expected.

[aaraney]

@goodspark

From my working hypothesis, the string literal comparison would fail because, you know the @main part.

It may be that pip is doing an atomic comparison between the deps, so if #egg=thing is included, then the comparison is false.

@andylamp having the same behavior here on < 20.3.*

[Tbilgere-td]

@andylamp right as described in documentation here 20.3 moves to the "new resolver" by default you can also add the --use-deprecated=legacy-resolver flag to <20.3.0 but that goes away when 21 drops...

[hellkite500]

@aaraney @andylamp @goodspark it isn't just the protocol or differences in the repo branch, it looks like ANY difference in the URL will trigger this. Confirmed by trying each of the following variants, simplified to aaraney's example:
B's setup.py

install_requires=[
"A@ git+https://github.com/someuser/A.git",
]

Then to install both from requirements.txt (or in any other way where dependencies are resolved, I would imagine)
You would have to write the requirement verbatim as defined in the install_requires

git+https://github.com/someuser/A.git
git+git://github.com/someuser/B

Even a simple change in the url causes an unresolvable dependency, i.e ommit the .git or just change case of part of the url

git+https://github.com/someuser/A
git+git://github.com/someuser/B

git+https://github.com/SOMEUSER/A.git
git+git://github.com/someuser/B

git+https://github.com/SOMEUSER/A.git#egg=thing
git+git://github.com/someuser/B

You can install A using any of the above variants by itself, but not in conjunction with B which defines install_requires

[Tbilgere-td]

Can confirm @hellkite500 's findings we have consistency throughout our git installs and do not hit this case either in any of our repos.

[andylamp]

@hellkite500 I think that's the problem I am hitting - I am using two packages A and B that depend on the same package C; C resides both in the requirements.txt of A and B as well as in their install_requires.