Missing an option to prioritize --find-links over --index-url

[vartec]

• Pip version: master (f1d0939)
• Python version: any
• Operating system: any

Description:

There is no option to force precedence of a wheelhouse passed to --find-links over --index-url (or default index). There is option to disable index completely, but it isn't always desired if you'd like to keep the default index as a fallback for any packages that are not in the wheelhouse.

This stems from the order being hardcoded here:

pip/pip/index.py

Lines 398 to 405 in f1d0939

	url_locations = [
	link for link in itertools.chain(
	(Link(url) for url in index_url_loc),
	(Link(url) for url in fl_url_loc),
	(Link(url) for url in dep_url_loc),
	)
	if self._validate_secure_origin(logger, link)
	]

It would be great if this order would be configurable.

[xavfernandez]

This is strongly related to #3454

[dstufft]

While I recognize that this can be an issue, I don't think we want to overload too many different options here. I think, if I'm guessing about the motivation correctly here, something like --prefer-binary from #3785 is a better general solution.

[alecbz]

If I'm understanding the original snippet correctly, packages found via index URLs will always be preferred over ones locally available, in the case where --find-links is used and --no-index is not?

At the very least, I think this deserves to be documented somewhere. This feels like very surprising behavior.

My use-case: I compiled tensorflow to take advantage of certain CPU features, and put the wheel into our local wheel directory. But pip happily continued to install the version available at pypi and ignored the one I built locally.

I'm not sure what the right answer is here. Would --prefer-binary help at all? I'm not sure if the version of tensorflow available on pypi is a source or binary wheel. I think maybe the best answer is to revert to using --no-index, but that means we need to pull down more wheels locally than we would have otherwise wanted.

I suppose using --hash is maybe another option?

[dstufft]

Change the version in your local tensorflow to add a +something.

[pfmoore]

To expand on that, pip does no prioritising at all of sources. It locates all possible candidates and installs the one with the latest version. If multiple sources offer the same version, all of those candidates should be identical (by definition, same version means same code) and which one is chosen is undefined.

So if you have a local build that's different from the published build, you should give it a local version modifier to express that fact. Your version will then be preferred because version X.Y.Z+local is greater than X.Y.Z.

There are potential use cases for defining (or even allowing the user control over) which source should be preferred when multiple copies of the same version are found, but so far we haven't really seen anything reported that isn't better solved using existing features (like local versions for this case).

[alecbz]

(by definition, same version means same code)

Is this codified somewhere in the context of pypa? You mean that, e.g., for any two wheel files a.whl, b.whl, if they have the same version, package name, platform, python version (maybe some other metadata?), then it is a bug for diff a.whl b.whl to report any differences?

[pfmoore]

I'm not sure it's explicitly written down in precisely that form in a document. But it's implicit in a lot of the assumptions in various tools (and IMO, it's pretty self-evident - if the version, name and target environment are the same, how else should we choose between 2 wheels?)

The last time I recall this being discussed was in the PEP 517 debates on distutils-sig. From what I recall, the debate was inconclusive, but that's mostly because there was no way we could mandate that build systems stick to that rule (and ultimately, it didn't matter as long as we didn't have a practical example of where we'd want to do anything else).

[alecbz]

(and IMO, it's pretty self-evident - if the version, name and target environment are the same, how else should we choose between 2 wheels?)

By preferring a local wheel to one fetched from a remote? :)

I agree that it's very desirable for this to be a true property of versioning systems, but it seems important to take note when it's not the case, and either:

• try to provision for the fact that it is not the case, by, e.g., adding features like the one proposed in this issue
• or, try to fix the fact that it's not always the case, by, e.g., documenting that this is in fact the intended behavior, adding features to tooling to catch these bugs when they occur (pip could maybe report a warning when it encounters two wheels that differ), making it easier to add build metadata when packaging a wheel (since it is often likely necessary to preserve this property), etc.