Speculative: --only-binary by default?

[pfmoore]

What's the problem this feature will solve?
A lot of users are reporting issues when there's no Python 3.9 binary for projects they need, and pip tries to build from source and fails with an obscure error (because the user doesn't have a compiler, or isn't set up to build the relevant packages).

Describe the solution you'd like
Pip shouldn't try to build from source if the user isn't prepared to deal with build errors. As it's not possible to know the user's level of expertise, we should err on the side of caution, and by default only allow wheels to be installed. Users who know they need to install from source and have checked that they can do so, can explicitly say so using a new --allow-source flag, which acts as an "opt-in" to source builds.

Alternative Solutions
Improve the error messages when a source build fails. This is hard, because the details of what went wrong are entirely the responsibility of the build backend.

Additional context
I don't realistically think this can be added without a lot of disruption, but given that significant numbers of projects ship wheels these days, maybe it isn't as unthinkable as it once was. I do think it's worth discussing the implications, if only as a thought experiment, and I don't know where else we could do that apart from here.

One big problem area is that we can't distinguish between "pure Python" projects that are shipped only as sdists, but which only need Python to build, and complex projects that need a compiler. So restricting to wheels only would require an explicit opt-in for some projects which currently install with no issue.

[dstufft]

We could attempt to make the default more intelligent (or maybe just more magical). Basically have the implicit default be that if a wheel is found at all for some project, that project defaults to only allowing wheels.

…

Sent from my iPhone

On Nov 16, 2020, at 12:40 PM, Paul Moore ***@***.***> wrote:  What's the problem this feature will solve? A lot of users are reporting issues when there's no Python 3.9 binary for projects they need, and pip tries to build from source and fails with an obscure error (because the user doesn't have a compiler, or isn't set up to build the relevant packages). Describe the solution you'd like Pip shouldn't try to build from source if the user isn't prepared to deal with build errors. As it's not possible to know the user's level of expertise, we should err on the side of caution, and by default only allow wheels to be installed. Users who know they need to install from source and have checked that they can do so, can explicitly say so using a new --allow-source flag, which acts as an "opt-in" to source builds. Alternative Solutions Improve the error messages when a source build fails. This is hard, because the details of what went wrong are entirely the responsibility of the build backend. Additional context I don't realistically think this can be added without a lot of disruption, but given that significant numbers of projects ship wheels these days, maybe it isn't as unthinkable as it once was. I do think it's worth discussing the implications, if only as a thought experiment, and I don't know where else we could do that apart from here. One big problem area is that we can't distinguish between "pure Python" projects that are shipped only as sdists, but which only need Python to build, and complex projects that need a compiler. So restricting to wheels only would require an explicit opt-in for some projects which currently install with no issue. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[uranusjr]

But we can’t know what “all projects” means before deciding whether to set the flag, since dependency information is inside the sdist/wheel 🙃

[pfmoore]

@uranusjr I'm suggesting making --only-binary :all: the default, which doesn't need to know dependency information...

[uranusjr]

Oops, my previous response was toward @dstufft’s “intelligent” suggestion. Sorry for the confusion.

To express my thoughts in more words, I think the “only wheel unless some project needs to compile from source” would be very difficult to implement since the two parts in the logic depend on each other. I would much prefer @pfmoore’s original suggestion of having --only-binary :all: unless the user explicitly allows source distributions.

[dstufft]

The logic isn’t hard and has nothing to do with dependency information. Current logic is roughly: 1. Fetch a list of links from the index for project X. 2. Filter said list of links using the value of —only-binary (among other things like platform tag). 3. Return list of links for use in the dep solver. The proposed change only slightly changes the logic in step 2 slightly, such that unless the user has explicitly configured only-binary, we will set the value of it implicitly by inspecting the entire list of links we’ve discovered for project X, and determining if there is a wheel file or not. This is simple, and would prevent the breakage that Paul is currently seeing, projects which generally make wheels available, but that haven’t for this version of Python / OS / Whatever. It wouldn’t change anything for projects which don’t ship wheels at all, some of which will be pure Python, some of which will be compiled code, but in any case there’s no “upgrade to Python 3.9 and suddenly start compiling code” problem for these projects since they are consistent in what they require. The biggest issue with this that I see is in the effort of being smarter about our default to not break certain kinds of projects, we make it easier for projects to accidentally break their users. If my project historically did not upload wheels, and then I start uploading wheels with version 3.1, all previous versions suddenly stop working without opting in to some flag. This is done without any obvious change by the user (upgrading versions of pip is an obvious change, but some thing I install starting to upload wheels is not). We could work around that problem by trying to reduce the blast radius of the implicit “wheels only” setting, by saying that we will only filter out non wheel links by default that are of the same version of a wheel we’ve found. Thus if we find an sdist for 1.0, 2.0, 3.0, and 3.1 and we find a wheel for 3.1, when we filter the list of links, we will filter it so it has the sdists for 1.0, 2.0, and 3.0 and the wheel for 3.1. This makes it so that as soon as you upload a wheel for a given version, you’re effectively signaling that not only should a wheel version be preferable, but that the sdist should only be used if explicitly configured to by the user.

…

Sent from my iPhone

On Nov 17, 2020, at 4:48 AM, Tzu-ping Chung ***@***.***> wrote:  Oops, my previous response was toward @dstufft’s “intelligent” suggestion. Sorry for the confusion. To express my thoughts in more words, I think the “only wheel unless some project needs to compile from source” would be very difficult to implement since the two parts in the logic depend on each other. I would much prefer @pfmoore’s original suggestion of having --only-binary :all: unless the user explicitly allows source distributions. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[pfmoore]

Maybe we simply make --prefer-binary the default (rather than --only-binary)? I didn't suggest that originally because it means that we trigger "why don't I get the latest version?" questions. But maybe that's a less serious breakage?

[pradyunsg]

This makes it so that as soon as you upload a wheel for a given version, you’re effectively signaling that not only should a wheel version be preferable, but that the sdist should only be used if explicitly configured to by the user.

I like it? I think something like 98% of packages on PyPI have wheels in the latest release, so I don't think this is catastrophically bad.

Improve the error messages when a source build fails. This is hard, because the details of what went wrong are entirely the responsibility of the build backend.

IMO one of the improvements we should make here is adding a sentence like: "This failure occurred while trying to generate [a wheel / metadata] for packageName. This is not an error in pip."

This also applies to the proposed approach here too -- clearer error messaging would be good. :)

[uranusjr]

I like it? I think something like 98% of packages on PyPI have wheels in the latest release, so I don't think this is catastrophically bad.

I suspect the number would be significantly lower if you count percentage of downloads instead. There are a bunch of popular pure-Python projects that don’t bother with wheels because the effect is minimal. django-grappelli is one of my favourite examples: it’s popular, well-maintained, regularly released, and has very spotty wheel support. --prefer-binary by default would break a lot of Django setups out there.

[pfmoore]

I think something like 98% of packages on PyPI have wheels in the latest release

I'm pretty sure that's a figure I gave you, and I found the bug in my calculation a bit later 🙁 I need to re-do the sums, but I think it's a lot lower than that, unfortunately.

I suspect the number would be significantly lower if you count percentage of downloads instead.

The number's a lot lower without doing the sums incorrectly 🙂 Sorry about that. I don't have download information, but I'm re-doing the numbers right now, and I'll see what things look like if you factor in "uploaded a file in the last 12 months" as well.

I might try getting download numbers from the BigQuery data for offline analysis. Downloads per project, per year (month?) might be sufficiently interesting, if I can work out how to get that relatively easily in a CSV format or similar.

To confirm, my query has just completed. Comparing "number of projects that distribute sdists but no wheels for their latest version", vs "number of projects that distribute wheels for their latest version", the numbers are almost identical (124508 vs 124782). Looking at projects which have released at least one file in the last year, the values are 32890 and 66635.

So half of all projects, 2/3 of projects active in the last year, have wheels.

As I say, I think that however we did this, it would result in a lot of breakage.

[dstufft]

It's a backwards incompatible change, so regardless it's going to break someone. The goal behind my proposal is to limit the blast radius, so that we limit the breakage, either to specific projects, or to specific versions within a project.

I think there's two questions here too:

• What do we want the long term position to be, are we happy saying that eventually a project that has never shipped, and will never ship wheels requires an opt in on the CLI to install?
• Given the answer to the first, what stepping stones can we make to get there? Is there any or do we need a big bang migration?

I'm not sure about the long term "right" answer. I can see an argument that we want to encourage wheels where possible.. but I also think that there are some projects that simply cannot be shipped as wheels, and maybe will never be able to be shipped as wheels. We need to figure out if going wheel only by default will end up being worth it, or if we will push too many projects out of viability.

For the second one, I think having the default by to filter out sdists, for any project version that has any wheels uploaded, solves the main driver to this proposal, without breaking projects that are not shipping wheels (or used to ship wheels, but found out that was problematic). That could be useful as a stepping stone for getting to a wheel only default (for instance, we could provide warning when installing from sdist then), or it could be a reasonable end state that solves the surprising accidental sdist install, without dropping support for sdist only projects by default.

[pfmoore]

I think we could do a lot better if we could somehow identify which projects are "hard" to build from source. I feel like blocking sdists that build into universal wheels is going a bit far. In the most general sense, that's basically impossible, but maybe we could add metadata somewhere (in the simple index?) to mark "pure Python" projects?

I agree it's not clear what the best long term answer is. We're seeing a lot more people using Python nowadays who honestly don't want to, or know how to, deal with building stuff from source. For those people, pip downloading a sdist that needs a compiler to build is almost certainly just a source of problems. But they are also precisely the sorts of user who won't know enough to add --prefer-binary. However, optimising for such users is going to impact a big chunk of our "traditional" user base negatively.

[dstufft]

I wonder if we can leverage PyPI in some way to encourage wheels, or to at least surface better information to highlight which projects don't ship wheels? This might be a better question for discourse? I dunno.

[pfmoore]

I've got a big chunk of downloaded data from PyPI that I am querying to get a better feel for this sort of stuff. The biggest problem is the vast amount of (to be polite) "limited value projects" on there - without some form of insight, it's hard to know for sure whether it's OK to ignore a project called "0html" or "django-3-jet-zupit" - especially when it comes up in the same query as "090807040506030201testpip"...