PEP-517 implementation for pip

[gaborbernat]

This is an issue ticket to track discussion and planning on adding PEP-517 support. Pip 10 does support PEP-518 (with the limitation that the build tool must be a wheel - this avoids the issue of circular dependencies). The next step now is to support PEP-517. All I describe here is the result of my discussion with @dstufft at PyCon 2018 sprints.

pip for PEP-518 does (for a given package to build):

• if an sdist is acquired extract it to a tree source,
• get the build tool and build requirements from pyproject.toml (must contain setuptools and wheel),
• for the current build environment (which is a temporary directory) install the build tool and requirements (this happens by invoking pip via a subprocess),
• invoke the build command by using this temporary folder to build a wheel,
• install then the wheel.

Phase 1: pip install in a PEP-517 and 518 would do:

• if an sdist is acquired extract it to a tree source,
• get the build tool pyproject.toml,
• for the current build environment (which is a temporary directory) install the build tool (this happens by invoking pip via a subprocess),
• use get_requires_for_build_sdist (or it's wheel counterpart) to get the build requirements (this happens via invoke python within a subprocess of the build environment),
• for the current build environment (which is a temporary directory) install the build requirements (this happens by invoking pip via a subprocess),
• invoke the build command by using this temporary environment to build a wheel (build_wheel).
• install then the wheel.

Phase 2: allow pip to build packages for distribution - pip build

• this follows the same paths as above with the sole difference that allows for the user to select either wheel or sdist, and that it's invoked from the cli (e.g. pip build . --sdist.

For phase 1 most of the things are already implemented in https://github.com/pypa/pep517 by @takluyver. It even follows the method pip uses to implement PEP-518. I suggest we just take that package as a vendored package within pip, and maybe make slight alternation to it where required. Once that's done we can go onto phase 2.

Please show your take on this if you have any obligations, otherwise I'll try to create a PR for this.

[gaborbernat]

@pradyunsg so how's triaging works?

[pradyunsg]

@gaborbernat Hey! So, there's of a lot of triage to do -- using that label to signal issues/PRs that need a maintainer to respond to. :)

---

Sure! Let me think out loud how things should work... I feel like I have so much here.

Phase 1

• sdist -> wheel -> installed when given an sdist is obviously what pip should do.
  • If we have an sdist, we should unpack it to a temporary directory and proceed.
• local-dir -> ? -> installed is more interesting. :)

Now, I do think having a discussion early about how things should be structured/implemented would be nice, before a PR. Mostly since I've spent a bit of my time thinking about #5051 (although that's a blue whale issue). Basically, I would prefer a clean separation between the current flow and a PEP 517/518 flow.

• What about non-PEP 517/518 source trees/sdists?
  • Their process/building shouldn't be affected. That is important for backward compatibility.
  • This can be tricky to verify -- Refactor InstallRequirement #5051 would be useful here but hey, that's a blue whale.
• Should pip try to do local-dir -> sdist -> wheel unless build_sdist screeches in pain?
  • Yes.
    • I have a preference for the 2 step approach so as to minimize the number of surprises for people. Additionally, both steps should have separate environments for building (or the same one "cleared" and reused).
    • It ensures that what's in the sdist, is enough to build a wheel -- integrity of sdists is "verified" in a way. (could have used better words?)
  • But we get dependency metadata only when building wheels. :(
    • I'm gonna shelve this thought and make a lot of noise when the time comes for this. :P

Phase 2

I'm noting these here now but think these should be discussed after Phase 1 is done.

• Does pip build mean we deprecate pip wheel?
  • Yes. I like the spelling of pip build. :)
• How does pip build work?
  • local-dir -> sdist -> wheel falling back to local-dir -> wheel
    • Yes.
      • The regular flow (as per above), which should be the default
    • pip build .
  • sdist -> wheel
    • Yes.
    • pip build foo-1.0.tar.gz
  • local-dir -> sdist
    • Yes.
    • pip build --sdist-only .
  • local-dir -> wheel
    • Yes.
      • Useful for build backends to test themselves (to actually verify that going directly to wheel gives the same wheel as going via sdist).
    • pip build --wheel-only .

---

Then, PEP 518 support needs to be improved/expanded and we have #5286 + #5336, which if I understand correctly, would get us there. (thanks to @benoit-pierre)

All I describe here is the result of my discussion with @dstufft at PyCon 2018 sprints.

You got to meet him! :O

maybe make slight alternation to it where required.

I do prefer that the slight alterations happen upstream (not modifying vendored packages unless absolutely needed). :)

---

self-note: We will need to be building wheels (or just wheel metadata) during resolution. zazo won't care about this -- it's designed to defer this to when integrating into pip; so let's look into this then. :)

local-dir -> sdist/wheel:

• create a build environment (buildenv)
  • do not copy the local-dir like pip does today
• install build-system.requires in buildenv
• "load" build-system.build-backend
• install build_backend.get_requires_for_build_sdist/wheel in buildenv (optional)
  • subprocess
• build_backend.build_sdist/wheel
  • subprocess
• Proceed to with the generated sdist/wheel

[gaborbernat]

@pradysung do we have an official definition of what an sdist is? I am personally not convinced that we do need the sdist step always. Seems something that could potentially slow us down and redundant. Any reason you want to Keep sdist around in all builds?

[pradyunsg]

@gaborbernat sdist is short for source distribution. The reason to keep it is as an integrity check of the installability of a source distribution. It should be possible to go from a source distribution to an installed package. Ensuring that by default makes sense to me, when installing from local directories.

[gaborbernat]

@pradyunsg integrity check? That just sounds to me like we're trying to validate that the build backend does its job correctly, or that the user does the packaging correctly. I would argue that this would violate separation of concern. Shouldn't pip take care of the package management itself, and leave the build to the responsibility of the build backend, as outlined in PEP-517?

[pradyunsg]

sdist is short for source distribution.

This was wrong from my end to think you didn't know that. Apologies. As for the format of sdists, it's mostly a defacto based on what setuptools generates right now. Moving forward, PEP 517 has a specification: https://www.python.org/dev/peps/pep-0517/#source-distributions

integrity check? That just sounds to me like we're trying to validate that the build backend does its job correctly, or that the user does the packaging correctly.

Mostly the latter. That said, I'm not sure how I feel strongly about this. I'm curious what other think about this.

[pfmoore]

From what I recall, this (whether builds should be done "via sdist" or not) was extensively debated in the PEP 517 discussions. I don't recall the answer, but I think it would be worth researching the discussion threads before starting any implementation. I'm not going to try to summarise what I recollect, as I was one of the participants in the thread, so I'm not unbiased, and I honestly don't remember what the ultimate consensus was.

Yes, I know the PEP should probably specify things like this, but there was way too much debate and controversy to realistically expect that in all cases.

One thing I will say is that we do not want to copy the source tree and build from the copy. That's the source of all the "pip takes too long to build from source" issues we have, because we can't know whether we need to copy (e.g.) .git directories, or huge data files. We should either build in place, or build from sdist, or there's an alternative efficient approach in the PEP that I don't recall right now :-)

[RonnyPfannschmidt]

@pfmoore pip could introduce a tool.pip section where projects can specify whether its safe to copy them among other things