PEP 770: Add the dist-info.files table and reserve dist-info dirs

[sethmlarson]

• Change is either:
  • To a Draft PEP
  • To an Accepted or Final PEP, with Steering Council approval
  • To fix an editorial issue (markup, typo, link, header, etc)
• PR title prefixed with PEP number (e.g. PEP 123: Summary of changes)

cc @rgommers

---

📚 Documentation preview 📚: https://pep-previews--4283.org.readthedocs.build/

[jkowalleck]

I believe this proposal constitutes a significant scope change and is not aligned with the original objectives.

The initial scope was to add SBOM to packages. However, the proposed changes introduce the addition of arbitrary data to packages, which significantly broadens the scope. This new capability/topic deviates from the original focus and introduces complexities that will progress at a different pace than the initial scope. It appears that this change could be an attempt to shift the focus or slow down the progress by diverting from the original scope.

I strongly oppose this PR and the proposed changes. If there is a substantial interest in including arbitrary data in packages, a separate PEP should be initiated.

[rgommers]

Thanks for the quick update @sethmlarson. I reviewed both the diff and the full new text, it looks pretty good to me. This change is a signifcant improvement overall in my opinion.

The checks on what .dist-info directories are already in use looks thorough, thanks for that.

The sentence under Acknowledgements referencing PEP 639 is out of date now, it still says that this PEP implements the PEP 639-like scheme.

You may want to mention somewhere that using a separate [additional-files] key means that packages can start using it directly after acceptance of this PEP, rather than have a >1 yr rollout plan that would have been needed with a core metadata version bump.

You may want to mention that SBOM tooling should always check the actual artifact of interest; no assumptions can be made about the sboms directory having the same contents between different artifacts for a single version of a package.

[rgommers]

@jkowalleck please don't post the same comment in two places. You seem to be new to Python packaging conversations, so for context: Discourse is the right place for high-level discussion and comments like "I prefer approach X over approach Y because Z"; PEP PRs are more editorial and for initial reviews on correctness/completeness before posting to a broader audience on Discourse.

[rgommers]

@sethmlarson the description of PEP 725 looks pretty good as well. I'd suggest one small tweak to make the first bullet point more clear, since "abstract dependencies" isn't a well-defined term and you're only giving virtual: rather than a pkg: example. I suggest to add a more apples to apples comparison for OpenSSL/libssl: the PEP 725 dependency specifier pkg:generic/openssl is what will result in pkg:pkg:rpm/almalinux/[email protected] in an SBOM.

The analogy that I tend to use is "dependencies in pyproject.toml vs. "resolved dependencies in a lock file".

[jkowalleck]

re: #4283 (review)

You may want to mention somewhere that using a separate [additional-files] key means that packages can start using it directly after acceptance of this PEP, rather than have a >1 yr rollout plan that would have been needed with a core metadata version bump.

Is "additional-files" part of core metadata already?

[pfmoore]

Is "additional-files" part of core metadata already?

It's not a metadata item, it's a key in pyproject.toml.

[jkowalleck]

re: #4283 (review)

You may want to mention somewhere that using a separate [additional-files] key means that packages can start using it directly after acceptance of this PEP, rather than have a >1 yr rollout plan that would have been needed with a core metadata version bump.

Is "additional-files" part of core metadata already?

re: #4283 (comment)

Is "additional-files" part of core metadata already?

It's not a metadata item, it's a key in pyproject.toml.

Could you explain what this "rather than have a >1 yr rollout plan that would have been needed with a core metadata version bump" is supposed to mean?

[pfmoore]

This looks great to me. I think any comments I might have had have already been covered by @rgommers

You've put a lot more details into the specification of additional-files than I had anticipated, but that's a good thing - it feels like it's well prepared for any future use.

[pfmoore]

@jkowalleck If we put the data into the core metadata (ignoring for now the problem that we don't have a viable way of doing so that conforms to the rules around dynamic and static metadata) , then we'd have to wait for a new metadata version to be implemented by build backends, recognised by PyPI and other tools, and supported throughout the ecosystem before it would be usable. The experience with the license-files metadata is that (a) this is a slow process, and (b) if tools try to implement support before the infrastructure is in place, we end up causing problems for users that take even longer to fix.

Conversely, if we add a new key to pyproject.toml, tools can add support for it with very little need to wait for infrastructure support. The .dist-info subdirectories are currently unregulated, so as long as tools write content that won't be invalidated by this PEP, they won't hit problems by being early in doing so.

But as @rgommers said, can this discussion be relocated to Discourse if you wish to continue it, please?

[sethmlarson]

Thanks @rgommers and @pfmoore for your review and kind words. I've pushed some changes (a307eb4) addressing feedback, please take a look.

[rgommers]

A few more comments. Overall LGTM. You may perhaps want to wait with merging until the [additional-files] naming bikeshed has been painted.

[brettcannon]

FYI the latest changes make the title of the PR a bit misleading. 😉

[sethmlarson]

@brettcannon hehe, good catch! I've fixed the PR title.