PEP 770: Add the additional-files table and reserve dist-info dirs

Author: sethmlarson

peps/pep-0770.rst

@@ -179,98 +179,58 @@ Specification
 
 The changes necessary to implement this PEP include:
 
-* Additions to `Core Metadata <770-spec-core-metadata_>`_, as defined in the
-  `Core Metadata specification <coremetadataspec_>`__.
-* Additions to the author-provided
-  `project source metadata <770-spec-project-source-metadata_>`_, as defined in the
-  `pyproject.toml specification <pyprojecttoml_>`__.
-* `Additions <770-spec-project-formats_>`_ to the source distribution (sdist),
-  built distribution (wheel), and installed project specifications.
+* A new reserved registry of subdirectory names in the ``.dist-info`` directory.
+* A new reserved optional ``[additional-files]`` table with an optional
+  ``sboms`` key added to
+  `project source metadata <770-spec-project-source-metadata_>`_,
+  as defined in the `pyproject.toml specification <pyprojecttoml_>`__.
+* `Additions <770-spec-project-formats_>`_ to the built distribution (wheel),
+  and installed project specifications
 
 In addition to the above, an informational PEP will be created for tools
 consuming included SBOM documents and other Python package metadata to
 generate complete SBOM documents for Python packages.
 
-Terminology
------------
+.. _770-spec-dist-info-subdirs:
+
+Reserved ``.dist-info`` subdirectories registry
+-----------------------------------------------
+
+This PEP introduces a new registry of reserved subdirectory names allowed in
+the ``.dist-info`` directory for the :term:`distribution archive`
+and :term:`installed project` s project types. Future additions to this registry
+will be made through the PEP process. The initial values in this registry are:
+
+.. table::
 
-This section describes terminology used later in the document:
-
-.. glossary::
-
-    root SBOM directory
-        The directory under which SBOM files are stored in a
-        :term:`project source tree`, :term:`distribution archive`
-        or :term:`installed project`.
-        Also, the root directory that their paths
-        recorded in the :ref:`Sbom-File <770-spec-sbom-file-field>`
-        :term:`Core Metadata field` are relative to.
-        Defined to be the :term:`project root directory`
-        for a :term:`project source tree` or
-        :term:`source distribution <Source Distribution (or "sdist")>`;
-        and a subdirectory named ``sboms`` of
-        the directory containing the :term:`built metadata`—
-        i.e., the ``.dist-info/sboms`` directory—
-        for a :term:`Built Distribution` or :term:`installed project`.
-
-.. _770-spec-core-metadata:
-
-Core Metadata
--------------
-
-.. _770-spec-sbom-file-field:
-
-Add ``Sbom-File`` field
-~~~~~~~~~~~~~~~~~~~~~~~
-
-The ``Sbom-File`` is a new optional Core Metadata field. Each instance contains a
-string representation of the path to an SBOM document. The path is specified
-relative to the :term:`root SBOM directory` for all project types. It is a
-multi-use field that MAY appear zero or more times and each instance lists the
-path to one such file. Files specified under this field are SBOM documents
-that are distributed with the package.
-
-As `specified by this PEP <#770-spec-project-formats>`__, its value is also
-that file's path relative to the :term:`root SBOM directory` in both installed
-projects and the standardized Distribution Package types.
-
-If an ``Sbom-File`` is listed in a
-:term:`Source Distribution <Source Distribution (or "sdist")>` or
-:term:`Built Distribution`'s Core Metadata:
-
-* That file MUST be included in the :term:`distribution archive` at the
-  specified path relative to the :term:`root SBOM directory`.
-* Installers MUST install the file with the :term:`project` at that same
-  relative path.
-* Inside the :term:`root SBOM directory`, packaging tools MUST reproduce the
-  directory structure under which the source files are located relative to the
-  project root.
-* Path delimiters MUST be the forward slash character (``/``), and parent
-  directory indicators (``..``) MUST NOT be used.
-
-For all newly-uploaded distribution archives that include one or more
-``Sbom-File`` fields in their Core Metadata and declare a ``Metadata-Version``
-of ``2.5`` or higher, PyPI and other indices SHOULD validate that all files
-specified with ``Sbom-File`` are present in the distribution archives.
+    ================= ==========
+    Directory name    PEP
+    ================= ==========
+    ``licenses``      :pep:`639`
+    ``license_files`` N/A (See :ref:`770-backwards-compat`)
+    ``sboms``         :pep:`770`
+    ================= ==========
+
+Build backends MUST NOT create subdirectories in the ``.dist-info`` directory
+beyond the names in the registry to avoid collisions with future reserved names.
 
 .. _770-spec-project-source-metadata:
 
 Project source metadata
 -----------------------
 
-This PEP specifies changes to the project's source metadata under a
-``[project]`` table in the ``pyproject.toml`` file.
+This PEP specifies changes to the project's source metadata
+in the ``pyproject.toml`` file:
 
-Add ``sbom-files`` key
-~~~~~~~~~~~~~~~~~~~~~~
+Add new ``[additional-files]`` table
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-A new optional ``sbom-files`` key is added to the ``[project]`` table for
-specifying paths in the project source tree relative to ``pyproject.toml`` to
-file(s) containing SBOMs to be distributed with the package. This key
-corresponds to the ``Sbom-File`` fields in the Core Metadata.
+A new optional ``[additional-files]`` table is added for specifying paths
+in the project source tree relative to ``pyproject.toml`` to file(s) which
+should be included in the built project to a defined directory.
 
-Its value is an array of strings which MUST contain valid glob patterns, as
-specified below:
+This new table has only one defined optional key: ``sboms``. The value of the
+``sboms`` key MUST be an array of valid glob patterns, as specified below:
 
 * Alphanumeric characters, underscores (``_``), hyphens (``-``) and dots (``.``)
   MUST be matched verbatim.
@@ -292,50 +252,48 @@ they can also be defined.
 
 Build tools:
 
-* MUST treat each value as a glob pattern, and MUST raise an error if the
-  pattern contains invalid glob syntax.
+* MUST treat each value in the array as a glob pattern, and MUST raise an error
+  if the pattern contains invalid glob syntax.
 * MUST include all files matched by a listed pattern in all distribution
-  archives.
-* MUST list each matched file path under an ``Sbom-File`` field in the
-  Core Metadata.
+  archives under the ``.dist-info/sboms`` directory.
 * MUST raise an error if any individual user-specified pattern does not match
   at least one file.
 
-If the ``sbom-files`` key is present and is set to a value of an empty array,
+If the ``sboms`` key is present and is set to a value of an empty array,
 then tools MUST NOT include any SBOM files and MUST NOT raise an error.
 
 Examples of valid SBOM files declarations:
 
 .. code-block:: toml
 
-    [project]
-    sbom-files = ["bom.json"]
+    [additional-files]
+    sboms = ["bom.json"]
 
-    [project]
-    sbom-files = ["sboms/openssl.cdx.json", "sboms/openssl.spdx.json"]
+    [additional-files]
+    sboms = ["sboms/openssl.cdx.json", "sboms/openssl.spdx.json"]
 
-    [project]
-    sbom-files = ["sboms/*"]
+    [additional-files]
+    sboms = ["sboms/*"]
 
-    [project]
-    sbom-files = []
+    [additional-files]
+    sboms = []
 
 Examples of invalid SBOM files declarations:
 
 .. code-block:: toml
 
-    [project]
-    sbom-files = ["..\bom.json"]
+    [additional-files]
+    sboms = ["..\bom.json"]
 
 Reason: ``..`` must not be used. ``\\`` is an invalid path delimiter, ``/``
 must be used.
 
 .. code-block:: toml
 
-    [project]
-    sbom-files = ["bom{.json*"]
+    [additional-files]
+    sboms = ["bom{.json*"]
 
-Reason: ``bom{.json`` is not a valid glob.
+Reason: ``bom{.json*`` is not a valid glob.
 
 .. _770-spec-project-formats:
 
@@ -345,36 +303,22 @@ SBOM files in project formats
 A few additions will be made to the existing specifications.
 
 :term:`Project source trees <Project source tree>`
-  Per :ref:`639-spec-source-metadata` section, the
+  Per :ref:`770-spec-project-source-metadata` section, the
   `Declaring Project Metadata specification <pyprojecttoml_>`__
-  will be updated to reflect that SBOM file paths MUST be relative to the
-  project root directory; i.e. the directory containing the ``pyproject.toml``
-  (or equivalently, other legacy project configuration,
-  e.g. ``setup.py``, ``setup.cfg``, etc).
-
-:term:`Source distributions (sdists) <Source Distribution (or "sdist")>`
-  The sdist specification will be updated to reflect that if the
-  ``Metadata-Version`` is ``2.5`` or greater, the sdist MUST contain any SBOM
-  files specified by the ``Sbom-File`` field in the ``PKG-INFO`` at their
-  respective paths relative to the sdist (containing the ``pyproject.toml`` and
-  the ``PKG-INFO`` Core Metadata).
+  will be updated to add the ``[additional-files]`` table
+  and optional ``sboms`` key.
 
 :term:`Built distributions <Built distribution>` (:term:`wheels <wheel>`)
-  The wheel specification will be updated to reflect that if the
-  ``Metadata-Version`` is ``2.5`` or greater and one or more ``Sbom-File``
-  fields are specified, the ``.dist-info`` directory MUST contain an ``sboms``
-  subdirectory, which MUST contain the files listed in the ``Sbom-File`` fields
-  in the ``METADATA`` file at their respective paths relative to the ``sboms``
-  directory.
+
+  The wheel specification will be updated to add the new registry of reserved
+  directory names and to reflect that if the ``.dist-info/sboms`` subdirectory
+  is specified that the directory contains SBOM files.
 
 :term:`Installed projects <Installed project>`
-  The Recording Installed Projects specification will be updated to reflect that
-  if the ``Metadata-Version`` is ``2.5`` or greater and one or more
-  ``Sbom-File`` fields is specified, the ``.dist-info`` directory MUST contain
-  an ``sboms`` subdirectory which MUST contain the files listed in the
-  ``Sbom-File`` fields in the ``METADATA`` file at their respective paths
-  relative to the ``sboms`` directory, and that any files in this directory MUST
-  be copied from wheels by install tools.
+  The Recording Installed Projects specification will be updated to reflect
+  that if the ``.dist-info/sboms`` subdirectory is specified that the directory
+  contains SBOM files and that any files in this directory MUST be copied from
+  wheels by install tools.
 
 SBOM data interoperability
 --------------------------
@@ -406,18 +350,69 @@ PyPI and other indices MAY validate the contents of SBOM documents specified by
 this PEP, but MUST NOT validate or reject data for unknown
 SBOM standards, versions, or fields.
 
+.. _770-backwards-compat:
+
 Backwards Compatibility
 =======================
 
-There are no backwards compatibility concerns for this PEP.
-
-The changes to Python package Core Metadata and ``pyproject.toml`` are
-only additive, this PEP doesn't change the behavior of any existing fields.
-
-Tools which are processing Python packages can use the ``Sbom-File`` core
-metadata field to clearly delineate between packages which include SBOM
-documents that implement this PEP (and thus have more requirements) and
-packages which include SBOM documents before this PEP was authored.
+Reserved ``.dist-info`` subdirectories registry
+-----------------------------------------------
+
+The new registry of reserved ``.dist-info`` subdirectories represents
+a new reservation that wasn't previously documented, thus has the potential to
+break assumptions being made by already existing tools.
+
+To check what ``.dist-info`` subdirectory names are in use today
+a query across
+`all files in package archives on PyPI <https://sethmlarson.dev/security-developer-in-residence-weekly-report-18>`__
+was executed:
+
+.. code-block:: sql
+
+    SELECT (
+      regexp_extract(archive_path, '.*\.dist-info/([^/]+)/', 1) AS dirname,
+      COUNT(DISTINCT project_name) AS projects
+    )
+    FROM '*.parquet'
+    WHERE archive_path LIKE '%.dist-info/%/%'
+    GROUP BY dirname ORDER BY projects DESC;
+
+Note that this only includes records for
+*files* and thus won't return results for empty directories. Empty directories
+being pervasively used and somehow load-bearing is unlikely, so is an accepted
+risk of using this method. This query yielded the following results:
+
+.. table::
+
+    ====================== ===============
+    Subdirectory           Unique Projects
+    ====================== ===============
+    ``licenses``           22,026
+    ``license_files``      1,828
+    ``LICENSES``           170
+    ``.ipynb_checkpoints`` 85
+    ``license``            18
+    ``.wex``               9
+    ``dist``               8
+    ``include``            6
+    ``build``              5
+    ``tmp``                4
+    ``src``                3
+    ``calmjs_artifacts``   3
+    ``.idea``              2
+    ====================== ===============
+
+Not shown above are around ~50 other subdirectory names that are used in a
+single project. From these results we can see:
+
+* Most subdirectories under ``.dist-info`` are to do with licensing,
+  one of which (``licenses``) is specified by :pep:`639` and other
+  (``license_files``) which is being used by the Hatch build backend.
+* The ``sboms`` subdirectory doesn't collide with existing use.
+* Other subdirectory names under ``.dist-info`` appear to be either not
+  widespread or accidental.
+
+As a result of this query
 
 Security Implications
 =====================
@@ -463,12 +458,11 @@ For packages which cannot be automatically annotated and if the package author
 wishes to provide an SBOM the approach will be to generate or author SBOM files
 and then include those files using ``pyproject.toml``:
 
- .. code-block:: toml
+.. code-block:: toml
 
-    [project]
-    ...
-    sbom-files = [
-        "sboms/bom.cdx.json"
+    [additional-files]
+    sboms = [
+       "sboms/bom.cdx.json"
     ]
 
 For projects manually specifying an SBOM document the challenge will be
@@ -558,6 +552,31 @@ a single SBOM standard. Tools that use SBOM data today already need to support
 multiple formats to handle this situation, so a future standard that updates to
 require only one standard would have no effect on downstream SBOM tools.
 
+Using metadata fields to specify SBOM files in archives
+-------------------------------------------------------
+
+A previous iteration of this specification used an ``Sbom-File`` metadata
+field to specify an SBOM file within a source or binary distribution archive.
+This would make the implementation similar to :pep:`639` which uses the
+``License-File`` field to enumerate license files in archives.
+
+The primary issue with this approach is that SBOM files can originate from both
+static and dynamic sources: like versioned source code, the build backend,
+or from tools adding SBOM files after the build has completed (like auditwheel).
+
+Metadata fields must either be static or dynamic, not both. This is
+in direct conflict with the best-case scenario for SBOM data: that SBOM files
+are added automatically by tools during the build of a Python package without
+user-involvement or knowledge. Compare this situation to license files which
+are almost always static.
+
+The 639-style approach was ultimately dropped in favor of defining SBOMs simply
+by their presence in the ``.dist-info/sboms`` directory and using a new table in
+``pyproject.toml`` called ``[additional-files]`` to define SBOMs in source
+distributions. This approach allows users to specify static SBOM files while
+still empowering build backends and tools to add their own SBOM data without the
+static/dynamic conflict.
+
 Open Issues
 ===========
 
@@ -579,6 +598,10 @@ References
   wheel and then use an SBOM generation tool Syft to detect the SBOM in the
   installed package.
 
+* `Querying every file in every release on PyPI <https://sethmlarson.dev/security-developer-in-residence-weekly-report-18>`_.
+  The dataset available on `py-code.org <py-code.org>`__ from Tom Forbes was
+  used to check subdirectory usage in ``.dist-info`` files.
+
 .. _phantom dependency: https://www.endorlabs.com/learn/dependency-resolution-in-python-beware-the-phantom-dependency
 .. _coremetadataspec: https://packaging.python.org/specifications/core-metadata
 .. _pyprojecttoml: https://packaging.python.org/en/latest/specifications/pyproject-toml/