diff --git a/pep-perennial.rst b/pep-perennial.rst new file mode 100644 index 00000000..7d6e3843 --- /dev/null +++ b/pep-perennial.rst @@ -0,0 +1,189 @@ +PEP: XXX +Title: Future 'manylinux' Platform Tags for Portable Linux Built Distributions +Version: $Revision$ +Last-Modified: $Date$ +Author: Nathaniel J. Smith + Thomas Kluyver +BDFL-Delegate: Paul Moore +Discussions-To: Discourse https://discuss.python.org/t/the-next-manylinux-specification/1043 +Status: Draft +Type: Informational +Content-Type: text/x-rst +Created: 3-May-2019 +Post-History: 3-May-2019 + +Abstract +======== + +This PEP proposes a scheme for new 'manylinux' distribution tags to be defined +without requiring a PEP for every specific tag. The naming scheme is based on +glibc versions, with profiles in the auditwheel tool defining what other +external libraries and symbols a compatible wheel may link against. + +While there is interest in defining tags for non-glibc Linux platforms, +this PEP does not attempt to address that. + +Rationale +========= + +Distributing compiled code for Linux is more complicated than for other popular +operating systems, because the Linux kernel and the libraries which typically +accompany it are built in different configurations and combinations for different +distributions. However, there are certain core libraries which can be expected in +many common distributions and are reliably backwards compatible, so binaries +built with older versions of these libraries will work with newer versions. +:pep:`513` describes these ideas in much more detail. + +The ``manylinux1`` (:pep:`513`) and ``manylinux2010`` (:pep:`571`) tags make +use of these features. They define a set of core libraries and symbol versions +which wheels may expect on the system, based on CentOS 5 and 6 respectively. +Typically, packages are built in Docker containers based on these old CentOS +versions, and then the ``auditwheel`` tool is used to check them and bundle any +other linked libraries into the wheel. + +If we were to define a ``manylinux2014`` tag based on CentOS 7, there would be +five steps involved to make it practically useful: + +1. Write a PEP +2. Prepare docker images based on CentOS 7. +3. Add the definition to auditwheel +4. Allow uploads with the new tag on PyPI +5. Add code to pip to recognise the new tag and check if the platform is + compatible + +Although preparing the docker images and updating auditwheel take more work, +these parts can be used as soon as that work is complete. The changes to pip +are more straightforward, but not all users will promptly install a new version +of pip, so distributors are concerned about moving to a new tag too quickly. + +This PEP aims to remove the need for steps 1 and 5 above, so new manylinux tags +can be adopted more easily. + +Naming +====== + +Tags using the new scheme will look like:: + + manylinux_glibc_2_17_x86_64 + +Where ``2_17`` is the major and minor version of glibc. I.e. for this example, +the platform must have glibc 2.17 or newer. Installer tools should be prepared +to handle any numeric values here, but building and publishing wheels to PyPI +will probably be constrained to specific profiles defined by auditwheel. + +The existing manylinux tags can also be represented in the new scheme, +for instance: + +- ``manylinux1_x86_64`` becomes ``manylinux_glibc_2_5_x86_64`` +- ``manylinux2010_x86_64`` becomes ``manylinux_glibc_2_12_x86_64`` + +``x86_64`` refers to the CPU architecture, as in previous tags. + +While this PEP does not attempt to define tags for non-glibc Linux, the name +glibc is included to leave room for future efforts in that direction. + +Wheel compatibility +=================== + +There are two components to a tag definition: a specification of what makes a +compatible wheel, and of what makes a compatible platform. + +A wheel may never use symbols from a newer version of glibc than that indicated +by its tag. Likewise, a wheel with a glibc tag under this scheme may not be +linked against another libc implementation. + +As with the previous manylinux tags, wheels will be allowed to link against +a limited set of external libraries and symbols. These will be defined by +profiles documented on https://packaging.python.org/ and implemented in +auditwheel. At least initially, they will likely be similar to +the list for manylinux2010 (:pep:`571`), and based on library versions in +newer versions of CentOS. + +The overall goal is to ensure that if a wheel is tagged as +``manylinux_glibc_2_Y``, then users can be reasonably confident that this wheel +will work in any real-world linux-based python environment that uses +``glibc 2.Y`` or later and matches the other wheel compatibility tags. +For example, this includes making sure that the wheel only uses symbols that +are available in the oldest supported glibc, and doesn't rely on the system to +provide any libraries that aren't universally available. + +One of the central points of this PEP is to move away from defining each +compatibility profile in its own PEP. +In part, this is to acknowledge that the details of compatibility profiles +evolve over time as the Linux distribution landscape changes and as we learn +more about real-world compatibility pitfalls. +For instance, Fedora 30 `removed `__ +``libcrypt.so.1``, which both ``manylinux1`` and ``manylinux2010`` previously +allowed wheels to externally link. +Auditwheel and the manylinux build images will be updated to avoid new wheels +relying on this as an external library. + +As with the previous manylinux tags, required libraries which are not on +the whitelist will need to be bundled into the wheel. + +Building compatible wheels +-------------------------- + +For each profile defined on https://packaging.python.org/, we plan to provide +a canonical build environment, such as a Docker image, available for people to +build wheels for that profile. +People can build in other environments, so long as the +resulting wheels can be verified by auditwheel, but the canonical environments +hopefully provide an easy answer for most packages. + +The definition of a new profile may well precede the construction of its +build environment; it's not expected that the definition in auditwheel +is held up until a corresponding environment is ready to use. + +Verification on upload +---------------------- + +In the future, PyPI may begin using auditwheel to automatically validate +uploaded manylinux wheels, and reject wheels that it can't determine are +compliant. If PyPI does this, then it will mean that only wheels that have a +corresponding auditwheel profile can be distributed publicly. + +If you need manylinux support for a platform that currently has no profile +in auditwheel, then you're encouraged to contribute a profile to auditwheel. +If that's not possible for some reason, then other tools can be used, +as long as you try to meet the same goal as auditwheel (i.e., the wheel should +work in all environments with the given glibc version and architecture) – +though you may not be able to upload these wheels to PyPI. + +Platform compatibility +====================== + +The checks for a compatible platform on installation consist of a heuristic +and an optional override. The heuristic is that the platform is compatible if +and only if it has a version of glibc equal to or greater than that indicated +in the tag name. + +The override is defined in an importable ``_manylinux`` module, +the same as already used for manylinux1 and manylinux2010 overrides. +For the new scheme, this module must define a function rather than an +attribute. ``manylinux_glibc_compatible(major, minor)`` takes two integers +for the glibc version number in the tag, and returns True, False or None. +If it is not defined or it returns None, the default heuristic is used. + +The compatibility check could be implemented like this:: + + def is_manylinux_glibc_compatible(major, minor): + # Check for presence of _manylinux module + try: + import _manylinux + f = _manylinux.manylinux_glibc_compatible + except (ImportError, AttributeError): + # Fall through to heuristic check below + pass + else: + compat = f(major, minor) + if compat is not None: + return bool(compat) + + # Check glibc version. + # PEP 513 contains an implementation of this function. + return have_compatible_glibc(major, minor) + +The installer should also check that the platform is Linux and that the +architecture in the tag matches that of the running interpreter. +These checks are not illustrated here.