Skip to content

Commit b05955d

Browse files
encukouencukou
authored
bpo-43795: PEP 652 user documentation (GH-25668)
- Reformat the C API and ABI Versioning page (and extend/clarify a bit) - Rewrite the stable ABI docs into a general text on C API Compatibility - Add a list of Limited API contents, and notes for the individual items. - Replace `Include/README.rst` with a link to a devguide page with the same info
1 parent d1b8157 commit b05955d

File tree

7 files changed

+1188
-1105
lines changed

7 files changed

+1188
-1105
lines changed

‎Doc/c-api/apiabiversion.rst

+52-29
Original file line numberDiff line numberDiff line change
@@ -6,34 +6,57 @@
66
API and ABI Versioning
77
***********************
88

9-
``PY_VERSION_HEX`` is the Python version number encoded in a single integer.
10-
11-
For example if the ``PY_VERSION_HEX`` is set to ``0x030401a2``, the underlying
12-
version information can be found by treating it as a 32 bit number in
13-
the following manner:
14-
15-
+-------+-------------------------+------------------------------------------------+
16-
| Bytes | Bits (big endian order) | Meaning |
17-
+=======+=========================+================================================+
18-
| ``1`` | ``1-8`` | ``PY_MAJOR_VERSION`` (the ``3`` in |
19-
| | | ``3.4.1a2``) |
20-
+-------+-------------------------+------------------------------------------------+
21-
| ``2`` | ``9-16`` | ``PY_MINOR_VERSION`` (the ``4`` in |
22-
| | | ``3.4.1a2``) |
23-
+-------+-------------------------+------------------------------------------------+
24-
| ``3`` | ``17-24`` | ``PY_MICRO_VERSION`` (the ``1`` in |
25-
| | | ``3.4.1a2``) |
26-
+-------+-------------------------+------------------------------------------------+
27-
| ``4`` | ``25-28`` | ``PY_RELEASE_LEVEL`` (``0xA`` for alpha, |
28-
| | | ``0xB`` for beta, ``0xC`` for release |
29-
| | | candidate and ``0xF`` for final), in this |
30-
| | | case it is alpha. |
31-
+-------+-------------------------+------------------------------------------------+
32-
| | ``29-32`` | ``PY_RELEASE_SERIAL`` (the ``2`` in |
33-
| | | ``3.4.1a2``, zero for final releases) |
34-
+-------+-------------------------+------------------------------------------------+
35-
36-
Thus ``3.4.1a2`` is hexversion ``0x030401a2``.
9+
CPython exposes its version number in the following macros.
10+
Note that these correspond to the version code is **built** with,
11+
not necessarily the version used at **run time**.
3712

38-
All the given macros are defined in :source:`Include/patchlevel.h`.
13+
See :ref:`stable` for a discussion of API and ABI stability across versions.
14+
15+
.. c:macro:: PY_MAJOR_VERSION
16+
17+
The ``3`` in ``3.4.1a2``.
18+
19+
.. c:macro:: PY_MINOR_VERSION
20+
21+
The ``4`` in ``3.4.1a2``.
22+
23+
.. c:macro:: PY_MICRO_VERSION
24+
25+
The ``1`` in ``3.4.1a2``.
26+
27+
.. c:macro:: PY_RELEASE_LEVEL
28+
29+
The ``a`` in ``3.4.1a2``.
30+
This can be ``0xA`` for alpha, ``0xB`` for beta, ``0xC`` for release
31+
candidate or ``0xF`` for final.
3932

33+
.. c:macro:: PY_RELEASE_SERIAL
34+
35+
The ``2`` in ``3.4.1a2``. Zero for final releases.
36+
37+
.. c:macro:: PY_VERSION_HEX
38+
39+
The Python version number encoded in a single integer.
40+
41+
The underlying version information can be found by treating it as a 32 bit
42+
number in the following manner:
43+
44+
+-------+-------------------------+-------------------------+--------------------------+
45+
| Bytes | Bits (big endian order) | Meaning | Value for ``3.4.1a2`` |
46+
+=======+=========================+=========================+==========================+
47+
| 1 | 1-8 | ``PY_MAJOR_VERSION`` | ``0x03`` |
48+
+-------+-------------------------+-------------------------+--------------------------+
49+
| 2 | 9-16 | ``PY_MINOR_VERSION`` | ``0x04`` |
50+
+-------+-------------------------+-------------------------+--------------------------+
51+
| 3 | 17-24 | ``PY_MICRO_VERSION`` | ``0x01`` |
52+
+-------+-------------------------+-------------------------+--------------------------+
53+
| 4 | 25-28 | ``PY_RELEASE_LEVEL`` | ``0xA`` |
54+
+ +-------------------------+-------------------------+--------------------------+
55+
| | 29-32 | ``PY_RELEASE_SERIAL`` | ``0x2`` |
56+
+-------+-------------------------+-------------------------+--------------------------+
57+
58+
Thus ``3.4.1a2`` is hexversion ``0x030401a2`` and ``3.10.0`` is
59+
hexversion ``0x030a00f0``.
60+
61+
62+
All the given macros are defined in :source:`Include/patchlevel.h`.

‎Doc/c-api/stable.rst

+153-33
Original file line numberDiff line numberDiff line change
@@ -2,37 +2,157 @@
22

33
.. _stable:
44

5-
***********************************
5+
***************
6+
C API Stability
7+
***************
8+
9+
Python's C API is covered by the Backwards Compatibility Policy, :pep:`387`.
10+
While the C API will change with every minor release (e.g. from 3.9 to 3.10),
11+
most changes will be source-compatible, typically by only adding new API.
12+
Changing existing API or removing API is only done after a deprecation period
13+
or to fix serious issues.
14+
15+
CPython's Application Binary Interface (ABI) is forward- and
16+
backwards-compatible across a minor release (if these are compiled the same
17+
way; see :ref:`stable-abi-platform` below).
18+
So, code compiled for Python 3.10.0 will work on 3.10.8 and vice versa,
19+
but will need to be compiled separately for 3.9.x and 3.10.x.
20+
21+
Names prefixed by an underscore, such as ``_Py_InternalState``,
22+
are private API that can change without notice even in patch releases.
23+
24+
625
Stable Application Binary Interface
7-
***********************************
8-
9-
Traditionally, the C API of Python will change with every release. Most changes
10-
will be source-compatible, typically by only adding API, rather than changing
11-
existing API or removing API (although some interfaces do get removed after
12-
being deprecated first).
13-
14-
Unfortunately, the API compatibility does not extend to binary compatibility
15-
(the ABI). The reason is primarily the evolution of struct definitions, where
16-
addition of a new field, or changing the type of a field, might not break the
17-
API, but can break the ABI. As a consequence, extension modules need to be
18-
recompiled for every Python release (although an exception is possible on Unix
19-
when none of the affected interfaces are used). In addition, on Windows,
20-
extension modules link with a specific pythonXY.dll and need to be recompiled to
21-
link with a newer one.
22-
23-
Since Python 3.2, a subset of the API has been declared to guarantee a stable
24-
ABI. Extension modules wishing to use this API (called "limited API") need to
25-
define ``Py_LIMITED_API``. A number of interpreter details then become hidden
26-
from the extension module; in return, a module is built that works on any 3.x
27-
version (x>=2) without recompilation.
28-
29-
In some cases, the stable ABI needs to be extended with new functions.
30-
Extension modules wishing to use these new APIs need to set ``Py_LIMITED_API``
31-
to the ``PY_VERSION_HEX`` value (see :ref:`apiabiversion`) of the minimum Python
32-
version they want to support (e.g. ``0x03030000`` for Python 3.3). Such modules
33-
will work on all subsequent Python releases, but fail to load (because of
34-
missing symbols) on the older releases.
35-
36-
As of Python 3.2, the set of functions available to the limited API is
37-
documented in :pep:`384`. In the C API documentation, API elements that are not
38-
part of the limited API are marked as "Not part of the limited API."
26+
===================================
27+
28+
Python 3.2 introduced the *Limited API*, a subset of Python's C API.
29+
Extensions that only use the Limited API can be
30+
compiled once and work with multiple versions of Python.
31+
Contents of the Limited API are :ref:`listed below <stable-abi-list>`.
32+
33+
To enable this, Python provides a *Stable ABI*: a set of symbols that will
34+
remain compatible across Python 3.x versions. The Stable ABI contains symbols
35+
exposed in the Limited API, but also other ones – for example, functions
36+
necessary to support older versions of the Limited API.
37+
38+
(For simplicity, this document talks about *extensions*, but the Limited API
39+
and Stable ABI work the same way for all uses of the API – for example,
40+
embedding Python.)
41+
42+
.. c:macro:: Py_LIMITED_API
43+
44+
Define this macro ``Py_LIMITED_API`` before including ``Python.h`` to
45+
opt in to only use the Limited API.
46+
47+
Defining ``Py_LIMITED_API`` to ``3`` will limit the available API so that
48+
the extension will work without recompilation with all Python 3.x releases
49+
(x>=2) on the particular :ref:`platform <stable-abi-platform>`.
50+
51+
Defining ``Py_LIMITED_API`` to a value of :c:data:`PY_VERSION_HEX` will
52+
limit the available API so that the extension will work without
53+
recompilation with all Python 3 releases from the specified one.
54+
This will allow using additional API introduced up to this version,
55+
but the extension will lose compatibility with earlier Python versions.
56+
Rather than using the ``PY_VERSION_HEX`` macro directly, hardcode a minimum
57+
minor version (e.g. ``0x030A0000`` for Python 3.10) for stability when
58+
compiling with future Python versions.
59+
60+
On Windows, extensions that use the Stable ABI should be linked against
61+
``python3.dll`` rather than a version-specific library such as
62+
``python39.dll``.
63+
64+
On some platforms, Python will look for and load shared library files named
65+
with the ``abi3`` tag (e.g. ``mymodule.abi3.so``).
66+
It does not check if such extensions conform to a Stable ABI.
67+
The user (or their packaging tools) need to ensure that, for example,
68+
extensions built with the 3.10+ Limited API are not installed for lower
69+
versions of Python.
70+
71+
All functions in the Stable ABI are present as functions in Python's shared
72+
library, not solely as macros. This makes them usable from languages that don't
73+
use the C preprocessor.
74+
75+
76+
Limited API Scope and Performance
77+
---------------------------------
78+
79+
The goal for the Limited API is to allow everything that is possible with the
80+
full C API, but possibly with a performance penalty.
81+
82+
For example, while :c:func:`PyList_GetItem` is available, its “unsafe” macro
83+
variant :c:func:`PyList_GET_ITEM` is not.
84+
The macro can be faster because it can rely on version-specific implementation
85+
details of the list object.
86+
87+
Without ``Py_LIMITED_API`` defined, some C API functions are inlined or
88+
replaced by macros.
89+
Defining ``Py_LIMITED_API`` disables this inlining, allowing stability as
90+
Python's data structures are improved, but possibly reducing performance.
91+
92+
By leaving out the ``Py_LIMITED_API`` definition, it is possible to compile
93+
a Limited API extension with a version-specific ABI. This can improve
94+
performance for that Python version, but will limit compatibility.
95+
Compiling with ``Py_LIMITED_API`` will then yield an extension that can be
96+
distributed where a version-specific one is not available – for example,
97+
for prereleases of an upcoming Python version.
98+
99+
100+
Limited API Caveats
101+
-------------------
102+
103+
Note that compiling with ``Py_LIMITED_API`` is *not* a complete guarantee that
104+
code conforms to the Limited API or the Stable ABI. ``Py_LIMITED_API`` only
105+
covers definitions, but an API also includes other issues, such as expected
106+
semantics.
107+
108+
One issue that ``Py_LIMITED_API`` does not guard against is calling a function
109+
with arguments that are invalid in a lower Python version.
110+
For example, consider a function that starts accepting ``NULL`` for an
111+
argument. In Python 3.9, ``NULL`` now selects a default behavior, but in
112+
Python 3.8, the argument will be used directly, causing a ``NULL`` dereference
113+
and crash. A similar argument works for fields of structs.
114+
115+
Another issue is that some struct fields are currently not hidden when
116+
``Py_LIMITED_API`` is defined, even though they're part of the Limited API.
117+
118+
For these reasons, we recommend testing an extension with *all* minor Python
119+
versions it supports, and preferably to build with the *lowest* such version.
120+
121+
We also recommend reviewing documentation of all used API to check
122+
if it is explicitly part of the Limited API. Even with ``Py_LIMITED_API``
123+
defined, a few private declarations are exposed for technical reasons (or
124+
even unintentionally, as bugs).
125+
126+
Also note that the Limited API is not necessarily stable: compiling with
127+
``Py_LIMITED_API`` with Python 3.8 means that the extension will
128+
run with Python 3.12, but it will not necessarily *compile* with Python 3.12.
129+
In particular, parts of the Limited API may be deprecated and removed,
130+
provided that the Stable ABI stays stable.
131+
132+
133+
.. _stable-abi-platform:
134+
135+
Platform Considerations
136+
=======================
137+
138+
ABI stability depends not only on Python, but also on the compiler used,
139+
lower-level libraries and compiler options. For the purposes of the Stable ABI,
140+
these details define a “platform”. They usually depend on the OS
141+
type and processor architecture
142+
143+
It is the responsibility of each particular distributor of Python
144+
to ensure that all Python versions on a particular platform are built
145+
in a way that does not break the Stable ABI.
146+
This is the case with Windows and macOS releases from ``python.org`` and many
147+
third-party distributors.
148+
149+
150+
.. _stable-abi-list:
151+
152+
Contents of Limited API
153+
=======================
154+
155+
156+
Currently, the Limited API includes the following items:
157+
158+
.. limited-api-list::

‎Doc/conf.py

+2-1
Original file line numberDiff line numberDiff line change
@@ -225,8 +225,9 @@
225225
# Options for extensions
226226
# ----------------------
227227

228-
# Relative filename of the reference count data file.
228+
# Relative filename of the data files
229229
refcount_file = 'data/refcounts.dat'
230+
stable_abi_file = 'data/stable_abi.dat'
230231

231232
# Sphinx 2 and Sphinx 3 compatibility
232233
# -----------------------------------

0 commit comments

Comments
 (0)