Skip to content

[3.8] bpo-38816: Add notes in the C-API docs about fork in subinterpreters. (GH-17176) #17178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 15, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion Doc/c-api/init.rst
Original file line number Diff line number Diff line change
Expand Up @@ -769,9 +769,19 @@ supports the creation of additional interpreters (using
:c:func:`Py_NewInterpreter`), but mixing multiple interpreters and the
:c:func:`PyGILState_\*` API is unsupported.


.. _fork-and-threads:

Cautions about fork()
---------------------

Another important thing to note about threads is their behaviour in the face
of the C :c:func:`fork` call. On most systems with :c:func:`fork`, after a
process forks only the thread that issued the fork will exist. That also
process forks only the thread that issued the fork will exist. This has a
concrete impact both on how locks must be handled and on all stored state
in CPython's runtime.

The fact that only the "current" thread remains
means any locks held by other threads will never be released. Python solves
this for :func:`os.fork` by acquiring the locks it uses internally before
the fork, and releasing them afterwards. In addition, it resets any
Expand All @@ -786,6 +796,17 @@ being held by a thread that is defunct after the fork.
:c:func:`PyOS_AfterFork_Child` tries to reset the necessary locks, but is not
always able to.

The fact that all other threads go away also means that CPython's
runtime state there must be cleaned up properly, which :func:`os.fork`
does. This means finalizing all other :c:type:`PyThreadState` objects
belonging to the current interpreter and all other
:c:type:`PyInterpreterState` objects. Due to this and the special
nature of the :ref:`"main" interpreter <sub-interpreter-support>`,
:c:func:`fork` should only be called in that interpreter's "main"
thread, where the CPython global runtime was originally initialized.
The only exception is if :c:func:`exec` will be called immediately
after.


High-level API
--------------
Expand Down
18 changes: 18 additions & 0 deletions Doc/c-api/sys.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,12 @@ Operating System Utilities
that clones the current process.
Only available on systems where :c:func:`fork` is defined.

.. warning::
The C :c:func:`fork` call should only be made from the
:ref:`"main" thread <fork-and-threads>` (of the
:ref:`"main" interpreter <sub-interpreter-support>`). The same is
true for ``PyOS_BeforeFork()``.

.. versionadded:: 3.7


Expand All @@ -44,6 +50,12 @@ Operating System Utilities
of whether process cloning was successful.
Only available on systems where :c:func:`fork` is defined.

.. warning::
The C :c:func:`fork` call should only be made from the
:ref:`"main" thread <fork-and-threads>` (of the
:ref:`"main" interpreter <sub-interpreter-support>`). The same is
true for ``PyOS_AfterFork_Parent()``.

.. versionadded:: 3.7


Expand All @@ -55,6 +67,12 @@ Operating System Utilities
any chance the process will call back into the Python interpreter.
Only available on systems where :c:func:`fork` is defined.

.. warning::
The C :c:func:`fork` call should only be made from the
:ref:`"main" thread <fork-and-threads>` (of the
:ref:`"main" interpreter <sub-interpreter-support>`). The same is
true for ``PyOS_AfterFork_Child()``.

.. versionadded:: 3.7

.. seealso::
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Provides more details about the interaction between :c:func:`fork` and
CPython's runtime, focusing just on the C-API. This includes cautions
about where :c:func:`fork` should and shouldn't be called.