Support for sub-interpreters

[b-pass]

Description

This PR add the ability for pybind11 modules to support subinterpreters. This support requires 2 things:

1. Multiphase init (implemented in feat: change PYBIND11_MODULE to use multi-phase init (PEP 489) #5574)
2. internals and local_internals have to have an instance per-interpreter (can no longer be static singletons), which is the primary subject of this PR.

Multiphase init

The PR adds tags which can be passed as the 3rd argument to the PYBIND11_MODULE definition macro (in addition to the existing mod_gil_not_used). If the tags are not specified, the module is defined without multiple interpreter support (as they are today).

If multiple interpreters is example, then when a module is imported a second time in a sub-interpreter, the module's exec slot is run again. For pybind11 this means the user's the module init function is re-run in the sub-interpreter. That's good, because the sub-interpreter needs it's own type_info for all of the bindings.

internals

This presents the problem that the place that pybind11 stores these is currently a singleton. But sub-interpreters need this state to be per-interpreter. That means the correct instance (for the current active interpreter) needs to be retrieved from the interpreter state dict. Fortunately, the internals pointer-to-pointer is already stored in the state dict.

In order to minimize performance costs, we can detect whether or not multi-subinterpreters are present by counting how many times the module has been imported. If it has only been imported once then it can only possibly have one internals (even if there are other sub-interpreters where it was not imported). When it has been imported more times, then we need to do additional work to make sure the right internals object is used (the one associated with the current interpreter in the current thread). We can switch between these two cases with a single simple branch, thus causing minimal performance overhead for existing code.

In the multi-interpreter case we would still like to minimized the cost of accessing internals, we don't want to have to reach into the interpreter state dict every time. So we cache the value in a thread_local along with the pointer to the PyInterpreterState to which it belongs. This means that the slow path (acquiring the GIL, doing a dict lookup, etc) is only done when the active PyInterpreterState changes (or the first time get_internals is called in an OS thread). So the fast path merely checks that the PyInterpreterState hasn't changed, and then returns the previously looked up value.

local_internals also has to change in a similar way to internals.

Both of these now share some templated code to manage their thread-local cache and the state-dict cache. Some code around these has also been cleaned up (like moving most of the internals initialization into its constructor).

Memory management / Future work

This PR does not add support for creating / deleting / switching between sub-interpreters.

In embed, pybind11 only cleans up the internals and local_interals associated with the main interpreter (when it is finalized). Since it doesn't currently manage any subinterpreters it can't clean up after them.

Suggested changelog entry:

* Support for sub-interpreters (both isolated (with separate GILs) and legacy (with a global GIL). Add the ``py::multiple_interpreters::per_interpreter_gil()`` tag (or, ``py::multiple_interpreters::shared_gil()`` for legacy interpreter support) to ``PYBIND11_MODULE`` calls (as the third parameter) to indicate that a module supports running with sub-interpreters.

The guide needs to add a short mention of py::multiple_interpreters::per_interpreter_gil() and py::multiple_interpreters::shared_gil() tags.

[henryiii]

Very excited to see this! I have a couple of comments/questions:

• What happens if someone doesn't define PYBIND11_SUBINTERPRETER_SUPPORT but tries to use the new tags? I would think you'd want a compile-time error?
• Do you know how much of an impact PYBIND11_SUBINTERPRETER_SUPPORT has? Just curious what constitutes "small". Having a compile time opt-in is a little unfortunate, but having it as a default-off option sounds safe for now, though.
• I believe communication between extensions even with different compile definitions is still supported due to @rwgk's interface work.

We'll need some docs, too. Maybe we should do a full test run with the define on, too.

[rwgk]

This is a complex PR. I need to find more time later for a full review.

High-level questions:

• Could it be useful to split this PR: 1. multi-phase init only. 2. multi-interpreter support? — That would make it easier to do the reviews now, and understand the development steps in the future. It might also help us dealing with bugs after this change is released.
• Are you still working on a new Python tests, to exercise the new multi-interpreter support?
• Is there a potential for bug or feature interference between free-threading and multi-interpreter functionality? — I think we'll need tests for all combinations of (free-threading on/off) x (multi-interpreter support on/off); not for all platforms, but maybe one each: Linux, macOS, Windows.

[b-pass]

* Could it be useful to split this PR: 1. multi-phase init only. 2. multi-interpreter support? — That would make it easier to do the reviews now, and understand the development steps in the future. It might also help us dealing with bugs after this change is released.

Sure. I have created #5574 for multi-phase init only. I will keep updating this PR for sub-interpreter support, and will remove the multiphase init from this branch shortly.

* Are you still working on a new Python tests, to exercise the new multi-interpreter support?

I'm not currently, but I can add a few more after/along with some additional changes from comments.

* Is there a potential for bug or feature interference between free-threading and multi-interpreter functionality? — I think we'll need tests for all combinations of (free-threading on/off) x (multi-interpreter support on/off); not for all platforms, but maybe one each: Linux, macOS, Windows.

No one would ever say less testing :) I think the potential problems are small. While they have similar goals, free threading and own-gil-sub-interpreters are fairly different and could be used together. Sub-interpreters were originally created (I think) to offer sandboxing features. So even with free threading, the idea that a module is used in two different sandboxes is still valid, and it still needs separation for each instance.

Also I considered whether free-threading is a superset of sub-interpreter support. I think it is not, for the same reason that the two have slightly different implications for a module. With free threading (only) it is perfectly reasonable to have, for example, a global/static atomic variable. With sub-interpreters that is probably incorrect, because each sub-interpreter should have its own separate state. However, if a module is free-threading safe (so, thread safe) and it is multi-interpreter aware (as no global state) then it should also be own-gil-sub-interpreter safe... that is, it doesn't need GIL synchronization across the many multiple subinterpreter states, which must be true or it would not be free-threading safe.

[wjakob]

My 2cts: TLS in shared libraries is real disaster (especially C++ thread_local involving nontrivial types). The following writeup is enlightening: https://yosefk.com/blog/cxx-thread-local-storage-performance.html. FWIW I saw significant perf gains in nanobind by basically removing all use of TLS except for a handful of rare/complex cases.

So adding more TLS to the internals data structure in general sounds like a pretty significant performance sink. I would encourage you to thoroughly benchmark function/method calling on Windows/macOS/Linux to see how bad this is, and to what extent these costs can be mitigated.

[henryiii]

FYI, this is a perfect example of where I'd personally always rebase and force push. ;)

[b-pass]

My 2cts: TLS in shared libraries is real disaster (especially C++ thread_local involving nontrivial types). The following writeup is enlightening: https://yosefk.com/blog/cxx-thread-local-storage-performance.html. FWIW I saw significant perf gains in nanobind by basically removing all use of TLS except for a handful of rare/complex cases.

Luckily, these are all zero-initialized pointers, so no constructors. And I think multiple interpreters in multiple threads a the same time might qualify as a complex case. Still, point taken, on this expert advice I have made a bunch of changes to get the thread_local s completely out of the path of code that isn't running multiple interpreters. It takes just one branch to do so, so that's pretty much the best we could hope for performance wise.

Unfortunately, in the multi-interpreter case there really isn't any choice but to use thread_local. The TLS is there to avoid an even more costly dict lookup in the interpreter state dict.

[b-pass]

I would encourage you to thoroughly benchmark function/method calling on Windows/macOS/Linux to see how bad this is, and to what extent these costs can be mitigated.

Looks like, with the rewrite, the cost is an extra 0.22ns per call to get_details (an increase of about 15%). IMO that is a very small cost, unlikely any real-world usage will notice it.

The cost is much more significant when a subinterpreter is actually created, the cost to access the internals triples. In my opinion, that's just the cost of using subinterpreters.

... I've added this information to the PR description.

[XuehaiPan]

Hi, I wonder if you have read these:

• Isolate Stdlib Extension Modules python/cpython#103092
• PEP 489 – Multi-phase extension module initialization
• PEP 573 – Module State Access from C Extension Methods
• PEP 630 – Isolating Extension Modules & Isolating Extension Modules HOWTO
• PEP 687 – Isolating modules in the standard library

To support sub-interpreters, I think we also need to implement PEP 573 and PEP 630. To clarify:

• Managing Per-Module State

  A new module-level state needs to be added. All types defined by py::class_ need to be heap types and need to be managed by the module state. Users also need to implement m_{traverse,clear,free} for the state and these classes. See gh-103092: Isolate _collections python/cpython#103093 as an example.

• Module State Access from Functions & Module State Access from Classes & Module State Access from Regular Methods

• Changing Static Types to Heap Types

---

I think it would be really hard to make PYBIND11_MODULE to both support module isolation while also keeping backward compatibility. Maybe we should add a new API PYBIND11_ISOLATED_MODULE. Or Opt-Out: Limiting to One Module Object per Process.

[b-pass]

To support sub-interpreters, I think we also need to implement PEP 573 and PEP 630.

The goal of these is to get rid of global state, and replace it with state that is tied to the module instance. While implementing this according to the python guides will definitely accomplish that goal -- and that might be the best way to do it --, that is not the only way to accomplish it. I think strict adherence to these would require major rewrites of several parts of pybind11, but I don't think that is necessary to support sub-interpreters/multi-interpreters. Definitely following PEP573 would make a module work with sub-interpreters. It is a sufficient condition but not a necessary condition.

Pybind11's global state is entirely contained within internals and local_internals. This PR makes these into per-interpreter state which is similar to how they behave today (when there is only one interpreter). But not into per-module state (how python suggests that state be managed). The existing implementation with global static state isn't compliant with PEP 573, this PR does not get pybind11 any closer to that either. Pybind11 has some problems that PEP 573 would solve already, and this PR doesn't fix them but it doesn't introduce any new issues AFAIK.

* [Changing Static Types to Heap Types](https://docs.python.org/3.13/howto/isolating-extensions.html#changing-static-types-to-heap-types)

I don't think this is required, since Pybind11's types are managed by its internals structures, they already are not globally static in the strict sense. Converting them to use Type_Spec and Slots is IMO unrelated to this PR.

(Edit: or, maybe pybind11 is already doing heap types? At least, some of them are...)

I think it would be really hard to make PYBIND11_MODULE to both support module isolation while also keeping backward compatibility.

I agree that is probably impossible. My goal here isn't full module isolation, pybind11 already doesn't have module isolation and it can't be added. That doesn't mean it can't support sub-interpreters. Maybe another way to think about it is that this PR adds interpreter isolation without adding module isolation. The examples you linked explain the kinds of problems that non-isolated modules have, which existing pybind11 modules all have, and they would continue to have after this PR.

[henryiii]

I've run @wjakob's benchmarks for nanobind on this PR. The binary size might be a bit off, since I can't run strip due to needing undefined symbols, but I don't think that has any affect here anyway. I couldn't get nanobind to load, so I had to take it off the runtime plot.

This seems to have a noticeable impact on debug (unoptimized) performance, but not really noticeable on runtime, probably within the uncertainty margins. I'd love for the runtime cost to go down (there's an old PR that did that, but not usable anymore) instead of up, but this looks acceptable to avoid complications building.

[henryiii]

I'd make module_options a namespace, no need for chained (). And we could use "mod", since that's the CPython term for it. So py::mod::multiple_interpreters() (or py::mod::multiple_interpeters_supported(), to exactly match the CPython API term).

[rwgk]

I'd make module_options a namespace, no need for chained (). And we could use "mod", since that's the CPython term for it. So py::mod::multiple_interpreters() (or py::mod::multiple_interpeters_supported(), to exactly match the CPython API term).

Sound good, but a question for @b-pass: are all combinations of options valid?

I think we should make sure we can clearly diagnosing invalid combinations.

[rwgk]

I just reproduced the error:

git switch master
git pull
git switch subinterpreters
git pull
git merge master

Then using the same commands as reported before, I'm still getting the same error.

This should be very easy to reproduce, I have a standard Ubuntu 24.04 x86_64 workstation.

[b-pass]

On Ubuntu 24.04 (x86-64), I did the below, could not reproduce the problem.

(on this PR branch)
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -r tests/requirements.txt
$ mkdir build
$ cmake -S . -B build -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON  -DCMAKE_CXX_STANDARD=17 -DCMAKE_BUILD_TYPE=Debug -DPYBIND11_PYTEST_ARGS=-vv

Output:

-- The CXX compiler identification is GNU 13.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- pybind11 v3.0.0 dev1
-- CMake 3.30.5
-- Found Python: /home/user/Code/pybind11/venv/bin/python3 (found suitable version "3.12.3", minimum required is "3.8") found components: Interpreter Development.Module Development.Embed
-- Python 3.12.3
Using compatibility mode for Python, set PYBIND11_FINDPYTHON to NEW/OLD to silence this message
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- pybind11::lto enabled
-- pybind11::thin_lto enabled
Cloning into 'eigen-src'...
error: 49641 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
Cloning into 'eigen-src'...
Had to git clone more than once: 2 times.
HEAD is now at 929bc0e19 Fix alias violation in BFloat16
-- Building tests with Eigen v3.4.0
-- Found pytest 8.3.5
-- Downloading catch v2.13.10...
-- Building pure C++ tests (not depending on Python) using Catch v2.13.10
-- Building interpreter tests using Catch v2.13.10
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Configuring done (9.6s)
-- Generating done (0.0s)
-- Build files have been written to: /home/user/Code/pybind11/build

$ cd build
$ make -j8 check

I also tried with a git rebase upstream/master and it was also successful.

[b-pass]

Ok, I got it to crash for me after running test_embed in a loop for a while.....

[rwgk]

Ok, I got it to crash for me after running test_embed in a loop for a while.....

Thanks for confirming! — I was beginning to develop doubts about the health of my Ubuntu workstation.

(So sad we don't have any sanitizers at the moment...)

[rwgk]

I merged your commit 22968e9 locally: unfortunately I'm still seeing the error.

[rwgk]

@henryiii Do you know how easy or difficult it is to build the embedding test with -fsanitize=address?

[b-pass]

@rwgk OK, I think it's fixed in 89e9599 . The problem was concurrent threads importing the same module at the same time overwrote the (global, static) module_def and slots, which occasionally would stomp on each other before their init was finished. Fixed (in detail/common.h PYBIND11_MODULE) by wrapping the init with a static so it only happens once and is atomic. (Once it is init'd, it that part does not need to be done again in order for the module to be used in other interpreters.)

I might try to do -fsanitize=address in a future PR (no promises!).....

I'd make module_options a namespace, no need for chained (). And we could use "mod", since that's the CPython term for it. So py::mod::multiple_interpreters() (or py::mod::multiple_interpeters_supported(), to exactly match the CPython API term).

Sound good, but a question for @b-pass: are all combinations of options valid?

gil_not_used can be mixed with the multiple interpreter options. Only one of the multiple_interpreters options can be specified ... per_interpreter_gil is more constrained than shared_gil, so in the presence of both it is safe to report per_interpreter_gil.

So is the suggestion to change the current py::multiple_interpreters::per_interpreter_gil()/py::multiple_interpreters::shared_gil() to py::mod::multiple_interpreters_per_interpreter_gil() / py::mod::multiple_interpreters_shared_gil()? Unfortunately, I don't think the CPython naming is very clear .. the option is named Py_mod_multiple_interpreters but the values are Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED (old/legacy/shared gil) and Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (new/default/multi gil).

[henryiii]

Personally, I'm fine with the current naming, which makes it clearer that you can pick one multiple_interperters option. Otherwise, it would be py::mod::multiple_interpreters() and py::mod::per_interpreter_gil(), I think. Optionally we could add the _supported.

[b-pass]

Ok, let's stick with the current implementation then. I added a PR for docs, #5659.

[henryiii]

@rwgk if the error has gone away and you're OK with the naming, then I think we're good to go.

[rwgk]

@rwgk if the error has gone away and you're OK with the naming, then I think we're good to go.

I still see the error :-(

I tried a couple times from scratch.

I'll click the Update branch button here, which will switch us over to the 24.04 runners. Let's see if the error shows up in the CI, too.

[rwgk]

I ran this:

#!/bin/bash
set -x
for i in {1..100}; do
    (cd /home/rgrossekunst/bld/cmake_build/tests/test_embed && /home/rgrossekunst/bld/cmake_build/tests/test_embed/test_embed)
done

With this outcome:

$ grep '1514 passed | 1 failed' ~/log_run100_2025-05-13+2057.txt | wc -l
100

I.e. it fails every single time.

IIUC, it only fails sometimes for @b-pass?

It's a mystery to me what's different on my machine.

[rwgk]

Again with MinSizeRel all else equal:

rm -rf cmake_build
cmake -S $HOME/forked/pybind11 -B cmake_build -DCMAKE_VERBOSE_MAKEFILE=ON -DDOWNLOAD_CATCH=ON -DPYTHON_EXECUTABLE=$(which python3) -DCMAKE_CXX_STANDARD=17 -DCMAKE_BUILD_TYPE=MinSizeRel -DPYBIND11_PYTEST_ARGS=-vv

0db6015-lcedt.nvidia.com:~/forked/pybind11 $ grep '1514 passed | 1 failed' ~/log_run100_2025-05-13+2107.txt | wc -l
100

Not a single success, again.

[rwgk]

This crossed my mind (PYTHONPATH):

#!/bin/bash
set -x
unset PYTHONPATH
printenv | grep PYTHONPATH
for i in {1..100}; do
    (cd /home/rgrossekunst/bld/cmake_build/tests/test_embed && /home/rgrossekunst/bld/cmake_build/tests/test_embed/test_embed)
done

But still:

$ grep '1514 passed | 1 failed' ~/log_run100_2025-05-13+2154.txt | wc -l
100

(I'm using a python in a venv, I'll try to eliminate that, too.)

[rwgk]

After resetting my PATH to /usr/sbin:/usr/bin:/sbin:/bin the error goes away!

I rebuilt from scratch after resetting PATH.

$ grep 'All tests passed ' ~/log_run100_2025-05-13+2204.txt | wc -l
100

I have to figure out what on my regular PATH makes it upset. Not sure if I can still do that tonight.

[henryiii]

Did that change which version of Python was being picked up? It only affects newer Pythons.

[rwgk]

Did that change which version of Python was being picked up? It only affects newer Pythons.

Yes and no.

Let's call this PathA:

export PATH=/home/rgrossekunst/venvs/0db6015-lcedt/base/bin:/usr/sbin:/usr/bin:/sbin:/bin

This is PathB:

export PATH=/usr/sbin:/usr/bin:/sbin:/bin

For completeness: In what follows, it does NOT matter if I had PathA or PathB while running the cmake and make commands.

But at runtime:

• test_embed fails if PathA is active.
• test_embed passes if PathB is active.

I can switch back and forth, the behavior is 100% deterministic.

Interestingly (but as I expected):

$ export PATH=/usr/sbin:/usr/bin:/sbin:/bin
$ which python
/usr/bin/python
$ realpath "$(which python)"
/usr/bin/python3.12
$ export PATH=/home/rgrossekunst/venvs/0db6015-lcedt/base/bin:/usr/sbin:/usr/bin:/sbin:/bin
$ which python
/home/rgrossekunst/venvs/0db6015-lcedt/base/bin/python
$ realpath "$(which python)"
/usr/bin/python3.12

Open question: How can that lead to the test failure?

[rwgk]

I'm getting the same behavior when using the master branch!

Note: Usually I run my tests with a scons-based build. With that, tests pass for master and this PR.

This is a bit twisty: I had to tweak my scons files to make it work for this PR. To avoid that before, I chose to simply use the cmake build instead. I totally didn't expect that to fail on master. I ran the cmake-based tests many times in the past, and it used to work, but it's only fairly recently that I started using a venv in my log-in environment.

But back to the cmake build:

This fails as reported before:

$ echo $PATH
/home/rgrossekunst/venvs/0db6015-lcedt/base/bin:/usr/sbin:/usr/bin:/sbin:/bin
$ (cd /home/rgrossekunst/bld/cmake_build/tests/test_embed && /home/rgrossekunst/bld/cmake_build/tests/test_embed/test_embed)

These work, everything else exactly equal:

$ (cd /home/rgrossekunst/bld/cmake_build/tests/test_embed && PATH=/usr/bin /home/rgrossekunst/bld/cma
ke_build/tests/test_embed/test_embed)
===============================================================================
All tests passed (1581 assertions in 19 test cases)

$ (cd /home/rgrossekunst/bld/cmake_build/tests/test_embed && PATH= /home/rgrossekunst/bld/cmake_build
/tests/test_embed/test_embed)
===============================================================================
All tests passed (1581 assertions in 19 test cases)

I don't know what to do about this, but it's pretty much certain now that the weird error has nothing to do with this PR.

[rwgk]

@b-pass wrote:

gil_not_used can be mixed with the multiple interpreter options.

OK, then my suggestion with the chaining doesn't make sense, and the current naming is fine.

@henryiii wrote:

if the error has gone away and you're OK with the naming, then I think we're good to go.

Awesome, I'll merge this now.

Ideally we should do something about the weird test_embed behavior, but certainly not in this PR.

[henryiii]

There's something wrong with the embed tests in CMake. I'll have to try to work it out, but I can't get them to pass locally, haven't for a while. But they work on many of our CI jobs (a few don't have the requirements present to run them).

[rwgk]

There's something wrong with the embed tests in CMake. I'll have to try to work it out, but I can't get them to pass locally, haven't for a while. But they work on many of our CI jobs (a few don't have the requirements present to run them).

I don't think it's cmake specific (maybe you didn't mean that, but JIC, to clarify). My scons-based test just happens to work interactively because I'm doing this (read that code from the bottom up):

  test_embed_dirpath = os.path.join(tests_dirpath, "test_embed")                
  test_embed, list_of_test_py = build_list_of_tests(tests_dirpath, substrings)  
  env = {"PYTHONPATH": normabspath("lib")}                                      
  if test_embed:                                                                
    bin_test_embed = normabspath("bin/test_embed")                              
    print('(cd "%s" && PYTHONPATH="%s" "%s")' % (                               
        test_embed_dirpath, env["PYTHONPATH"], bin_test_embed))                 
    sys.stdout.flush()                                                          
    subprocess.call([bin_test_embed], cwd=test_embed_dirpath, env=env)          

What I didn't realize in the past: passing in env like this means PATH is not set.

I believe if you change the cmake files to do this

(cd /home/rgrossekunst/bld/cmake_build/tests/test_embed && PATH= /home/rgrossekunst/bld/cmake_build
/tests/test_embed/test_embed)

local testing will pass for you, too.