Support sub-interpreters

[acminor]

Does pyo3 allow the use case of starting multiple separate interpreters? This would be similar to Python's multiprocessing.

[reem]

Specifically would be amazing if it was possible to create multiple separate python interpreters that could be run on different threads in parallel, but which share the same memory space (with the type system used to ensure this is only observable from rust).

[davidhewitt]

The complexity with sub-interpreters is that no Python objects should ever be shared between them. The pyO3 API isn't set up to prevent this at the moment, so it's going to take a fair whack of experimentation before we get anything stable in place.

[clouds56]

I'd like to have multiple thread, each one has an interpreter. No PyObject would be send between thread.
IIRC, we couldn't hold a PyObject in rust, everything is ref to UnsafeCell and by default not send.

[mohitreddy1996]

@davidhewitt with https://peps.python.org/pep-0684/, using sub interpreters or multiple interpreters to unlock true multi-core parallelism becomes possible.

Is adding support for this in pyo3 timelines or consideration?

[AnatolyBuga]

@davidhewitt with https://peps.python.org/pep-0684/, using sub interpreters or multiple interpreters to unlock true multi-core parallelism becomes possible.

Is adding support for this in pyo3 timelines or consideration?

Found this interesting article on current usage of sub interpreters in python (no rust there)

[davidhewitt]

We are very aware of the per-intepreter parallelism landing in Python 3.12. There are significant changes which need to happen to PyO3's current implementation to support this correctly. We have been discussing some of these challenges in multiple discussions across this repo, such as #2885 which looks at the possible nogil option.

There are several main issues which are prominent in my mind, although others may exist:

• I understand interpreters cannot share Python objects. This implies that Py<T> needs to be removed or reworked significantly, maybe by removing Send and Sync from that type, probably also by somehow making the operation to attach Py<T> to a Python thread to be unsafe or runtime-checked in some way.
• We need to fully transition PyO3 to PEP 630 compatibility, which requires elimination of all static data which contains Python state. This is probably linked to the first bullet.
  • APIs like GILOnceCell and GILProtected can no longer be Sync if multiple GILs exist. Transition to PEP 630 compatibility will probably force us to replace these types with alternative solutions.

Solving these problems is likely to create significant churn of PyO3's API, so we can only make progress once someone has proposed a relatively complete solution which we can adopt with a suitable migration path for users.