Skip to content

Stash pybind11 data structures in interpreter state dictionary #4293

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 10 commits into from
Closed

Stash pybind11 data structures in interpreter state dictionary #4293

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
3b14721
stash pybind11 data structures in interpreter state dictionary
wjakob Oct 28, 2022
7a94406
style: pre-commit fixes
pre-commit-ci[bot] Oct 28, 2022
3a89f67
Make changes specific to Python 3.8+, incorporate feedback
wjakob Oct 31, 2022
daf768a
fix segfault
wjakob Oct 31, 2022
8df2e81
fix C++ test
wjakob Nov 2, 2022
f4e6c78
style: pre-commit fixes
pre-commit-ci[bot] Nov 2, 2022
cc03a56
continue to use PyEval_GetBuiltins for PyPy
wjakob Nov 2, 2022
780c6ff
incorporated feedback
wjakob Nov 2, 2022
39c21a3
style: pre-commit fixes
pre-commit-ci[bot] Nov 2, 2022
78e074b
clang-tidy fix
wjakob Nov 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 38 additions & 6 deletions include/pybind11/detail/internals.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -401,9 +401,30 @@ inline void translate_local_exception(std::exception_ptr p) {
}
#endif

inline object get_internals_state_dict() {
object state_dict;
#if PY_VERSION_HEX < 0x03080000 || defined(PYPY_VERSION)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It turns out it is extremely easy to invite the Python 3.12 core developers to test with pybind11, with a 1 or 2 line change here, depending on how you count. Diff below.

I tested that diff with a local installation of Python 3.12.0a1. Log also below.

diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h
index 03c10506..c8ac22e7 100644
--- a/include/pybind11/detail/internals.h
+++ b/include/pybind11/detail/internals.h
@@ -403,7 +403,8 @@ inline void translate_local_exception(std::exception_ptr p) {

 inline object get_internals_state_dict() {
     object state_dict;
-#if PY_VERSION_HEX < 0x03080000 || defined(PYPY_VERSION)
+#if (PYBIND11_INTERNALS_VERSION <= 4 && PY_VERSION_HEX < 0x030C0000)                              \
+    || PY_VERSION_HEX < 0x03080000 || defined(PYPY_VERSION)
     state_dict = reinterpret_borrow<object>(PyEval_GetBuiltins());
 #else
 #    if PY_VERSION_HEX < 0x03090000
+ /usr/local/google/home/rwgk/usr_local_like/Python-3.12.0a1/bin/python3 /usr/local/google/home/rwgk/clone/pybind11_scons/run_tests.py ../pybind11
Running tests in directory "/usr/local/google/home/rwgk/forked/pybind11/tests/test_embed":
===============================================================================
All tests passed (1554 assertions in 12 test cases)

Running tests in directory "/usr/local/google/home/rwgk/forked/pybind11/tests":
=========================================================== test session starts ============================================================
platform linux -- Python 3.12.0a1, pytest-7.2.0, pluggy-1.0.0
C++ Info: Debian Clang 14.0.6 C++17 __pybind11_internals_v4_clang_libstdcpp_cxxabi1002__
rootdir: /usr/local/google/home/rwgk/forked/pybind11/tests, configfile: pytest.ini
collected 698 items

test_async.py ..                                                                                                                     [  0%]
test_buffers.py .........                                                                                                            [  1%]
test_builtin_casters.py ....................                                                                                         [  4%]
test_call_policies.py ........                                                                                                       [  5%]
test_callbacks.py ............                                                                                                       [  7%]
test_chrono.py ...........................................                                                                           [ 13%]
test_class.py ...............................                                                                                        [ 17%]
test_const_name.py ......................                                                                                            [ 21%]
test_constants_and_functions.py .....                                                                                                [ 21%]
test_copy_move.py ........                                                                                                           [ 22%]
test_custom_type_casters.py ...                                                                                                      [ 23%]
test_custom_type_setup.py ..                                                                                                         [ 23%]
test_docstring_options.py .                                                                                                          [ 23%]
test_eigen_matrix.py .....................ss.......                                                                                  [ 28%]
test_eigen_tensor.py .............................................................................................................   [ 43%]
test_enum.py .........                                                                                                               [ 44%]
test_eval.py ....                                                                                                                    [ 45%]
test_exceptions.py ......................                                                                                            [ 48%]
test_factory_constructors.py ...............                                                                                         [ 50%]
test_gil_scoped.py .....                                                                                                             [ 51%]
test_iostream.py ......................                                                                                              [ 54%]
test_kwargs_and_defaults.py ........                                                                                                 [ 55%]
test_local_bindings.py ..........                                                                                                    [ 57%]
test_methods_and_attributes.py ......................                                                                                [ 60%]
test_modules.py .......                                                                                                              [ 61%]
test_multiple_inheritance.py ..................                                                                                      [ 64%]
test_numpy_array.py .........................................................                                                        [ 72%]
test_numpy_dtypes.py ...............                                                                                                 [ 74%]
test_numpy_vectorize.py ........                                                                                                     [ 75%]
test_opaque_types.py ...                                                                                                             [ 75%]
test_operator_overloading.py .....                                                                                                   [ 76%]
test_pickling.py ........                                                                                                            [ 77%]
test_pytypes.py ..................................................................................                                   [ 89%]
test_sequences_and_iterators.py ..............                                                                                       [ 91%]
test_smart_ptr.py .............                                                                                                      [ 93%]
test_stl.py .........s.............                                                                                                  [ 96%]
test_stl_binders.py .........                                                                                                        [ 97%]
test_tagbased_polymorphic.py .                                                                                                       [ 98%]
test_thread.py ..                                                                                                                    [ 98%]
test_union.py .                                                                                                                      [ 98%]
test_virtual_functions.py ..........                                                                                                 [100%]

========================================================= short test summary info ==========================================================
SKIPPED [1] test_eigen_matrix.py:718: could not import 'scipy': No module named 'scipy'
SKIPPED [1] test_eigen_matrix.py:728: could not import 'scipy': No module named 'scipy'
SKIPPED [1] test_stl.py:143: no <experimental/optional>
===================================================== 695 passed, 3 skipped in 10.04s ======================================================

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also triggered CI testing under #4307, after git rebase master.

state_dict = reinterpret_borrow<object>(PyEval_GetBuiltins());
#else
# if PY_VERSION_HEX < 0x03090000
PyInterpreterState *istate = _PyInterpreterState_Get();
# else
PyInterpreterState *istate = PyInterpreterState_Get();
# endif
if (istate) {
state_dict = reinterpret_borrow<object>(PyInterpreterState_GetDict(istate));
}
#endif
if (!state_dict) {
raise_from(PyExc_SystemError, "get_internals(): could not acquire state dictionary!");
}

return state_dict;
}

/// Return a reference to the current `internals` data
PYBIND11_NOINLINE internals &get_internals() {
auto **&internals_pp = get_internals_pp();
internals **&internals_pp = get_internals_pp();
if (internals_pp && *internals_pp) {
return **internals_pp;
}
Expand All @@ -419,11 +440,22 @@ PYBIND11_NOINLINE internals &get_internals() {
} gil;
error_scope err_scope;

PYBIND11_STR_TYPE id(PYBIND11_INTERNALS_ID);
auto builtins = handle(PyEval_GetBuiltins());
if (builtins.contains(id) && isinstance<capsule>(builtins[id])) {
internals_pp = static_cast<internals **>(capsule(builtins[id]));
constexpr const char *id_cstr = PYBIND11_INTERNALS_ID;
str id(id_cstr);

dict state_dict = get_internals_state_dict();

if (state_dict.contains(id_cstr)) {
object o = state_dict[id];
// May fail if 'capsule_obj' is not a capsule, or if it has a different
// name. We clear the error status below in that case
internals_pp = static_cast<internals **>(PyCapsule_GetPointer(o.ptr(), id_cstr));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the raw CAPI here? Any particular reason we are changing it from pytype.h API it was before?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is my current version:

        void *raw_ptr = PyCapsule_GetPointer(state_dict[id].ptr(), id_cstr);
        if (raw_ptr == nullptr) {
            raise_from(
                PyExc_SystemError,
                "pybind11::detail::get_internals(): Retrieve internals** from capsule FAILED");
        }
        internals_pp = static_cast<internals **>(raw_ptr);

The nice thing is that PyCapsule_GetPointer() does everything "just right" in one simple line. Additionally, in this particular situation I'd definitely want to avoid the throw PYBIND11_OBJECT_CHECK_FAILED() that comes with the capsule constructor.

I don't have a good idea for using capsule in an elegant way here TBH.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While looking around more (work under PR #4329) I noticed this code in embed.h (finalize_interpreter()):

    handle builtins(PyEval_GetBuiltins());
    ...
    if (builtins.contains(id) && isinstance<capsule>(builtins[id])) {
        internals_ptr_ptr = capsule(builtins[id]);
    }

Two things learned:

  • That needs to be changed, too, to inspect the correct dict.
  • The implementation is a bit on the high-level overkill side, scraping by an if (...) throw, but it is elegant!

if (!internals_pp) {
PyErr_Clear();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR (with the #4307 tweak) is now baked into PR #4329:

c2e6b38

Copying the commit message here (I think the link will stop working in case I have to rebase):

Modified version of PR #4293 by @wjakob

Modifications are:

* Backward compatibility (no ABI break), as originally under PR #4307.
* Naming: `get_python_state_dict()`, `has_pybind11_internals_capsule()`
* Report error retrieving `internals**` from capsule instead of clearing it.

Locally tested with ASAN, MSAN, TSAN, UBSAN (Google-internal toolchain).

My commit changes the code here, to report the error rather than suppressing it.

What is the rationale for suppressing it?

}
}

if (internals_pp && *internals_pp) {
// We loaded builtins through python's builtins, which means that our `error_already_set`
// and `builtin_exception` may be different local classes than the ones set up in the
// initial exception translator, below, so add another for our local exception classes.
Expand Down Expand Up @@ -459,7 +491,7 @@ PYBIND11_NOINLINE internals &get_internals() {
# endif
internals_ptr->istate = tstate->interp;
#endif
builtins[id] = capsule(internals_pp);
state_dict[id] = capsule(internals_pp, id_cstr);
internals_ptr->registered_exception_translators.push_front(&translate_exception);
internals_ptr->static_property_type = make_static_property_type();
internals_ptr->default_metaclass = make_default_metaclass();
Expand Down
19 changes: 9 additions & 10 deletions tests/test_embed/test_interpreter.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,9 +168,8 @@ TEST_CASE("There can be only one interpreter") {
py::initialize_interpreter();
}

bool has_pybind11_internals_builtin() {
auto builtins = py::handle(PyEval_GetBuiltins());
return builtins.contains(PYBIND11_INTERNALS_ID);
bool has_pybind11_internals_state_dict() {
return py::detail::get_internals_state_dict().contains(PYBIND11_INTERNALS_ID);
};

bool has_pybind11_internals_static() {
Expand All @@ -181,7 +180,7 @@ bool has_pybind11_internals_static() {
TEST_CASE("Restart the interpreter") {
// Verify pre-restart state.
REQUIRE(py::module_::import("widget_module").attr("add")(1, 2).cast<int>() == 3);
REQUIRE(has_pybind11_internals_builtin());
REQUIRE(has_pybind11_internals_state_dict());
REQUIRE(has_pybind11_internals_static());
REQUIRE(py::module_::import("external_module").attr("A")(123).attr("value").cast<int>()
== 123);
Expand All @@ -198,10 +197,10 @@ TEST_CASE("Restart the interpreter") {
REQUIRE(Py_IsInitialized() == 1);

// Internals are deleted after a restart.
REQUIRE_FALSE(has_pybind11_internals_builtin());
REQUIRE_FALSE(has_pybind11_internals_state_dict());
REQUIRE_FALSE(has_pybind11_internals_static());
pybind11::detail::get_internals();
REQUIRE(has_pybind11_internals_builtin());
REQUIRE(has_pybind11_internals_state_dict());
REQUIRE(has_pybind11_internals_static());
REQUIRE(reinterpret_cast<uintptr_t>(*py::detail::get_internals_pp())
== py::module_::import("external_module").attr("internals_at")().cast<uintptr_t>());
Expand All @@ -216,13 +215,13 @@ TEST_CASE("Restart the interpreter") {
py::detail::get_internals();
*static_cast<bool *>(ran) = true;
});
REQUIRE_FALSE(has_pybind11_internals_builtin());
REQUIRE_FALSE(has_pybind11_internals_state_dict());
REQUIRE_FALSE(has_pybind11_internals_static());
REQUIRE_FALSE(ran);
py::finalize_interpreter();
REQUIRE(ran);
py::initialize_interpreter();
REQUIRE_FALSE(has_pybind11_internals_builtin());
REQUIRE_FALSE(has_pybind11_internals_state_dict());
REQUIRE_FALSE(has_pybind11_internals_static());

// C++ modules can be reloaded.
Expand All @@ -244,7 +243,7 @@ TEST_CASE("Subinterpreter") {

REQUIRE(m.attr("add")(1, 2).cast<int>() == 3);
}
REQUIRE(has_pybind11_internals_builtin());
REQUIRE(has_pybind11_internals_state_dict());
REQUIRE(has_pybind11_internals_static());

/// Create and switch to a subinterpreter.
Expand All @@ -254,7 +253,7 @@ TEST_CASE("Subinterpreter") {
// Subinterpreters get their own copy of builtins. detail::get_internals() still
// works by returning from the static variable, i.e. all interpreters share a single
// global pybind11::internals;
REQUIRE_FALSE(has_pybind11_internals_builtin());
REQUIRE_FALSE(has_pybind11_internals_state_dict());
REQUIRE(has_pybind11_internals_static());

// Modules tags should be gone.
Expand Down