[C API] Make Py_TYPE() opaque in limited C API 3.14

[vstinner]

In the limited C API 3.14 and newer, I propose to change Py_TYPE() and Py_SET_TYPE() implementation to opaque function calls to hide implementation details. I made a similar change for Py_REFCNT() and Py_SET_REFCNT() in Python 3.12.

The problem is that with Free Threading (PEP 703), the implementation of these functions become less trivial than just getting/setting an object member:

static inline PyTypeObject* Py_TYPE(PyObject *ob) {
    return (PyTypeObject *)_Py_atomic_load_ptr_relaxed(&ob->ob_type);
}

static inline void Py_SET_TYPE(PyObject *ob, PyTypeObject *type) {
    _Py_atomic_store_ptr(&ob->ob_type, type);
}

_Py_atomic_load_ptr_relaxed() and _Py_atomic_store_ptr() must now be called. But I would prefer to not "leak" such implementation detail into the limited C API.

cc @colesbury @Fidget-Spinner

Linked PRs

• gh-120600: Make Py_TYPE() opaque in limited C API 3.14 #120601

[encukou]

I'd prefer if Py_TYPE was a public exported function. If it remains a macro, non-C languages will still need to access ob_type directly (or start using private API).

As for Py_SET_TYPE, you need knowledge of the internals to use it correctly. I support changing it to call an opaque function, but unlike Py_TYPE I'd prefer not exposing any more than it is (so: keep it as a macro).
The proper way to change a type is currently setting the __class__ attribute. We could add a new API function, like PyObject_SetType, that does the required checks and updates. And then deprecate Py_SET_TYPE.

Please add tests for the old implementation (getting/setting ob_type directly), to ensure existing extensions aren't unintentionally broken.

[vstinner]

Please add tests for the old implementation (getting/setting ob_type directly), to ensure existing extensions aren't unintentionally broken.

_testcapi has test_set_type_size() which tests Py_TYPE() and Py_SET_TYPE():

    // Ensure that following tests don't modify the object,
    // to ensure that Py_DECREF() will not crash.
    assert(Py_TYPE(obj) == &PyList_Type);
    assert(Py_SIZE(obj) == 0);

    // bpo-39573: Test Py_SET_TYPE() and Py_SET_SIZE() functions.
    Py_SET_TYPE(obj, &PyList_Type);
    Py_SET_SIZE(obj, 0);

[vstinner]

I modified my PR to only reuse Py_TYPE() name (instead of adding a _Py_TYPE() function) and to leave Py_SET_TYPE() unchanged.

[encukou]

Sounds good, thanks!

I don't think it'll solve thread safety -- Py_TYPE returns a borrowed reference, and something could invalidate that reference while you're using it. But for the use cases where Py_TYPE is already safe, it's nice to have functions rather than macros in the limited API.

[vstinner]

I don't think it'll solve thread safety -- Py_TYPE returns a borrowed reference

PyObject_Type(obj) already exists to get a strong reference.

I also wrote PEP 737 "C API to format a type fully qualified name" to avoid a common usage of Py_TYPE() borrowed reference. (The old usage can lead to a crash, see the PEP!)

[vstinner]

Related change done in Python 3.12: Limited C API: implement Py_INCREF() and Py_DECREF() as function calls.

[nineteendo]

Is there something left to do here?

[vstinner]

Fixed by change 16f8e22.

[encukou]

There's still Py_SET_TYPE, which IMO could use a private opaque function.
Should that be done with a new issue?

[vstinner]

There's still Py_SET_TYPE, which IMO could use a private opaque function. Should that be done with a new issue?

I suggest to open a new issue. If I understood correctly, Py_SET_TYPE() should not exist in the limited C API, since static types cannot be defined with the limited C API.

[vstinner]

I created an issue for Py_REFCNT(): #124127