Make `struct _typeobject` a plain C struct and use it to build immutable, shareable `PyTypeObject`s

[markshannon]

The mutability of PyTypeObject (builtin classes) and the shareability of statically allocated (C) objects makes safe handling of builtin types fragile, difficult, and ultimately unsafe.

Whereas, immutable objects are great for free-threading and multiple interpreters. They can be freely shared and there are no race conditions, and for immortal objects no contention when using them.

Immortal, immutable objects are also much easier to use from C code. Accessing int class is easy: &PyLong_Type. But accessing array.array is a real pain, requiring API calls.

We are making unreasonable demands of third-party code to support multiple interpreters, for code that used to be simple.

We can provide immutable, sharable PyTypeObjects with a simple, easy to use API that makes it easy to port old code with the following few API changes:

• Strip the PyObject_VAR_HEAD from the start of struct _typeobject making it a plain C struct, struct _classdef not a Python object.

• Make PyTypeObject an opaque struct

• Provide an API to create a PyTypeObject from a struct _classdef: int Py_MakeClass(struct _classdef def, PyInterpreterState *interp, PyTypeObject **result)

• interp == NULL. The result would be a pointer to an immortal and immortal class.

• interp != NULL. The result would be a new reference to a mortal, mutable class belonging to the given interpreter.

The sooner we can make PyTypeObject opaque, the better, but we might need to keep it open for a release or two for backwards compatibility reasons.

[markshannon]

We can add the function Py_MakeClass easily for 3.13.
All it needs to do is allocate space for PyTypeObject, copy the fields from the struct and call PyType_Ready.

[gvanrossum]

I think this would have to be debated by the C API WG. It feels like a big deal to propose these changes (with a long migration trajectory) that ought to be considered and planned carefully. Casually adding one new C API in 3.13 and hoping the rest will be easy sounds, um, optimistic. :-)

[colesbury]

In the interp == NULL immortal case, what provides the storage for the PyTypeObject returned from Py_MakeClass()? Is it heap allocated? Embedded in struct _classdef? Something else?

[erlend-aasland]

• Provide an API to create a PyTypeObject from a struct _classdef: int Py_MakeClass(struct _classdef def, PyInterpreterState *interp, PyTypeObject **result)
• interp == NULL. The result would be a pointer to an immortal and immortal class.
• interp != NULL. The result would be a new reference to a mortal, mutable class belonging to the given interpreter.

FWIW: I'd definitely make that two APIs: one API for the interp == NULL case and API for the interp != NULL case.

[ronaldoussoren]

If I read this correctly this (making PyTypeObject opaque) would break PyObjC, which subclasses type in C. The following is the struct declaration for those subclasses:

https://github.com/ronaldoussoren/pyobjc/blob/5a24aad5e91a874adb7c23d712d1fee5d94d5f18/pyobjc-core/Modules/objc/objc-class.h#L80-L95

Functionally I need a low-cost way to get additional C data associated with a Python class. I could probably find a way to store that data out of line, but that's likely a lot more expensive because this would require looking up the additional data in a separate data structure.

And to be clear: I can and will adjust to a new API when available, but would prefer to keep a cheap way to get at the associated data.

[zooba]

I'm strongly in favour of the idea, but as Guido says this isn't going to be a fast transition.

I believe we (@ericsnowcurrently mainly) cleaned up a decent amount of our own state on type objects so they are close to being immutable, but going the rest of the way is going to be tougher.

Is there a way we can get benefit from easily being able to tell that a type object is immutable (checking the extra size for 0)? If we can start telling people that not allocating extra bytes is a significant benefit for certain operations, they have some motivation to migrate. If we haven't already, we ought to be able to add an API that hides whether the extra data is stored out of line (returns a pointer to it) and then migrate it behind the scenes to interpreter state rather than directly on the type object.

Like Ronald, I would also quite like a cheap way to get data associated with any object (having just spent some time writing native profiling code, that needs to store data against any arbitrary callable). It's possible that our HAMT implementation may suit, and it could be worth exposing that as an alternative for per-interpreter, per-object data?

The first step is certainly adding the APIs to access members of PyTypeObject indirectly rather than directly, even if we don't make anything opaque yet.

[ericsnowcurrently]

@markshannon wrote:

The mutability of PyTypeObject (builtin classes) and the shareability of statically allocated (C) objects makes safe handling of builtin types fragile, difficult, and ultimately unsafe.

Whereas, immutable objects are great for free-threading and multiple interpreters. They can be freely shared and there are no race conditions, and for immortal objects no contention when using them.

Immortal, immutable objects are also much easier to use from C code. Accessing int class is easy: &PyLong_Type. But accessing array.array is a real pain, requiring API calls.

FWIW, there is probably room to explore solutions that extend beyond PyTypeObject. For example, in the past we've discussed how we could make static MappingProxyType instances or static code objects. There's enough overlap with what we'd need for PyTypeObject that we should at least keep all that in mind.

@markshannon wrote:

• Provide an API to create a PyTypeObject from a struct _classdef: int Py_MakeClass(struct _classdef def, PyInterpreterState *interp, PyTypeObject **result)

Alternately, we could add a public equivalent to _PyStaticType_InitBuiltin(). That would minimize the impact on extension maintainers. Then again, it's probably worth the extra effort to make PyTypeObject opaque.

@colesbury wrote:

In the interp == NULL immortal case, what provides the storage for the PyTypeObject returned from Py_MakeClass()? Is it heap allocated? Embedded in struct _classdef? Something else?

I'd think heap allocated.

@zooba wrote:

The first step is certainly adding the APIs to access members of PyTypeObject indirectly rather than directly, even if we don't make anything opaque yet.

+1

[gvanrossum]

If I hasn't just had an in-person chat with @ericsnowcurrently about this I would still be utterly confused. It appears that types have exactly three mutable attributes that are a problem: tp_dict, tp_subclasses, and tp_weaklist. For heap types, this is solved by having the whole type live in the heap (leading to Mark's issue with accessing array.array, which is a heap type).

For static types that are part of the CPython implementation, this is currently solved by storing those three attributes (and only those) in the interpreter state, in the array interp->types.builtins, using an index that is stored (sneakily) in the static type's tp_subclasses field. There are (sometimes only internal) APIs to handle the differences, e.g. PyType_GetDict(), _PyType_GetSubclasses(), and _PyObject_GET_WEAKREFS_LISTPTR(). (All these use _PyStaticType_GetState() to access the state if it's stored in the interpreter, i.e., if tp_flags & _Py_TPFLAGS_STATIC_BUILTIN is set.)

I am nevertheless still confused about the motivation of the proposal. The motivational section above seems high on advantages of immutable, immortal objects (I have no argument there) but doesn't go into specifics about current pain points. Is the goal to have the struct _classdef loaded in read-only memory? (It currently can't, despite the offloading of the mutable parts, because the index to the mutable parts still is written to the object dynamically, and of course because it has a refcount field.) Or is the goal to add more (per-interpreter, possibly internal) mutable members to type objects, which would be a pain (requiring all their accessors to use _PyStaticType_GetState())?

And if we use the struct _classdef as a template for the construction of a per-interpreter type object, why can't we use heap types for that? Perhaps the more important use case is interp == NULL, where we would create a single shareable, immutable, immortal type object, allocated on the heap (?). Would those also have an immutable, immortal tp_dict, and somehow disable tp_subclasses and tp_weaklist? Or would they use an extension of the above mechanism for storing the state in the interpreter?

It would seem that (assuming we eat our own dogfood) this proposal would defeat the advantage of &PyLong_Type, since that would henceforth just be the address of the template.

Finally, @zooba wrote:

Like Ronald, I would also quite like a cheap way to get data associated with any object [...]

But in @ronaldoussoren's post I only see a need for data associated with specific type objects, not with arbitrary objects (which is how I understand @zooba's post).

All in all I really hope that @markshannon can clarify his motivation and proposed implementation (at least at a high level so I can reason about some properties of the new type objects).

[ronaldoussoren]

Like Ronald, I would also quite like a cheap way to get data associated with any object [...]

But in @ronaldoussoren's post I only see a need for data associated with specific type objects, not with arbitrary objects (which is how I understand @zooba's post).

That's correct. PyObjC dynamically creates (a fairly large number of) subclasses of type with additional state, something like:

class NSObject(type):
     native_class: ...

The native_class is C-only slot and not a Python value. Having a single pointer value that's owned by a the subtype would work for me (e.g. tp_type_state slot, with an associated slot for a cleanup function).

Finally: PyObjC's use case generates these type objects at runtime. I currently don't support subinterpreters, but when I do add that support this will not involve sharing these type objects between sub interpreters.

[vstinner]

In June, I took notes on PyTypeObject members and how they are used: python/cpython#105970

[markshannon]

I guess I was too general with the API. Per-interpreter types can be mutable and need to have attached data as @ronaldoussoren and @zooba point out.

So let's focus on the immortal, immutable case for now. This should cover the Cython generator type, decimal.Decimal, array.aray and other important classes.

Dropping the PyInterpreterState *interp argument we have int Py_MakeClass(struct _classdef def, PyTypeObject **result).

Most of the machinery is already present in _PyStaticType_InitBuiltin.
The main difference is that we will need to allow a variable number of static types and to allocate __dict__ and __subclasses__ lazily, since the main interpreter may not exist when we create the PyTypeObject.
Once that is done, the initial implementation can be as simple as:

typedef _typeobject PyClassDefinition;
int
Py_MakeClass(PyClassDefinition *def, PyTypeObject **result)
{
    *result = malloc(sizeof(PyTypeObject));
    if (*result == NULL) return -1;
    memcpy(*result, def);
    return _PyStaticType_InitBuiltin(*result);
}

[zooba]

But in @ronaldoussoren's post I only see a need for data associated with specific type objects, not with arbitrary objects (which is how I understand @zooba's post).

You understood correctly. I skipped the bit about "by storing a dynamic index in the so-called 'immutable' type struct" straight to "what if we just had a per-interpreter data structure that could look up indirectly-attached data for anything". That works both for the current attached data on static types (which, I'll note, are not performance critical members) and for other tasks that may require storing attached data against any object.

As far as I can tell, the only benefit of doing this at all if is we can make the entire type object completely immutable. It doesn't actually matter if it's opaque or not - I don't think we can avoid supporting native "subclasses" of PyTypeObject - but if we can at least identify the ones that are immutable at runtime, we can handle certain scenarios differently (e.g. passing types between subinterpreters).

Also, opaque structs are just generally good for other languages that integrate with CPython. That really just means having all the APIs you need to be able to treat them as opaque if you want, but long-term I do hope that using those APIs becomes the default even for C developers.

[gvanrossum]

Sounds like there's quite a number of different motivations, solutions, and properties getting mixed in here. (E.g. Do we do something for all objects, or just for types? Are types going to be opaque or not?) It sure feels like it's going to take a PEP to sort out what we're going to do and why, what impact it will have, and how the APIs involved will be able to evolve.

[gvanrossum]

In my mind I now summarize the motivation here as a simpler alternative to PyType_FromSpec() when the it's okay for the type to be immortal, allowing us to store a pointer to the type in a C global during the first module initialization. Other code in the module that needs access to the type can then get it from there rather than having to dig through the module state. And it will support multi-phase init for modules (needed to support multiple interpreters).