bpo-32388: Remove cross-version binary compatibility requirement in tp_flags

[pitrou]

Starting with 3.8, it is now allowed to add new fields at the end of the PyTypeObject struct in feature releases without having to allocate a dedicated compatibility flag in tp_flags. This will reduce the risk of running out of bits in the 32-bit tp_flags value.

https://bugs.python.org/issue32388

[jdemeyer]

You write "Starting with 3.7, binary compatibility of C extensions accross feature releases of Python is not supported anymore" but has such binary compatibility ever been supported? This PR seems to imply that it was, but PEP 384 explicitly says that the layout of PyTypeObject is not part of the stable ABI.

PS: typo accross -> across

[pitrou]

I updated this PR against master and addressed the review comments.

[pitrou]

@zooba Is there a way to retry an Azure job which failed because it couldn't "provision the agent"?

[jdemeyer]

It would be good to get this done before the first 3.8 beta.

@scoder: is there any chance that you could finish the "core review"?

[jdemeyer]

Starting with 3.8, it is now allowed to add new fields at the end of the PyTypeObject struct (and even in the middle, if we are so inclined)

Not "in the middle" as that would break the typical code of the form

PyTypeObject PyCFunction_Type = {
    PyVarObject_HEAD_INIT(&PyType_Type, 0)
    "builtin_function_or_method",
    sizeof(PyCFunctionObject),
    0,
    (destructor)meth_dealloc,                   /* tp_dealloc */
    0,                                          /* tp_print */
    0,                                          /* tp_getattr */
    0,                                          /* tp_setattr */
    0,                                          /* tp_reserved */
    (reprfunc)meth_repr,                        /* tp_repr */
    0,                                          /* tp_as_number */
    0,                                          /* tp_as_sequence */
    0,                                          /* tp_as_mapping */
    (hashfunc)meth_hash,                        /* tp_hash */
    PyCFunction_Call,                           /* tp_call */
    0,                                          /* tp_str */
    PyObject_GenericGetAttr,                    /* tp_getattro */
    0,                                          /* tp_setattro */
    0,                                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */
    0,                                          /* tp_doc */
    (traverseproc)meth_traverse,                /* tp_traverse */
    0,                                          /* tp_clear */
    meth_richcompare,                           /* tp_richcompare */
    offsetof(PyCFunctionObject, m_weakreflist), /* tp_weaklistoffset */
    0,                                          /* tp_iter */
    0,                                          /* tp_iternext */
    meth_methods,                               /* tp_methods */
    meth_members,                               /* tp_members */
    meth_getsets,                               /* tp_getset */
    0,                                          /* tp_base */
    0,                                          /* tp_dict */
};

With C99, there are better ways to write the above, but we should still support that.

[jdemeyer]

One thing that keeps bugging me when I see this diff: Do we really have to change the value of Py_TPFLAGS_HAVE_FINALIZE? I don't see any immediate problem with changing the value, but it would be more prudent to keep it as 1 << 0.

[vstinner]

cc @encukou

[encukou]

Can we keep the bit for backwards compatibility?
We're not running out of bits any time soon, especially since we'll not be adding one for each new feature. (Even though I want to add two of them now.)

Pedantically, something like MY_USUAL_FLAGS & ~Py_TPFLAGS_HAVE_FINALIZE would be correct in 3.7, so it should work in 3.8 as well.

[pitrou]

Pedantically, something like MY_USUAL_FLAGS & ~Py_TPFLAGS_HAVE_FINALIZE would be correct in 3.7, so it should work in 3.8 as well.

What value do you suggest Py_TPFLAGS_HAVE_FINALIZE should have then?

[jdemeyer]

What value do you suggest Py_TPFLAGS_HAVE_FINALIZE should have then?

The same value that it always had.

[pitrou]

Ok. Unless someone opposes, I'll merge this once CI is green.

[encukou]

Thank you!

[scoder]

Nice, happy to see that this was merged.