From 3dbeb2c33e611f6a86aca0943ff8ba72604b9d94 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Mon, 23 Oct 2023 15:05:10 +0200 Subject: [PATCH 1/5] Isolating Extension Modules HOWTO: List GC-related gotchas --- Doc/howto/isolating-extensions.rst | 76 +++++++++++++++++++++++++++--- 1 file changed, 70 insertions(+), 6 deletions(-) diff --git a/Doc/howto/isolating-extensions.rst b/Doc/howto/isolating-extensions.rst index 8f3787f2d2f145..f7e24f986fad65 100644 --- a/Doc/howto/isolating-extensions.rst +++ b/Doc/howto/isolating-extensions.rst @@ -339,12 +339,44 @@ That is, heap types should: - Define a traverse function using ``Py_tp_traverse``, which visits the type (e.g. using :c:expr:`Py_VISIT(Py_TYPE(self))`). -Please refer to the :ref:`the documentation ` of +Please refer to the the documentation of :c:macro:`Py_TPFLAGS_HAVE_GC` and :c:member:`~PyTypeObject.tp_traverse` for additional considerations. -If your traverse function delegates to the ``tp_traverse`` of its base class -(or another type), ensure that ``Py_TYPE(self)`` is visited only once. +The API for defining heap types grew organically, leaving it +somewhat awkward to use in its current state. +The following sections will guide you through common issues. + + +``tp_traverse`` in Python 3.8 and lower +....................................... + +The requirement to visit the type from ``tp_traverse`` was added in Python 3.9. +If you support Python 3.8 and lower, the traverse function must *not* +visit the type, so it must be more complicated:: + + static int my_traverse(PyObject *self, visitproc visit, void *arg) + { + if (Py_Version >= 0x03090000) { + Py_VISIT(Py_TYPE(self)); + } + return 0; + } + +Unfortunately, :c:data:`Py_Version` was only added in Python 3.11. +As a replacement, use: + +* :c:macro:`PY_VERSION_HEX`, if not using the stable ABI, or +* :py:data:`sys.version_info` (via :c:func:`PySys_GetObject` and + :c:func:`PyArg_ParseTuple`). + + +Delegating ``tp_traverse`` +.......................... + +If your traverse function delegates to the :c:member:`~PyTypeObject.tp_traverse` +of its base class (or another type), ensure that ``Py_TYPE(self)`` is visited +only once. Note that only heap type are expected to visit the type in ``tp_traverse``. For example, if your traverse function includes:: @@ -356,11 +388,43 @@ For example, if your traverse function includes:: if (base->tp_flags & Py_TPFLAGS_HEAPTYPE) { // a heap type's tp_traverse already visited Py_TYPE(self) } else { - Py_VISIT(Py_TYPE(self)); + if (Py_Version >= 0x03090000) { + Py_VISIT(Py_TYPE(self)); + } } -It is not necessary to handle the type's reference count in ``tp_new`` -and ``tp_clear``. +It is not necessary to handle the type's reference count in +:c:member:`~PyTypeObject.tp_new` and :c:member:`~PyTypeObject.tp_clear`. + + +Defining ``tp_dealloc`` +....................... + +If your type has a custom :c:member:`~PyTypeObject.tp_dealloc` function, +it needs to decrement the reference count of the type. + +To keep the type valid while ``tp_free`` is called, this needs to be done +*after* the instance is deallocated. For example:: + + static void my_dealloc(PyObject *self) + { + ... + PyTypeObject *type = Py_TYPE(self); + type->tp_free(self); + Py_DECREF(type); + } + +The default ``tp_dealloc`` function does this; +If your type does *not* override +``tp_dealloc`` you don't need to add it. + + +Not overriding ``tp_free`` +.......................... + +The :c:member:`~PyTypeObject.tp_free` slot of a heap type must be set to +:c:func:`PyObject_GC_Del`. +This is the default; avoid overriding it. Module State Access from Classes From b6c5ae8609c80d4cc41bd209a7beb47fea5eb8c9 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 31 Oct 2023 15:16:07 +0100 Subject: [PATCH 2/5] Mention PyObject_GC_UnTrack and PyObject_GC_New --- Doc/howto/isolating-extensions.rst | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/Doc/howto/isolating-extensions.rst b/Doc/howto/isolating-extensions.rst index f7e24f986fad65..763563058c0a4b 100644 --- a/Doc/howto/isolating-extensions.rst +++ b/Doc/howto/isolating-extensions.rst @@ -401,21 +401,25 @@ Defining ``tp_dealloc`` ....................... If your type has a custom :c:member:`~PyTypeObject.tp_dealloc` function, -it needs to decrement the reference count of the type. +it needs to: -To keep the type valid while ``tp_free`` is called, this needs to be done -*after* the instance is deallocated. For example:: +- call :c:func:`PyObject_GC_UnTrack` before any fields are invalidated, and +- decrement the reference count of the type. + +To keep the type valid while ``tp_free`` is called, the type's refcount needs +to be decremented *after* the instance is deallocated. For example:: static void my_dealloc(PyObject *self) { + PyObject_GC_UnTrack(self); ... PyTypeObject *type = Py_TYPE(self); type->tp_free(self); Py_DECREF(type); } -The default ``tp_dealloc`` function does this; -If your type does *not* override +The default ``tp_dealloc`` function does this, so +if your type does *not* override ``tp_dealloc`` you don't need to add it. @@ -427,6 +431,18 @@ The :c:member:`~PyTypeObject.tp_free` slot of a heap type must be set to This is the default; avoid overriding it. +Avoiding ``PyObject_New`` +......................... + +GC-tracked objects need to be allocated using GC-aware functions. + +If you use use :c:func:`PyObject_New` or :c:func:`PyObject_NewVar`: + +- Get and call type's :c:member:`~PyTypeObject.tp_alloc` slot , if possible. +- If not possible (e.g. inside a custom ``tp_alloc``), + call :c:func:`PyObject_GC_New` or :c:func:`PyObject_GC_NewVar`. + + Module State Access from Classes -------------------------------- From 78e0f8459bc313b463a6de90144db1899277006c Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 31 Oct 2023 16:43:40 +0100 Subject: [PATCH 3/5] Stronger language for overriding tp_free Co-authored-by: Eric Snow --- Doc/howto/isolating-extensions.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/isolating-extensions.rst b/Doc/howto/isolating-extensions.rst index 763563058c0a4b..18a2ae0ea04383 100644 --- a/Doc/howto/isolating-extensions.rst +++ b/Doc/howto/isolating-extensions.rst @@ -428,7 +428,7 @@ Not overriding ``tp_free`` The :c:member:`~PyTypeObject.tp_free` slot of a heap type must be set to :c:func:`PyObject_GC_Del`. -This is the default; avoid overriding it. +This is the default; do not override it. Avoiding ``PyObject_New`` From 2a4568a0293a48099da4685c6782c04b338d40f7 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Fri, 3 Nov 2023 11:52:21 +0100 Subject: [PATCH 4/5] Copy-pastable examples for avoiding PyObject_New --- Doc/howto/isolating-extensions.rst | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/Doc/howto/isolating-extensions.rst b/Doc/howto/isolating-extensions.rst index 18a2ae0ea04383..1924772c7da97d 100644 --- a/Doc/howto/isolating-extensions.rst +++ b/Doc/howto/isolating-extensions.rst @@ -438,9 +438,20 @@ GC-tracked objects need to be allocated using GC-aware functions. If you use use :c:func:`PyObject_New` or :c:func:`PyObject_NewVar`: -- Get and call type's :c:member:`~PyTypeObject.tp_alloc` slot , if possible. -- If not possible (e.g. inside a custom ``tp_alloc``), - call :c:func:`PyObject_GC_New` or :c:func:`PyObject_GC_NewVar`. +- Get and call type's :c:member:`~PyTypeObject.tp_alloc` slot, if possible. + That is, replace ``TYPE *o = PyObject_New(TYPE, typeobj)`` by:: + + TYPE *o = typeobj->tp_alloc(typeobj, 0); + + Replace ``o = PyObject_NewVar(TYPE, typeobj, size)`` with the same, + but use size instead of the 0. + +- If the above is not possible (e.g. inside a custom ``tp_alloc``), + call :c:func:`PyObject_GC_New` or :c:func:`PyObject_GC_NewVar`:: + + TYPE *o = PyObject_GC_New(TYPE, typeobj); + + TYPE *o = PyObject_GC_NewVar(TYPE, typeobj, size); Module State Access from Classes From 0b01d7821f020c80ea74b885e81fc584853d2cb8 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Mon, 13 Nov 2023 11:31:23 +0100 Subject: [PATCH 5/5] Update Doc/howto/isolating-extensions.rst Co-authored-by: Eric Snow --- Doc/howto/isolating-extensions.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/howto/isolating-extensions.rst b/Doc/howto/isolating-extensions.rst index 1924772c7da97d..835c0afad7c6c8 100644 --- a/Doc/howto/isolating-extensions.rst +++ b/Doc/howto/isolating-extensions.rst @@ -439,7 +439,7 @@ GC-tracked objects need to be allocated using GC-aware functions. If you use use :c:func:`PyObject_New` or :c:func:`PyObject_NewVar`: - Get and call type's :c:member:`~PyTypeObject.tp_alloc` slot, if possible. - That is, replace ``TYPE *o = PyObject_New(TYPE, typeobj)`` by:: + That is, replace ``TYPE *o = PyObject_New(TYPE, typeobj)`` with:: TYPE *o = typeobj->tp_alloc(typeobj, 0);