Skip to content

[WIP, DO NOT MERGE] bpo-41188: Prepare CPython for opague PyObject structure. #21262

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions Doc/c-api/typeobj.rst
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,10 @@ Quick Reference
+------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+
| :c:member:`~PyTypeObject.tp_vectorcall` | :c:type:`vectorcallfunc` | | | | | |
+------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+
| :c:member:`~PyTypeObject.tp_obj_offset` | const Py_ssize_t | | | X | ~ | |
+------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+
| :c:member:`~PyTypeObject.tp_obj_size` | const Py_ssize_t | | | X | ~ | |
+------------------------------------------------+-----------------------------------+-------------------+---+---+---+---+

.. [#slots]
A slot name in parentheses indicates it is (effectively) deprecated.
Expand Down Expand Up @@ -1015,6 +1019,12 @@ and :c:type:`PyType_Type` effectively act as defaults.)
:c:func:`PyType_HasFeature` takes a type and a flags value, *tp* and *f*, and
checks whether ``tp->tp_flags & f`` is non-zero.

.. data:: Py_TPFLAGS_USES_OPAQUE_OBJECT

This bit is set when the type object's :c:member:`PyTypeObject.tp_basicsize` is configured
for an opaque :c:type:`PyObject` structure. The value of :c:member:`PyTypeObject.tp_basicsize` is the size of the type's
internal object structure EXCLUDING the base type's structure size.

.. data:: Py_TPFLAGS_HEAPTYPE

This bit is set when the type object itself is allocated on the heap, for
Expand Down
4 changes: 4 additions & 0 deletions Include/cpython/object.h
Original file line number Diff line number Diff line change
Expand Up @@ -263,6 +263,10 @@ struct _typeobject {

destructor tp_finalize;
vectorcallfunc tp_vectorcall;

/* INTERNAL USE ONLY! MODIFYING THIS CAN CRASH PYTHON! */
const Py_ssize_t tp_obj_offset; /* Offset from "PyObject *" pointer */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that this is needed. When I get a "PyObject *ob", the offset to the PyObject structure is 0: "PyObject copy = *ob;" in Python internals is valid.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not required if everything inherited ONLY PyObject, that will never be true.

The offset is calculated from the immediate base type of the type, which can be for example: list, tuple, dict, type, custom types.

Same thing applies to "tp_obj_size", without these members, heap corruption can be very likely.

const Py_ssize_t tp_obj_size; /* Total memory allocation size */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the point of adding a new member. You can just check for Py_TPFLAGS_USES_OPAQUE_OBJECT flag in functions like PyType_GenericAlloc(), _PyObject_SIZE() and _PyObject_VAR_SIZE(). If the flag is set, add sizeof(PyObject).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment.

};

/* The *real* layout of a type object when allocated on the heap */
Expand Down
9 changes: 5 additions & 4 deletions Include/cpython/objimpl.h
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,11 @@
# error "_PyObject_VAR_SIZE requires SIZEOF_VOID_P be a power of 2"
#endif

#define _PyObject_VAR_SIZE(typeobj, nitems) \
_Py_SIZE_ROUND_UP((typeobj)->tp_basicsize + \
(nitems)*(typeobj)->tp_itemsize, \
SIZEOF_VOID_P)
static inline Py_ssize_t _PyObject_VAR_SIZE(PyTypeObject *typeobj, Py_ssize_t nitems)
{
Py_ssize_t size = (PyType_HasFeature(typeobj, Py_TPFLAGS_OMIT_PYOBJECT_SIZE) ? typeobj->tp_obj_size : typeobj->tp_basicsize);
return _Py_SIZE_ROUND_UP((size + (nitems * (typeobj->tp_itemsize))), SIZEOF_VOID_P);
}


/* This example code implements an object constructor with a custom
Expand Down
76 changes: 65 additions & 11 deletions Include/object.h
Original file line number Diff line number Diff line change
Expand Up @@ -102,26 +102,62 @@ typedef struct _typeobject PyTypeObject;
* by hand. Similarly every pointer to a variable-size Python object can,
* in addition, be cast to PyVarObject*.
*/
typedef struct _object {

typedef struct _object PyObject;
typedef struct _varobject PyVarObject;

// TODO: Send "_object" and "_varobject" to CPython internal only.
struct _object {
_PyObject_HEAD_EXTRA
Py_ssize_t ob_refcnt;
PyTypeObject *ob_type;
} PyObject;
};

struct _varobject {
PyObject ob_base;
Py_ssize_t ob_size; /* Number of items in variable part */
};

/* Cast argument to PyObject* type. */
#define _PyObject_CAST(op) ((PyObject*)(op))
#define _PyObject_CAST_CONST(op) ((const PyObject*)(op))

typedef struct {
PyObject ob_base;
Py_ssize_t ob_size; /* Number of items in variable part */
} PyVarObject;

/* Cast argument to PyVarObject* type. */
#define _PyVarObject_CAST(op) ((PyVarObject*)(op))
#define _PyVarObject_CAST_CONST(op) ((const PyVarObject*)(op))

Py_SLIB_LOCAL(Py_ssize_t) PyObject_GetRefCount(const PyObject *ob);
Py_SLIB_LOCAL(PyTypeObject *) PyObject_GetType(const PyObject *ob);

Py_SLIB_LOCAL(void) PyObject_SetRefCount(PyObject *ob, const Py_ssize_t refcnt);
Py_SLIB_LOCAL(void) PyObject_SetType(PyObject *ob, const PyTypeObject *type);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python 3.9 already have Py_SET_REFCNT() and Py_SET_TYPE() functions. What is the advantage of adding new functions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These aren't essentially "new functions", they are bridges into CPython's internals WITHOUT leaking implementation details.

As mentioned before, the existing functions WILL FAIL when PyObject goes full opaque.



Py_SLIB_LOCAL(Py_ssize_t) PyVarObject_GetSize(const PyVarObject *ob);

Py_SLIB_LOCAL(void) PyVarObject_SetSize(PyVarObject *ob, const Py_ssize_t size);


Py_SLIB_LOCAL(int) PyObject_IsType(const PyObject *ob, const PyTypeObject *type);


Py_SLIB_LOCAL(void *) PyObject_GetStructure(const PyObject *ob, const PyTypeObject *type);
#define Py_GET_STRUCTURE(ob, type) PyObject_GetStructure(_PyObject_CAST_CONST(ob), (const PyTypeObject *)type)

#ifdef Py_USE_SLIB
#define Py_REFCNT(ob) PyObject_GetRefCount(_PyObject_CAST_CONST(ob))
#define Py_TYPE(ob) PyObject_GetType(_PyObject_CAST_CONST(ob))

#define Py_SET_REFCNT(ob, refcnt) PyObject_SetRefCount(_PyObject_CAST(ob), refcnt)
#define Py_SET_TYPE(ob, type) PyObject_SetType(_PyObject_CAST(ob), type)


#define Py_SIZE(ob) PyVarObject_GetSize(_PyVarObject_CAST_CONST(ob))

#define Py_SET_SIZE(ob, size) PyVarObject_SetSize(_PyVarObject_CAST(ob), size)

#define Py_IS_TYPE(ob, type) PyObject_IsType(_PyObject_CAST_CONST(ob), type)
#else
static inline Py_ssize_t _Py_REFCNT(const PyObject *ob) {
return ob->ob_refcnt;
}
Expand Down Expand Up @@ -162,7 +198,7 @@ static inline void _Py_SET_SIZE(PyVarObject *ob, Py_ssize_t size) {
ob->ob_size = size;
}
#define Py_SET_SIZE(ob, size) _Py_SET_SIZE(_PyVarObject_CAST(ob), size)

#endif

/*
Type objects contain a string containing the type name (to help somewhat
Expand Down Expand Up @@ -317,6 +353,9 @@ Code can use PyType_HasFeature(type_ob, flag_value) to test whether the
given type object has a specified feature.
*/

/* Set if the type object's tp_basicsize is set for opague object */
#define Py_TPFLAGS_OMIT_PYOBJECT_SIZE (1UL << 8)

/* Set if the type object is dynamically allocated */
#define Py_TPFLAGS_HEAPTYPE (1UL << 9)

Expand Down Expand Up @@ -419,14 +458,30 @@ PyAPI_FUNC(void) _Py_NegativeRefcount(const char *filename, int lineno,

PyAPI_FUNC(void) _Py_Dealloc(PyObject *);

Py_SLIB_LOCAL(void) PyObject_IncRef(PyObject *op);

Py_SLIB_LOCAL(void) PyObject_DecRef(
#ifdef Py_REF_DEBUG
const char *filename, int lineno,
#endif
PyObject *op);

#ifdef Py_USE_SLIB
# define Py_INCREF(op) PyObject_IncRef(_PyObject_CAST(op))

# ifdef Py_REF_DEBUG
# define Py_DECREF(op) PyObject_DecRef(__FILE__, __LINE__, _PyObject_CAST(op))
# else
# define Py_DECREF(op) PyObject_DecRef(_PyObject_CAST(op))
# endif
#else
static inline void _Py_INCREF(PyObject *op)
{
#ifdef Py_REF_DEBUG
_Py_RefTotal++;
#endif
op->ob_refcnt++;
}

#define Py_INCREF(op) _Py_INCREF(_PyObject_CAST(op))

static inline void _Py_DECREF(
Expand All @@ -449,13 +504,12 @@ static inline void _Py_DECREF(
_Py_Dealloc(op);
}
}

#ifdef Py_REF_DEBUG
# define Py_DECREF(op) _Py_DECREF(__FILE__, __LINE__, _PyObject_CAST(op))
#else
# define Py_DECREF(op) _Py_DECREF(_PyObject_CAST(op))
#endif

#endif

/* Safely decref `op` and set `op` to NULL, especially useful in tp_clear
* and tp_dealloc implementations.
Expand Down
2 changes: 2 additions & 0 deletions Include/pyport.h
Original file line number Diff line number Diff line change
Expand Up @@ -187,9 +187,11 @@ typedef int Py_ssize_clean_t;
/* ignore warnings if the compiler decides not to inline a function */
# pragma warning(disable: 4710)
/* fastest possible local call under MSVC */
# define Py_SLIB_LOCAL(type) type __fastcall
# define Py_LOCAL(type) static type __fastcall
# define Py_LOCAL_INLINE(type) static __inline type __fastcall
#else
# define Py_SLIB_LOCAL(type) type
# define Py_LOCAL(type) static type
# define Py_LOCAL_INLINE(type) static inline type
#endif
Expand Down
2 changes: 1 addition & 1 deletion Lib/test/test_sys.py
Original file line number Diff line number Diff line change
Expand Up @@ -1334,7 +1334,7 @@ def delx(self): del self.__x
check((1,2,3), vsize('') + 3*self.P)
# type
# static type: PyTypeObject
fmt = 'P2nPI13Pl4Pn9Pn11PIPP'
fmt = 'P2nPI13Pl4Pn9Pn11PIPPll'
s = vsize(fmt)
check(int, s)
# class
Expand Down
43 changes: 39 additions & 4 deletions Objects/typeobject.c
Original file line number Diff line number Diff line change
Expand Up @@ -2086,11 +2086,19 @@ best_base(PyObject *bases)
return base;
}

static inline const Py_ssize_t get_type_totalsize(PyTypeObject *type)
{
if (type->tp_obj_size < sizeof(PyObject)) // PyType_Type->tp_new manages to trigger this.
return type->tp_basicsize;

return type->tp_obj_size;
}

static int
extra_ivars(PyTypeObject *type, PyTypeObject *base)
{
size_t t_size = type->tp_basicsize;
size_t b_size = base->tp_basicsize;
size_t t_size = get_type_totalsize(type);
size_t b_size = get_type_totalsize(base);

assert(t_size >= b_size); /* Else type smaller than base! */
if (type->tp_itemsize || base->tp_itemsize) {
Expand Down Expand Up @@ -4846,7 +4854,7 @@ object___sizeof___impl(PyObject *self)
isize = Py_TYPE(self)->tp_itemsize;
if (isize > 0)
res = Py_SIZE(self) * isize;
res += Py_TYPE(self)->tp_basicsize;
res += get_type_totalsize(Py_TYPE(self));

return PyLong_FromSsize_t(res);
}
Expand Down Expand Up @@ -4945,7 +4953,8 @@ PyTypeObject PyBaseObject_Type = {
PyObject_GenericGetAttr, /* tp_getattro */
PyObject_GenericSetAttr, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | /* tp_flags */
Py_TPFLAGS_OMIT_PYOBJECT_SIZE,
object_doc, /* tp_doc */
0, /* tp_traverse */
0, /* tp_clear */
Expand Down Expand Up @@ -5364,6 +5373,30 @@ inherit_slots(PyTypeObject *type, PyTypeObject *base)

static int add_operators(PyTypeObject *);

void set_type_memory_data(PyTypeObject *type, PyTypeObject *base)
{
// Obtain the true size of the base type.
Py_ssize_t base_size = 0, base_offset = 0;
if (base != NULL) { // Only possible if we're not dealing with PyBaseObject_Type
if (_PyType_HasFeature(base, Py_TPFLAGS_OMIT_PYOBJECT_SIZE)) {
base_size = base->tp_obj_size;
base_offset = base->tp_obj_offset;
} else {
base_size = base->tp_basicsize;
base_offset = 0; // PyObject ALWAYS occupies the start of the memory block.
}
}

Py_ssize_t *size_ptr = (Py_ssize_t *)&type->tp_obj_size, *offset_ptr = (Py_ssize_t *)&type->tp_obj_offset;
if (_PyType_HasFeature(type, Py_TPFLAGS_OMIT_PYOBJECT_SIZE)) {
*size_ptr = base_size + type->tp_basicsize;
*offset_ptr = base_offset + base_size; // The type's internal structure occupies the next block of memory.
} else {
*size_ptr = type->tp_basicsize; // tp_basicsize already includes base_size.
*offset_ptr = 0; // The type's internal structure already includes "PyObject".
}
}

int
PyType_Ready(PyTypeObject *type)
{
Expand Down Expand Up @@ -5426,6 +5459,8 @@ PyType_Ready(PyTypeObject *type)
goto error;
}

set_type_memory_data(type, base);

/* Initialize ob_type if NULL. This means extensions that want to be
compilable separately on Windows can call PyType_Ready() instead of
initializing the ob_type field of their type objects. */
Expand Down
7 changes: 7 additions & 0 deletions PC/pyconfig.h
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,10 @@ Py_NO_ENABLE_SHARED to find out. Also support MS_NO_COREDLL for b/w compat */
/* All windows compilers that use this header support __declspec */
#define HAVE_DECLSPEC_DLL

#if !defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_BUILTIN)
# define Py_USE_SLIB
#endif

/* For an MSVC DLL, we can nominate the .lib files used by extensions */
#ifdef MS_COREDLL
# if !defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_BUILTIN)
Expand All @@ -269,10 +273,13 @@ Py_NO_ENABLE_SHARED to find out. Also support MS_NO_COREDLL for b/w compat */
file in their Makefile (other compilers are
generally taken care of by distutils.) */
# if defined(_DEBUG)
# pragma comment(lib, "slib_python310_d.lib")
# pragma comment(lib,"python310_d.lib")
# elif defined(Py_LIMITED_API)
# pragma comment(lib, "slib_python310.lib")
# pragma comment(lib,"python3.lib")
# else
# pragma comment(lib, "slib_python310.lib")
# pragma comment(lib,"python310.lib")
# endif /* _DEBUG */
# endif /* _MSC_VER */
Expand Down
30 changes: 30 additions & 0 deletions PCbuild/pcbuild.sln
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "pythonw_uwp", "pythonw_uwp.
EndProject
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_uuid", "_uuid.vcxproj", "{CB435430-EBB1-478B-8F4E-C256F6838F55}"
EndProject
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "slib_pythoncore", "slib_pythoncore.vcxproj", "{983BBA6B-A34B-40F8-927C-A57C714F1887}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|ARM = Debug|ARM
Expand Down Expand Up @@ -1508,6 +1510,34 @@ Global
{CB435430-EBB1-478B-8F4E-C256F6838F55}.Release|Win32.Build.0 = Release|Win32
{CB435430-EBB1-478B-8F4E-C256F6838F55}.Release|x64.ActiveCfg = Release|x64
{CB435430-EBB1-478B-8F4E-C256F6838F55}.Release|x64.Build.0 = Release|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.Debug|ARM.ActiveCfg = Debug|Win32
{983BBA6B-A34B-40F8-927C-A57C714F1887}.Debug|ARM64.ActiveCfg = Debug|Win32
{983BBA6B-A34B-40F8-927C-A57C714F1887}.Debug|Win32.ActiveCfg = Debug|Win32
{983BBA6B-A34B-40F8-927C-A57C714F1887}.Debug|Win32.Build.0 = Debug|Win32
{983BBA6B-A34B-40F8-927C-A57C714F1887}.Debug|x64.ActiveCfg = Debug|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.Debug|x64.Build.0 = Debug|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGInstrument|ARM.ActiveCfg = Release|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGInstrument|ARM.Build.0 = Release|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGInstrument|ARM64.ActiveCfg = Release|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGInstrument|ARM64.Build.0 = Release|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGInstrument|Win32.ActiveCfg = Release|Win32
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGInstrument|Win32.Build.0 = Release|Win32
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGInstrument|x64.ActiveCfg = Release|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGInstrument|x64.Build.0 = Release|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGUpdate|ARM.ActiveCfg = Release|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGUpdate|ARM.Build.0 = Release|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGUpdate|ARM64.ActiveCfg = Release|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGUpdate|ARM64.Build.0 = Release|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGUpdate|Win32.ActiveCfg = Release|Win32
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGUpdate|Win32.Build.0 = Release|Win32
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGUpdate|x64.ActiveCfg = Release|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.PGUpdate|x64.Build.0 = Release|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.Release|ARM.ActiveCfg = Release|Win32
{983BBA6B-A34B-40F8-927C-A57C714F1887}.Release|ARM64.ActiveCfg = Release|Win32
{983BBA6B-A34B-40F8-927C-A57C714F1887}.Release|Win32.ActiveCfg = Release|Win32
{983BBA6B-A34B-40F8-927C-A57C714F1887}.Release|Win32.Build.0 = Release|Win32
{983BBA6B-A34B-40F8-927C-A57C714F1887}.Release|x64.ActiveCfg = Release|x64
{983BBA6B-A34B-40F8-927C-A57C714F1887}.Release|x64.Build.0 = Release|x64
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
Expand Down
5 changes: 5 additions & 0 deletions PCbuild/python3dll.vcxproj
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,11 @@
<ItemGroup>
<ResourceCompile Include="..\PC\python_nt.rc" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="slib_pythoncore.vcxproj">
<Project>{983bba6b-a34b-40f8-927c-a57c714f1887}</Project>
</ProjectReference>
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
Expand Down
8 changes: 6 additions & 2 deletions PCbuild/pythoncore.vcxproj
Original file line number Diff line number Diff line change
Expand Up @@ -501,6 +501,11 @@
<ItemGroup>
<ResourceCompile Include="..\PC\python_nt.rc" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="slib_pythoncore.vcxproj">
<Project>{983bba6b-a34b-40f8-927c-a57c714f1887}</Project>
</ProjectReference>
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
Expand Down Expand Up @@ -533,7 +538,6 @@
<Target Name="_WarnAboutZlib" BeforeTargets="PrepareForBuild" Condition="!$(IncludeExternals)">
<Warning Text="Not including zlib is not a supported configuration." />
</Target>

<Target Name="_CopyVCRuntime" AfterTargets="Build" Inputs="@(VCRuntimeDLL)" Outputs="$(OutDir)%(Filename)%(Extension)" DependsOnTargets="FindVCRuntime">
<!-- bpo-38597: When we switch to another VCRuntime DLL, include vcruntime140.dll as well -->
<Warning Text="A copy of vcruntime140.dll is also required" Condition="!$(VCToolsRedistVersion.StartsWith(`14.`))" />
Expand All @@ -542,4 +546,4 @@
<Target Name="_CleanVCRuntime" AfterTargets="Clean">
<Delete Files="@(VCRuntimeDLL->'$(OutDir)%(Filename)%(Extension)')" />
</Target>
</Project>
</Project>
Loading