Heap types implemented in C must have Py_TPFLAGS_HAVE_GC and implement tp_traverse

[vstinner]

If a heap type does not have Py_TPFLAGS_HAVE_GC or doesn't not implement tp_traverse, the garbage collector may fail to collect the type and most of its instances at Python exit. The problem is worse when sub-interpreters are involved: if a sub-interpreter exits and does not clear all of its memory, the other interpreters will continue running with these leaks.

See issue gh-87138 for the rationale.

I wrote an article on the lock type which had this bug and caused a memory leak: https://vstinner.github.io/subinterpreter-leaks.html

Linked PRs

• [draft] gh-116946: Emit RuntimeError if a heap type does not implement traverse #116947

[vstinner]

Attached PR gh-116947 should help to discover heap types which don't implement traverse.

For example, the make command lists 52 heap types which don't implement tp_traverse:

• _blake2.blake2b
• _blake2.blake2s
• _bz2.BZ2Compressor
• _bz2.BZ2Decompressor
• _curses_panel.panel
• functools._lru_list_elem
• _hashlib.HASH
• _hashlib.HASHXOF
• _hashlib.HMAC
• _lzma.LZMACompressor
• _lzma.LZMADecompressor
• posix.DirEntry
• posix.ScandirIterator
• _random.Random
• select.epoll
• select.poll
• _sha3.sha3_224
• _sha3.sha3_256
• _sha3.sha3_384
• _sha3.sha3_512
• _sha3.shake_128
• _sha3.shake_256
• _ssl.Certificate
• _testcapi.HeapCType
• _testcapi.HeapCTypeSetattr
• _testcapi.HeapCTypeSubclass
• _testcapi.HeapCTypeSubclassWithFinalizer
• _testcapi.HeapCTypeWithBuffer
• _testcapi.HeapCTypeWithDict
• _testcapi.HeapCTypeWithDict2
• _testcapi.HeapCTypeWithNegativeDict
• _testcapi.HeapCTypeWithWeakref
• _testcapi.HeapCTypeWithWeakref2
• _testcapi.HeapDocCType
• _testcapi.NullTpDocType
• _testcapi._test_structmembersType_NewAPI
• _testimportexec.Str
• _testlimitedcapi.LimitedVectorCallClass
• _thread._localdummy
• _thread._ThreadHandle
• _tkinter.Tcl_Obj
• _tkinter.tkapp
• _tkinter.tktimertoken
• _tokenize.TokenizerIter
• _xxinterpchannels.ChannelID
• xxlimited_35.Null
• xxlimited_35.Str
• xxlimited.Str
• _xxsubinterpreters.CrossInterpreterBufferView
• zlib.Compress
• zlib.Decompress
• zlib._ZlibDecompressor

[vstinner]

See also issue gh-116915 "test_capi leaks references" and PR gh-116934 "Make _thread._ThreadHandle support GC" which fix it.