_socket extension C capsule cannot be visited by the GC and so _socket.socket type may stay alive longer than expected

[vstinner]

Bug report

Checklist

• I am confident this is a bug in CPython, not a bug in a third-party project
• I have searched the CPython issue tracker,
  and am confident this bug has not been reported before

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.13.0a0 (heads/main:d63972e289, Aug 21 2023, 22:29:24) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)]

A clear and concise description of the bug:

The following script will never delete _socket.socket type:

import _socket

# uncomment to workaround the bug
#_socket.CAPI = None

_socket = None

import sys
del sys.modules['_socket']

import gc
gc.collect()

print("exit")

Example on a Python debug build:

$ ./python -X showrefcount script.py 
exit
[408 refs, 260 blocks]

The GC cannot visit the strong reference to the type stored in _socket.CAPI capsule (C API). In Python, creating a heap type creates a reference cycle. For example, a type MRO contains the type. Methods also contain a strong reference to their type.

$ ./python
Python 3.13.0a0 (heads/main:d63972e289, Aug 21 2023, 22:29:24) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)] on linux
>>> import _socket
>>> t=_socket.socket
>>> t.__mro__[0] is t
True

See also:

• https://pythondev.readthedocs.io/garbage_collector.html#reference-cycles
• https://vstinner.github.io/debug-python-refleak.html
• https://vstinner.github.io/subinterpreter-leaks.html

Linked PRs

• gh-108240: Fix a reference cycle in _socket.CAPI capsule #108241
• gh-108240: Add _PyCapsule_SetTraverse() internal function #108339
• gh-108240: _PyCapsule_SetTraverse() rejects NULL callbacks #108417
• gh-108240: Add pycore_capsule.h internal header file #108596

[vstinner]

I wrote PR #108241 to fix the issue for the _socket extension: avoid strong references in the capsule object.

[vstinner]

_curses extension is affected by the same bug: _curses._C_API capsule stores a strong reference to _curses.window heap type.

Extensions not affected by the issue, using borrowed references:

• _datetime.datetime_CAPI stores borrowed references to _datetime heap types
• pyexpat.expat_CAPI does not store types
• unicodedata._ucnhash_CAPI does not store types
• _testcapi capsules don't store types

[vstinner]

By the way, commit 46366ca may have introduced a leak: see PR #107577.

[vstinner]

I closed issue #107789 as a duplicate of this issue.

[vstinner]

Oh, @Eclips4 found that the _decimal extension is also affected: #107577 (comment)

[Eclips4]

Oh, @Eclips4 found that the _decimal extension is also affected: #107577 (comment)

Probably related: #107994

[vstinner]

Probably related: #107994

Oh right, the _decimal extension has no capsule, so the refleak is unrelated to this issue.

[vstinner]

cc @erlend-aasland @corona10 @pablogsal: yet another complex GC issue involving extension multi-phase initialization and the garbage collector.

[vstinner]

The _curses extension was not ported to multi-phase init API (PEP 489) yet. It still uses static types.

[Eclips4]

The _curses extension was not ported to multi-phase init API (PEP 489) yet. It still uses static types.

You mean leak in the _curses is a known issue?

import importlib
import sys

def foo():
    name = "_curses"
    importlib.import_module(name)
    del sys.modules[name]


for _ in range(4):
    foo()

./python.exe -X showrefcount example.py
[1006 refs, 491 blocks]

[vstinner]

You mean leak in the _curses is a known issue?

Yes. It should be ported to multi-phase init and its static types should be converted to heap types.

[CharlieZhao95]

Yes. It should be ported to multi-phase init and its static types should be converted to heap types.

FYI, it seems that work on isolating the _curses extension module is suspended. #101714 (comment)

[Eclips4]

FYI, it seems that work on isolating the _curses extension module is suspended. #101714 (comment)

So, should we re-open @chgnrdv PR about isolating _curses? Seems work almost done, but isolating module only for avoid reference leak looks kinda weird.

[erlend-aasland]

Using curses in a subinterpreter makes as little sense as using any other terminal-dependent extension module (like for example the readline module). Those extension modules only make sense if used from the main interpreter. Is there a use case where you want two extension modules trying to use the terminal from two different interpreters?

[vstinner]

The issue with _curses is that it leaks references, and so it makes debugging real leaks more complicated at Python exit:

$ ./python -X showrefcount -c 'pass'
[0 refs, 0 blocks]
$ ./python -X showrefcount -c 'import _curses'
[90 refs, 87 blocks]

I fixed some stdlib extensions by adding _PyTypes_FiniTypes() function which is called at Python exit. The problem here is that the _curses extension is a shared library, so Python cannot track its types. Maybe an "at exit" callback can be used for such cases (but not Py_AtExit() which calls the callback after the interpreter is deleted). Converting _curses to multi-phase init would give more freedom on how resources are released.

[erlend-aasland]

The problem with curses and readline is that they rely on process wide globals, which mean module state just won't do it. Try converting readline to multi-phase init, and you'll see what I mean :) Tkinter has similar issues.

[vstinner]

The _ast extension caused me lot of troubles since there are APIs to access its state without importing the extension. Its state is now stored in the interpreter :-)

[erlend-aasland]

Yes, storing the state in the interpreter is an option, but I don't think we want to do that for tkinter, readline, and curses.

[vstinner]

Fixed by commit 513c89d

Thanks for your bug report @Eclips4.