linecache.checkcache() is not threadsafe or GC finalizer re-entrancy safe

[graingert]

Bug report

Bug description:

import linecache
import weakref


def gen_func(n):
    func_code = """
def func():
    pass
"""
    g = {}
    exec(func_code, g, g)
    func = g['func']

    filename = f"<generated-{n}>"
    linecache.cache[filename] = (len(func_code), None, func_code.splitlines(True), filename)

    def cleanup_linecache(filename):
        def _cleanup():
            if filename in linecache.cache:
                del linecache.cache[filename]
        return _cleanup

    weakref.finalize(func, cleanup_linecache(filename))

    return func

def main():
    n = 0
    while True:
        func = gen_func(n)
        del func
        linecache.checkcache()
        n += 1
        if n % 100000 == 0:
            print(n)

if __name__ == '__main__':
    main()

This crashes with a KeyError every time for me on 3.12 and 3.13, but some people report that it never crashes on CPython:

 ✘  graingert@conscientious  ~/projects/weakref-func-cycle-never-gc   main  python demo.py
Traceback (most recent call last):
  File "/home/graingert/projects/weakref-func-cycle-never-gc/demo.py", line 38, in <module>
    main()
  File "/home/graingert/projects/weakref-func-cycle-never-gc/demo.py", line 32, in main
    linecache.checkcache()
  File "/usr/lib/python3.12/linecache.py", line 64, in checkcache
    entry = cache[filename]
            ~~~~~^^^^^^^^^^
KeyError: '<generated-147>'
 ✘  graingert@conscientious  ~/projects/weakref-func-cycle-never-gc   main  phyt 
 ✘  graingert@conscientious  ~/projects/weakref-func-cycle-never-gc   main  python3.13 demo.py 
Traceback (most recent call last):
  File "/home/graingert/projects/weakref-func-cycle-never-gc/demo.py", line 38, in <module>
    main()
    ~~~~^^
  File "/home/graingert/projects/weakref-func-cycle-never-gc/demo.py", line 32, in main
    linecache.checkcache()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/linecache.py", line 59, in checkcache
    entry = cache[filename]
            ~~~~~^^^^^^^^^^
KeyError: '<generated-2637>'

The script seems to run "forever" on Python3.9, 3.10 and 3.11 3.14.0a1+ (heads/main:ba088c8f9cf.

CPython versions tested on:

3.12, 3.13

Operating systems tested on:

No response

Linked PRs

• gh-126775: make linecache.checkcache threadsafe and GC re-entrency safe #126776
• [3.13] gh-126775: make linecache.checkcache threadsafe and GC re-entrency safe (GH-126776) #127778
• [3.12] gh-126775: make linecache.checkcache threadsafe and GC re-entrency safe (GH-126776) #127779

important meta issue:

This reproducer absolutely should not reproduce on cpython!!

There does seem to be another issue, because this function should be deleted instantly because its refcount drops to 0, and never run the finalizer during the linecache.checkcache call

[mattip]

Originally reported on pypy pypy/pypy#5109 and found in SymPy
cc @asmeurer

[graingert]

oh I totally forgot to link to the pypy issue! I thought I had done that, sorry