Skip to content

test_threading_local_clear_race segmentation fault in free-threaded CI #118490

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
colesbury opened this issue May 1, 2024 · 1 comment · Fixed by #121655
Closed

test_threading_local_clear_race segmentation fault in free-threaded CI #118490

colesbury opened this issue May 1, 2024 · 1 comment · Fixed by #121655
Labels
tests Tests in the Lib/test dir topic-free-threading type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@colesbury
Copy link
Contributor

colesbury commented May 1, 2024 •
edited by github-actions bot
Loading

Crash report

A segmentation fault in the macOS M1 arm64 refleaks buildbot:

Seen in https://buildbot.python.org/all/#/builders/1367/builds/146/steps/5/logs/stdio

test_threading_local_clear_race (test.test_threading_local.PyThreadingLocalTest.test_threading_local_clear_race) ... Fatal Python error: Segmentation fault
Current thread 0x00000001d8941300 (most recent call first):
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/test_threading_local.py", line 206 in test_threading_local_clear_race
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/unittest/case.py", line 606 in _callTestMethod
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/unittest/case.py", line 651 in run
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/unittest/case.py", line 707 in __call__
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/unittest/suite.py", line 122 in run
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/unittest/suite.py", line 122 in run
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/unittest/suite.py", line 84 in __call__
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/unittest/runner.py", line 240 in run
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/libregrtest/single.py", line 57 in _run_suite
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/libregrtest/single.py", line 37 in run_unittest
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/libregrtest/single.py", line 132 in test_func
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/libregrtest/refleak.py", line 106 in runtest_refleak
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/libregrtest/single.py", line 84 in regrtest_runner
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/libregrtest/single.py", line 135 in _load_run_test
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/libregrtest/single.py", line 178 in _runtest_env_changed_exc
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/libregrtest/single.py", line 278 in _runtest
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/libregrtest/single.py", line 309 in run_single_test
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/libregrtest/worker.py", line 77 in worker_process
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/libregrtest/worker.py", line 112 in main
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/libregrtest/worker.py", line 116 in <module>
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/runpy.py", line 88 in _run_code
  File "/Users/ec2-user/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/runpy.py", line 198 in _run_module_as_main
@colesbury colesbury added tests Tests in the Lib/test dir type-crash A hard crash of the interpreter, possibly with a core dump topic-free-threading labels May 1, 2024
@colesbury
Copy link
Contributor Author 

WARNING: ThreadSanitizer: heap-use-after-free (pid=1832131)
  Read of size 8 at 0x72a400041018 by main thread:
    #0 PyThreadState_Next /raid/sgross/cpython/Python/pystate.c:2488:20 (python+0x58fd3a) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #1 local_clear /raid/sgross/cpython/./Modules/_threadmodule.c:1579:22 (python+0x67b585) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #2 local_dealloc /raid/sgross/cpython/./Modules/_threadmodule.c:1597:5 (python+0x67abbe) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #3 _Py_Dealloc /raid/sgross/cpython/Objects/object.c:2855:5 (python+0x30ff14) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #4 _Py_MergeZeroLocalRefcount /raid/sgross/cpython/Objects/object.c (python+0x310542) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #5 Py_DECREF /raid/sgross/cpython/./Include/object.h:890:13 (python+0x49d3b8) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #6 Py_XDECREF /raid/sgross/cpython/./Include/object.h:1040:9 (python+0x49d3b8)
    #7 _PyEval_EvalFrameDefault /raid/sgross/cpython/Python/generated_cases.c.h:5627:13 (python+0x49d3b8)
    #8 _PyEval_EvalFrame /raid/sgross/cpython/./Include/internal/pycore_ceval.h:119:16 (python+0x47a833) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #9 _PyEval_Vector /raid/sgross/cpython/Python/ceval.c:1818:12 (python+0x47a833)
    #10 _PyFunction_Vectorcall /raid/sgross/cpython/Objects/call.c (python+0x25552c) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #11 _PyObject_VectorcallTstate /raid/sgross/cpython/./Include/internal/pycore_call.h:168:11 (python+0x25b01b) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #12 method_vectorcall /raid/sgross/cpython/Objects/classobject.c:92:18 (python+0x259722) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #13 _PyVectorcall_Call /raid/sgross/cpython/Objects/call.c:273:16 (python+0x254f15) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #14 _PyObject_Call /raid/sgross/cpython/Objects/call.c:348:16 (python+0x25512b) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #15 PyObject_Call /raid/sgross/cpython/Objects/call.c:373:12 (python+0x255357) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #16 _PyEval_EvalFrameDefault /raid/sgross/cpython/Python/generated_cases.c.h:1353:26 (python+0x483601) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #17 _PyEval_EvalFrame /raid/sgross/cpython/./Include/internal/pycore_ceval.h:119:16 (python+0x47a833) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #18 _PyEval_Vector /raid/sgross/cpython/Python/ceval.c:1818:12 (python+0x47a833)
    #19 _PyFunction_Vectorcall /raid/sgross/cpython/Objects/call.c (python+0x25552c) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #20 _PyObject_VectorcallDictTstate /raid/sgross/cpython/Objects/call.c:135:15 (python+0x25392d) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #21 _PyObject_Call_Prepend /raid/sgross/cpython/Objects/call.c:504:24 (python+0x25594d) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #22 slot_tp_call /raid/sgross/cpython/Objects/typeobject.c:9400:15 (python+0x3822bc) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
...

  Previous write of size 8 at 0x72a400041018 by thread T371:
    #0 free <null> (python+0xdc10f) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #1 _PyMem_RawFree /raid/sgross/cpython/Objects/obmalloc.c:90:5 (python+0x337339) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #2 _PyMem_DebugRawFree /raid/sgross/cpython/Objects/obmalloc.c:2754:5 (python+0x33c14b) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #3 PyMem_RawFree /raid/sgross/cpython/Objects/obmalloc.c:963:5 (python+0x3398a8) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #4 free_threadstate /raid/sgross/cpython/Python/pystate.c:1436:9 (python+0x590fe4) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)
    #5 _PyThreadState_DeleteCurrent /raid/sgross/cpython/Python/pystate.c:1838:5 (python+0x590fe4)
    #6 PyGILState_Release /raid/sgross/cpython/Python/pystate.c:2812:9 (python+0x590fe4)
    #7 temporary_c_thread /raid/sgross/cpython/./Modules/_testcapimodule.c:1623:5 (_testcapi.cpython-313td-x86_64-linux-gnu.so+0x1d684) (BuildId: 7b30b566ddc9cb08f2fcc308c2a802d985ffb2ef)
    #8 pythread_wrapper /raid/sgross/cpython/Python/thread_pthread.h:243:5 (python+0x5b54bb) (BuildId: 99e53ab3ef9a039265821c7e1b283c799a17a778)

@corona10 corona10 mentioned this issue Jun 29, 2024
Closed
@mpage mpage mentioned this issue Jul 12, 2024
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 19, 2024
@mpage @miss-islington
pythongh-120973: Fix thread-safety issues with threading.local (pyt…
1204d28 
…honGH-121655)

This is a small refactoring to the current design that allows us to
avoid manually iterating over threads.

This should also fix pythongh-118490.
(cherry picked from commit e059aa6)

Co-authored-by: mpage <[email protected]>
@miss-islington miss-islington mentioned this issue Jul 19, 2024
Merged
colesbury pushed a commit that referenced this issue Jul 19, 2024
@miss-islington @mpage
[3.13] gh-120973: Fix thread-safety issues with threading.local (GH…
bf7f4f3 
…-121655) (#122042)

This is a small refactoring to the current design that allows us to
avoid manually iterating over threads.

This should also fix gh-118490.
(cherry picked from commit e059aa6)

Co-authored-by: mpage <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
tests Tests in the Lib/test dir topic-free-threading type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

gh-120973: Fix thread-safety issues with threading.local mpage/cpython
1 participant
@colesbury