Optimizer/executor crash when using address sanitizer

[gvanrossum]

Bug report

This was reported on [email protected], but it's on the main branch, so I consider it's just a crasher bug.

The repro (for me) comes down to:

./configure --with-address-sanitizer
make
./python.exe -m test test_capi.test_opt -m test_basic_loop

I've bisected it to 7b21403: GH-112354: Initial implementation of warm up on exits and trace-stitching (GH-114142).

Note that the test passes, the crash happens somewhere during cleanup.

The address sanitizer output:

(.venv) ~/cpython$ ./python.exe -m test -v test_capi.test_opt -m test_basic_loop
== CPython 3.13.0a4+ (bisect/good-acda1757bc682922292215906459c2735ee99c04-1-g7b21403ccd1:7b21403ccd1,) [Clang 15.0.0 (clang-1500.3.9.4)]
== macOS-14.4.1-arm64-arm-64bit-Mach-O little-endian
== Python build: release ASAN
== cwd: /Users/guido/cpython/build/test_python_worker_84494æ
== CPU count: 12
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests
== sanitizers: address

Using random seed: 68290980
0:00:00 load avg: 1.35 Run 1 test sequentially
0:00:00 load avg: 1.35 [1/1] test_capi.test_opt
test_basic_loop (test.test_capi.test_opt.TestUops.test_basic_loop) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.001s

OK
=================================================================
==84494==ERROR: AddressSanitizer: global-buffer-overflow on address 0x0001030e20f8 at pc 0x000102adc79c bp 0x00016d7246e0 sp 0x00016d7246d8
READ of size 8 at 0x0001030e20f8 thread T0
    #0 0x102adc798 in visit_decref gc.c:444
    #1 0x102b33e84 in executor_traverse optimizer.c:342
    #2 0x102adc3a4 in deduce_unreachable gc.c:1087
    #3 0x102ad82a8 in gc_collect_main gc.c:1359
    #4 0x102bbb61c in gc_collect gcmodule.c.h:140
    #5 0x102a4739c in _PyEval_EvalFrameDefault generated_cases.c.h:1122
    #6 0x102a28b50 in PyEval_EvalCode ceval.c:593
    #7 0x102a221d8 in builtin_exec bltinmodule.c.h:521
    #8 0x10287f000 in cfunction_vectorcall_FASTCALL_KEYWORDS methodobject.c:441
    #9 0x1027b96c8 in PyObject_Vectorcall call.c:327
    #10 0x102a43348 in _PyEval_EvalFrameDefault generated_cases.c.h:816
    #11 0x1027b94b4 in _PyVectorcall_Call call.c:273
    #12 0x102bb9f00 in pymain_run_module main.c:297
    #13 0x102bb8388 in Py_RunMain main.c:707
    #14 0x102bb9794 in pymain_main main.c:737
    #15 0x102bb9cb4 in Py_BytesMain main.c:761
    #16 0x1863260dc  (<unknown module>)

0x0001030e20f8 is located 8 bytes before global variable 'COLD_EXITS' defined in 'Python/optimizer.c' (0x1030e2100) of size 69632
0x0001030e20f8 is located 23 bytes after global variable 'cold_exits_initialized' defined in 'Python/optimizer.c' (0x1030e20e0) of size 1
SUMMARY: AddressSanitizer: global-buffer-overflow gc.c:444 in visit_decref
Shadow bytes around the buggy address:
  0x0001030e1e00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0001030e1e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0001030e1f00: 00 00 00 00 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x0001030e1f80: 01 f9 f9 f9 00 00 00 00 00 00 00 00 00 00 00 00
  0x0001030e2000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0001030e2080: 00 00 00 02 f9 f9 f9 f9 00 f9 f9 f9 01 f9 f9[f9]
  0x0001030e2100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0001030e2180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0001030e2200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0001030e2280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0001030e2300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==84494==ABORTING
Fatal Python error: Aborted

Current thread 0x00000001ee3bbac0 (most recent call first):
  Garbage-collecting
  File "/Users/guido/cpython/Lib/test/support/__init__.py", line 776 in gc_collect
  File "/Users/guido/cpython/Lib/test/libregrtest/single.py", line 141 in _load_run_test
  File "/Users/guido/cpython/Lib/test/libregrtest/single.py", line 178 in _runtest_env_changed_exc
  File "/Users/guido/cpython/Lib/test/libregrtest/single.py", line 278 in _runtest
  File "/Users/guido/cpython/Lib/test/libregrtest/single.py", line 306 in run_single_test
  File "/Users/guido/cpython/Lib/test/libregrtest/main.py", line 351 in run_test
  File "/Users/guido/cpython/Lib/test/libregrtest/main.py", line 385 in run_tests_sequentially
  File "/Users/guido/cpython/Lib/test/libregrtest/main.py", line 531 in _run_tests
  File "/Users/guido/cpython/Lib/test/libregrtest/main.py", line 566 in run_tests
  File "/Users/guido/cpython/Lib/test/libregrtest/main.py", line 729 in main
  File "/Users/guido/cpython/Lib/test/libregrtest/main.py", line 737 in main
  File "/Users/guido/cpython/Lib/test/__main__.py", line 2 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Extension modules: _testcapi, _testinternalcapi (total: 2)
Abort trap: 6
(.venv) ~/cpython$ 

[gvanrossum]

Looks like the problem is that the GC is trying to traverse the statically allocated executor objects in COLD_EXITS -- but those don't have space for the GC stuff that lives before the header. I'm not sure what to do about this yet.

OT: The address sanitizer output is incredibly helpful here, it led me straight to this problem.

[brianschubert]

Is this a duplicate of GH-118074?

[gvanrossum]

Oh, it is. Thanks for finding that! (I looked but didn't find it.) Will close this and continue the conversation there.