3.13b3 crashes in Cython test suite #121253

scoder · 2024-07-02T07:48:28Z

Crash report

What happened?

With the release of CPython 3.13b3, Cython's test suite started crashing on Linux/macOS/Windows in Github Actions. The position given by the CPython crash stack trace looks random, but it seems to be related to running the @no_gc test. However, I failed to reproduce the crash locally when running only that test.

Some of the crash logs (look for "Fatal Python error"):
https://github.com/cython/cython/actions/runs/9746719041/job/26897743924
https://github.com/cython/cython/actions/runs/9746719041/job/26897748128

Note that the test runs are sharded, so there's some noise in the output from other test processes. The crash predictably happens in shard [2].

[Side note: Py3.13 is missing from the issue form in crash.yml.]

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Output from running 'python -VV' on the command line:

No response

The text was updated successfully, but these errors were encountered:

vstinner · 2024-07-05T19:59:33Z

https://github.com/cython/cython/actions/runs/9746719041/job/26897743924

(...)

[2] compiling (c/cy2) and running no_gcrunTest (__main__.CythonRunTestCase.runTest)
[6] compiling (c/cy2) and running funcexcept ...  ... Fatal Python error: Segmentation fault

Current thread 0x00007f8395376740 (most recent call first):
  File "/home/runner/work/cython/cython/Cython/Compiler/ExprNodes.py", line 11143 in unop_node
  File "/home/runner/work/cython/cython/Cython/Compiler/Main.py", line 373 in parse
  File "/home/runner/work/cython/cython/Cython/Compiler/Pipeline.py", line 41 in parse
  File "/home/runner/work/cython/cython/Cython/Compiler/Pipeline.py", line 386 in run
  File "/home/runner/work/cython/cython/Cython/Compiler/Pipeline.py", line 404 in run_pipeline
  File "/home/runner/work/cython/cython/Cython/Compiler/Main.py", line 134 in process_pxd
  File "/home/runner/work/cython/cython/Cython/Compiler/Main.py", line 255 in find_module
  File "/home/runner/work/cython/cython/Cython/Compiler/Symtab.py", line 1504 in find_module
  File "/home/runner/work/cython/cython/Cython/Compiler/Nodes.py", line 8945 in analyse_declarations
  File "/home/runner/work/cython/cython/Cython/Compiler/Nodes.py", line 386 in analyse_declarations
  File "/home/runner/work/cython/cython/Cython/Compiler/ModuleNode.py", line 202 in analyse_declarations
  File "/home/runner/work/cython/cython/Cython/Compiler/ParseTreeTransforms.py", line 2166 in visit_ModuleNode
  File "/home/runner/work/cython/cython/Cython/Compiler/ParseTreeTransforms.py", line 2156 in __call__
  File "/home/runner/work/cython/cython/Cython/Compiler/Pipeline.py", line 386 in run
  File "/home/runner/work/cython/cython/Cython/Compiler/Pipeline.py", line 404 in run_pipeline
  File "/home/runner/work/cython/cython/Cython/Compiler/Main.py", line 134 in process_pxd
  File "/home/runner/work/cython/cython/Cython/Compiler/Main.py", line 255 in find_module
  File "/home/runner/work/cython/cython/Cython/Compiler/Symtab.py", line 1504 in find_module
  File "/home/runner/work/cython/cython/Cython/Compiler/Nodes.py", line 8945 in analyse_declarations
  File "/home/runner/work/cython/cython/Cython/Compiler/Nodes.py", line 386 in analyse_declarations
  File "/home/runner/work/cython/cython/Cython/Compiler/Nodes.py", line 386 in analyse_declarations
  File "/home/runner/work/cython/cython/Cython/Compiler/ModuleNode.py", line 202 in analyse_declarations
  File "/home/runner/work/cython/cython/Cython/Compiler/ParseTreeTransforms.py", line 2166 in visit_ModuleNode
(...)

https://github.com/cython/cython/actions/runs/9746719041/job/26897748128

(...)
[1] compiling (c/cy2) and running king1 ... f (multass)
Doctest: multass.f ... ok
g (multass)
Doctest: multass.g ... ok
h (multass)
Doctest: multass.h ... ok
j (multass)
Doctest: multass.j ... ok
runTest (__mp_main__.CythonRunTestCase.runTest)
[2] compiling (c/cy2) and running no_gc ... Windows fatal exception: access violation

Current thread 0x000014dc (most recent call first):
  File "<frozen ntpath>", line 235 in split
  File "<frozen ntpath>", line 263 in dirname
  File "D:\a\cython\cython\Cython\Compiler\Main.py", line 690 in search_include_directories
  File "D:\a\cython\cython\Cython\Compiler\Main.py", line 301 in search_include_directories
  File "D:\a\cython\cython\Cython\Compiler\Main.py", line 273 in find_pxd_file
  File "D:\a\cython\cython\Cython\Compiler\Main.py", line 209 in find_module
  File "D:\a\cython\cython\Cython\Compiler\Symtab.py", line 1504 in find_module
  File "D:\a\cython\cython\Cython\Compiler\Nodes.py", line 8945 in analyse_declarations
  File "D:\a\cython\cython\Cython\Compiler\Nodes.py", line 386 in analyse_declarations
  File "D:\a\cython\cython\Cython\Compiler\ModuleNode.py", line 202 in analyse_declarations
  File "D:\a\cython\cython\Cython\Compiler\ParseTreeTransforms.py", line 2166 in visit_ModuleNode
  File "D:\a\cython\cython\Cython\Compiler\ParseTreeTransforms.py", line 2156 in __call__
  File "D:\a\cython\cython\Cython\Compiler\Pipeline.py", line 386 in run
  File "D:\a\cython\cython\Cython\Compiler\Pipeline.py", line 404 in run_pipeline
  File "D:\a\cython\cython\Cython\Compiler\Main.py", line 134 in process_pxd
  File "D:\a\cython\cython\Cython\Compiler\Main.py", line 255 in find_module
  File "D:\a\cython\cython\Cython\Compiler\Symtab.py", line 1504 in find_module
  File "D:\a\cython\cython\Cython\Compiler\Nodes.py", line 8945 in analyse_declarations
  File "D:\a\cython\cython\Cython\Compiler\Nodes.py", line 386 in analyse_declarations
  File "D:\a\cython\cython\Cython\Compiler\ModuleNode.py", line 202 in analyse_declarations
  File "D:\a\cython\cython\Cython\Compiler\ParseTreeTransforms.py", line 2166 in visit_ModuleNode
(...)

vstinner · 2024-07-05T20:05:09Z

Are you using a debug build of Python? How can I only re-run tests which are triggering the crashes (not run the full test suite)?

scoder · 2024-07-06T05:38:21Z

It's using the regular CPython package that the setup-python action installs. But I was using a debug build locally (when I failed to reproduce the crash), so yes, that might make a difference. To run the test suite, see https://github.com/cython/cython/wiki/HackerGuide#the-test-suite

vstinner · 2024-07-06T08:19:58Z

I ran tests on Python 3.13 built in debug mode:

~/python/3.13/python -u runtests.py -vv --no-code-style -x Debugger --backends=c -j14 2>&1|tee tests.log

I got an assertion error:

(...)
[6] compiling (c/cy2) and running async_def ... [8] ['/home/vstinner/python/3.13/python', 'setup.py', 'build_ext', '--inplace']
Compiling dealloc_raise.pyx because it changed.
[1/1] Cythonizing dealloc_raise.pyx

/home/vstinner/dev/cython/Cython/Compiler/Main.py:373: FutureWarning: Cython directive 'language_level' not set, using '3' (Py3). This has changed from earlier releases! File: /home/vstinner/dev/cython/TEST_TMP/8/run/dealloc_raise/dealloc_raise.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)


[8] ['/home/vstinner/python/3.13/python', 'test_dealloc_raise1.py']

Objects/dictobject.c:699: _PyDict_CheckConsistency: Assertion failed: mp->ma_values->valid == 1
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7f5edac76330
object refcount : 1
object type     : 0x9039e0
object type name: dict
object repr     : {}

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007f5ee89e2740 (most recent call first):
  File "/home/vstinner/dev/cython/TEST_TMP/8/run/dealloc_raise/test_dealloc_raise1.py", line 10 in <module>

Extension modules: dealloc_raise (total: 1)

and a Fatal Error:

[2] running pure doctests in methodmangling_T5 ... ok
ExtTypeTotalOrderingEqNeLeGe (exttype_total_ordering)
Doctest: exttype_total_ordering.ExtTypeTotalOrderingEqNeLeGe ... C (methodmangling_T5)
Doctest: methodmangling_T5.C ... ok
ExtTypeTotalOrderingEqNeLt (exttype_total_ordering)
Doctest: exttype_total_ordering.ExtTypeTotalOrderingEqNeLt ... ok
f1 (methodmangling_T5.CMultiplyNested)
Doctest: methodmangling_T5.CMultiplyNested.f1 ... Fatal Python error: Segmentation fault

Current thread 0x00007ffa9ca99740 (most recent call first):
  File "<doctest methodmangling_T5.CMultiplyNested.f1[1]>", line 12 in <module>
  File "/home/vstinner/python/3.13/Lib/doctest.py", line 1395 in __run
  File "/home/vstinner/python/3.13/Lib/doctest.py", line 1566 in run
  File "/home/vstinner/python/3.13/Lib/doctest.py", line 2320 in runTest
  File "/home/vstinner/python/3.13/Lib/unittest/case.py", line 606 in _callTestMethod
  File "/home/vstinner/python/3.13/Lib/unittest/case.py", line 651 in run
  File "/home/vstinner/python/3.13/Lib/unittest/case.py", line 707 in __call__
  File "/home/vstinner/python/3.13/Lib/unittest/suite.py", line 122 in run
  File "/home/vstinner/dev/cython/runtests.py", line 1644 in run
  File "/home/vstinner/python/3.13/Lib/unittest/case.py", line 707 in __call__
  File "/home/vstinner/python/3.13/Lib/unittest/suite.py", line 122 in run
  File "/home/vstinner/python/3.13/Lib/unittest/suite.py", line 84 in __call__
  File "/home/vstinner/python/3.13/Lib/unittest/suite.py", line 122 in run
  File "/home/vstinner/python/3.13/Lib/unittest/suite.py", line 84 in __call__
  File "/home/vstinner/python/3.13/Lib/unittest/suite.py", line 122 in run
  File "/home/vstinner/python/3.13/Lib/unittest/suite.py", line 84 in __call__
  File "/home/vstinner/python/3.13/Lib/unittest/runner.py", line 240 in run
  File "/home/vstinner/dev/cython/runtests.py", line 2927 in runtests
  File "/home/vstinner/dev/cython/runtests.py", line 2640 in runtests_callback
  File "/home/vstinner/python/3.13/Lib/multiprocessing/pool.py", line 125 in worker
  File "/home/vstinner/python/3.13/Lib/multiprocessing/process.py", line 108 in run
  File "/home/vstinner/python/3.13/Lib/multiprocessing/process.py", line 313 in _bootstrap
  File "/home/vstinner/python/3.13/Lib/multiprocessing/popen_fork.py", line 74 in _launch
  File "/home/vstinner/python/3.13/Lib/multiprocessing/popen_fork.py", line 20 in __init__
  File "__init__", line ??? in __init__
  File "/home/vstinner/python/3.13/Lib/multiprocessing/context.py", line 282 in _Popen
  File "/home/vstinner/python/3.13/Lib/multiprocessing/process.py", line 121 in start
  File "/home/vstinner/python/3.13/Lib/multiprocessing/pool.py", line 329 in _repopulate_pool_static
  File "/home/vstinner/python/3.13/Lib/multiprocessing/pool.py", line 306 in _repopulate_pool
  File "/home/vstinner/python/3.13/Lib/multiprocessing/pool.py", line 215 in __init__
  File "/home/vstinner/python/3.13/Lib/multiprocessing/context.py", line 119 in Pool
  File "/home/vstinner/dev/cython/runtests.py", line 2479 in main
  File "/home/vstinner/dev/cython/runtests.py", line 2978 in <module>

Extension modules: cython.cimports.libc.math, refnanny (total: 2)
ok

and another assertion error:

[6] compiling (c/cy2) and running async_def ... [8] ['/home/vstinner/python/3.13/python', 'setup.py', 'build_ext', '--inplace']
Compiling dealloc_raise.pyx because it changed.
[1/1] Cythonizing dealloc_raise.pyx

/home/vstinner/dev/cython/Cython/Compiler/Main.py:373: FutureWarning: Cython directive 'language_level' not set, using '3' (Py3). This has changed from earlier releases! File: /home/vstinner/dev/cython/TEST_TMP/8/run/dealloc_raise/dealloc_raise.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)


[8] ['/home/vstinner/python/3.13/python', 'test_dealloc_raise1.py']

Objects/dictobject.c:699: _PyDict_CheckConsistency: Assertion failed: mp->ma_values->valid == 1
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7f5edac76330
object refcount : 1
object type     : 0x9039e0
object type name: dict
object repr     : {}

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007f5ee89e2740 (most recent call first):
  File "/home/vstinner/dev/cython/TEST_TMP/8/run/dealloc_raise/test_dealloc_raise1.py", line 10 in <module>

Extension modules: dealloc_raise (total: 1)


Final directory layout of 'dealloc_raise':
./setup.py
./dealloc_raise.pyx
./test_dealloc_raise1.py
./test_dealloc_raise2.py
./dealloc_raise.c
./dealloc_raise.cpython-313d-x86_64-linux-gnu.so
./build/temp.linux-x86_64-cpython-313-pydebug/dealloc_raise.o
./build/lib.linux-x86_64-cpython-313-pydebug/dealloc_raise.cpython-313d-x86_64-linux-gnu.so

FAIL

vstinner · 2024-07-08T13:12:23Z

To run the test suite, see: https://github.com/cython/cython/wiki/HackerGuide#the-test-suite

The doc says "To select a specific test (or a set of tests), just pass the name(s) as parameters."

If I ran runtests.py 'methodmangling_T5', no test is run. Or at least, I cannot reproduce the Fatal Error.

vstinner · 2024-07-11T14:52:17Z

@scoder @da-woods: It would help if you can explain how to write a reproducer which doesn't need runtests.py, or if you can provide a reproducer.

da-woods · 2024-07-11T15:33:14Z

I also failed to reproduce it on my own laptop so I'm not sure I can really help. I don't really know what's failing and what I'm trying to reproduce.

runtests.py methodmangling_T5

This should be the right way to run it. I'd add -vv (tells you want tests it's running), and --no-cleanup --no-cleanup-shared (leaves the generated files in TEST_TMP to inspect later outside runtests).

Sorry not to be more useful... This bug beat me too.

vstinner · 2024-07-11T16:38:38Z

I don't reproduce the crash if I only run methodmangling_T5.

da-woods · 2024-07-11T19:51:44Z

I made some progress:

on Windows (using the official installer) it crashes semi-reliably (most but not all of the time) the first time it runs (that's running just python runtests.py methodmangling_T5 -vv). The second and subsequent times it runs it doesn't crash.

If I clear all the .pyc files from the cython directory then it crashes again.

I wasn't able to find much useful from trying to debug it, but it does seem like it needs to generate pyc files at least on Windows.

(Don't know right now if this applies to other platforms - I suspect it does but haven't tried)

vstinner · 2024-07-16T16:37:05Z

If I run python runtests.py methodmangling_T5 -vv on a Python 3.13 debug build on Windows, I get the error: _Py_NegativeRefcount: Assertion failed: object has negative ref count.

(...)
Doctest: methodmangling_T5.can_find_global_arg ... ok
cant_find_global_arg (methodmangling_T5)
Doctest: methodmangling_T5.cant_find_global_arg ... ok
run (__main__.PureDoctestTestCase.run)
[-1] running pure doctests in methodmangling_T5 ... C (methodmangling_T5)
Doctest: methodmangling_T5.C ... ok
f1 (methodmangling_T5.CMultiplyNested)
Doctest: methodmangling_T5.CMultiplyNested.f1 ... C:\victor\python\3.13\Python\generated_cases.c.h:5710: _Py_NegativeRefcount: Assertion failed: object has negative ref count
<object at 000002075BEB9FD0 is freed>
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Thread 0x0000124c (most recent call first):
  File "C:\victor\cython\runtests.py", line 2589 in time_stamper
  File "C:\victor\python\3.13\Lib\threading.py", line 990 in run
  File "C:\victor\python\3.13\Lib\threading.py", line 1039 in _bootstrap_inner
  File "C:\victor\python\3.13\Lib\threading.py", line 1010 in _bootstrap

Current thread 0x00001264 (most recent call first):
  File "<doctest methodmangling_T5.CMultiplyNested.f1[1]>", line 1 in <module>
  File "C:\victor\python\3.13\Lib\doctest.py", line 1395 in __run
  File "C:\victor\python\3.13\Lib\doctest.py", line 1566 in run
  File "C:\victor\python\3.13\Lib\doctest.py", line 2320 in runTest
  File "C:\victor\python\3.13\Lib\unittest\case.py", line 606 in _callTestMethod
  File "C:\victor\python\3.13\Lib\unittest\case.py", line 651 in run
  File "C:\victor\python\3.13\Lib\unittest\case.py", line 707 in __call__
  File "C:\victor\python\3.13\Lib\unittest\suite.py", line 122 in run
  File "C:\victor\cython\runtests.py", line 1644 in run
  File "C:\victor\python\3.13\Lib\unittest\case.py", line 707 in __call__
  File "C:\victor\python\3.13\Lib\unittest\suite.py", line 122 in run
  File "C:\victor\python\3.13\Lib\unittest\suite.py", line 84 in __call__
  File "C:\victor\python\3.13\Lib\unittest\suite.py", line 122 in run
  File "C:\victor\python\3.13\Lib\unittest\suite.py", line 84 in __call__
  File "C:\victor\python\3.13\Lib\unittest\suite.py", line 122 in run
  File "C:\victor\python\3.13\Lib\unittest\suite.py", line 84 in __call__
  File "C:\victor\python\3.13\Lib\unittest\runner.py", line 240 in run
  File "C:\victor\cython\runtests.py", line 2927 in runtests
  File "C:\victor\cython\runtests.py", line 2514 in main
  File "C:\victor\cython\runtests.py", line 2978 in <module>

Extension modules: _wmi, cython.cimports.libc.math, refnanny (total: 3)

vstinner · 2024-07-16T17:05:19Z

I reproduced the methodmangling_T5 bug on Linux: I created #121863 for this bug.

vstinner · 2024-07-16T17:08:29Z

Objects/dictobject.c:699: _PyDict_CheckConsistency: Assertion failed: mp->ma_values->valid == 1

This bug is tracked as #121860 which has no reproducer yet.

da-woods · 2024-07-17T15:57:29Z

At some point I'll try get the Cython test suite running against a debug build of Python as part of our regular CI runs (we already do some runs with Py_DEBUG defined but that only affects macros and inline functions).

It's needs these versions to be available on GitHub actions easily.

vstinner · 2024-07-17T18:02:26Z

It's needs these versions to be available on GitHub actions easily.

Python CI clones the Git repository and builds Python from scratch, it takes 60 seconds in total. You can clone a specific Git tag.

vstinner · 2024-08-09T14:02:44Z

@da-woods: What's the status of this issue with Python 3.13 rc1?

scoder · 2024-08-09T14:33:20Z

Py3.13rc1 runs fine on our side. Thanks!

scoder added type-crash A hard crash of the interpreter, possibly with a core dump 3.13 bugs and security fixes labels Jul 2, 2024

vstinner mentioned this issue Jul 16, 2024

Crash on _PyDict_CheckConsistency with an empty instance dict #121860

Closed

scoder closed this as completed Aug 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

3.13b3 crashes in Cython test suite #121253

3.13b3 crashes in Cython test suite #121253

scoder commented Jul 2, 2024 •

edited by github-actions bot

Loading

vstinner commented Jul 5, 2024

vstinner commented Jul 5, 2024

scoder commented Jul 6, 2024 via email

vstinner commented Jul 6, 2024

vstinner commented Jul 8, 2024

vstinner commented Jul 11, 2024

da-woods commented Jul 11, 2024

vstinner commented Jul 11, 2024

da-woods commented Jul 11, 2024 •

edited

Loading

vstinner commented Jul 16, 2024

vstinner commented Jul 16, 2024

vstinner commented Jul 16, 2024

da-woods commented Jul 17, 2024

vstinner commented Jul 17, 2024

vstinner commented Aug 9, 2024

scoder commented Aug 9, 2024

3.13b3 crashes in Cython test suite #121253

3.13b3 crashes in Cython test suite #121253

Comments

scoder commented Jul 2, 2024 • edited by github-actions bot Loading

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

vstinner commented Jul 5, 2024

vstinner commented Jul 5, 2024

scoder commented Jul 6, 2024 via email

vstinner commented Jul 6, 2024

vstinner commented Jul 8, 2024

vstinner commented Jul 11, 2024

da-woods commented Jul 11, 2024

vstinner commented Jul 11, 2024

da-woods commented Jul 11, 2024 • edited Loading

vstinner commented Jul 16, 2024

vstinner commented Jul 16, 2024

vstinner commented Jul 16, 2024

da-woods commented Jul 17, 2024

vstinner commented Jul 17, 2024

vstinner commented Aug 9, 2024

scoder commented Aug 9, 2024

scoder commented Jul 2, 2024 •

edited by github-actions bot

Loading

da-woods commented Jul 11, 2024 •

edited

Loading