Inspection of some cython produced functions causes segfault

[jakebailey]

Code Sample, a copy-pastable example if possible

from pandas._libs.algos import backfill
getattr(backfill, "__kwdefaults__", None)

Run with -X faulthandler to get more info.

Problem description

Trying to access __kwdefaults__, either by getattr or hasattr, segfaults the Python process. This occurs at least on macOS when pandas is installed from PyPI using pip install pandas. If it is instead installed via pip install pandas --no-binary :all:, then it does not segfault. There may be other functions which do this, but this is the first we encounter.

This impacts our Python language server as it uses the inspect library to examine libraries without Python source. The code sample above is a minimal repro, but in reality it's being called by inspect.getfullargspec() (which eventually does this). When it segfaults, our process crashes (and on some OSs like macOS produces a visible popup as the OS is tracking these sorts of crashes). See: microsoft/python-language-server#740

cython/cython#1470 looks to be related, and would be fixed in Cython 0.29.6, so maybe a version bump is all that would be required. Given I can do --no-binary to do the compilation locally, it may be that what is building for PyPI is older than what happens on my machine.

Expected Output

Anything, just not crash. Here's what macOS's crash reporter says:

Process: Python [80749] Path: /usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python Identifier: Python Version: 3.7.3 (3.7.3) Code Type: X86-64 (Native) Parent Process: zsh [72384] Responsible: Python [80749] User ID: 501

Date/Time: 2019-04-17 16:18:22.448 -0700
OS Version: Mac OS X 10.14.4 (18E226)
Report Version: 12
Bridge OS Version: 3.0 (14Y674)
Anonymous UUID: 5A957B3E-4E8F-3DE2-C606-5B11FE48E6DD

Sleep/Wake UUID: 02FDA72B-8D53-471B-80AE-6514E0B386FB

Time Awake Since Boot: 23000 seconds
Time Since Wake: 2200 seconds

System Integrity Protection: disabled

Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY

Termination Signal: Segmentation fault: 11
Termination Reason: Namespace SIGNAL, Code 0xb
Terminating Process: exc handler [80749]

VM Regions Near 0:
-->
__TEXT 000000010a32a000-000000010a32c000 [ 8K] r-x/rwx SM=COW /usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 algos.cpython-37m-darwin.so 0x0000000116045520 pyx_pf_6pandas_5_libs_5algos_600__defaults + 32
1 algos.cpython-37m-darwin.so 0x0000000116195ed0 __Pyx_CyFunction_get_kwdefaults + 48
2 org.python.python 0x000000010a352587 getset_get + 58
3 org.python.python 0x000000010a379c2f _PyObject_GenericGetAttrWithDict + 180
4 org.python.python 0x000000010a379b20 _PyObject_LookupAttr + 166
5 org.python.python 0x000000010a3d6d66 builtin_getattr + 141
6 org.python.python 0x000000010a34d6f2 _PyMethodDef_RawFastCallKeywords + 495
7 org.python.python 0x000000010a34cc8e _PyCFunction_FastCallKeywords + 44
8 org.python.python 0x000000010a3e1db2 call_function + 636
9 org.python.python 0x000000010a3dac35 _PyEval_EvalFrameDefault + 6594
10 org.python.python 0x000000010a3e26d3 _PyEval_EvalCodeWithName + 1867
11 org.python.python 0x000000010a3d91d0 PyEval_EvalCode + 51
12 org.python.python 0x000000010a40779b run_mod + 54
13 org.python.python 0x000000010a4067c5 PyRun_FileExFlags + 163
14 org.python.python 0x000000010a405e6b PyRun_SimpleFileExFlags + 263
15 org.python.python 0x000000010a41e9b0 pymain_main + 5367
16 org.python.python 0x000000010a41f088 _Py_UnixMain + 56
17 libdyld.dylib 0x00007fff5dead3d5 start + 1

Thread 1:
0 libsystem_kernel.dylib 0x00007fff5dfe586a __psynch_cvwait + 10
1 libsystem_pthread.dylib 0x00007fff5e09e56e _pthread_cond_wait + 722
2 libopenblasp-r0.3.5.dev.dylib 0x000000010af14a3b blas_thread_server + 619
3 libsystem_pthread.dylib 0x00007fff5e09b2eb _pthread_body + 126
4 libsystem_pthread.dylib 0x00007fff5e09e249 _pthread_start + 66
5 libsystem_pthread.dylib 0x00007fff5e09a40d thread_start + 13

Thread 2:
0 libsystem_kernel.dylib 0x00007fff5dfe586a __psynch_cvwait + 10
1 libsystem_pthread.dylib 0x00007fff5e09e56e _pthread_cond_wait + 722
2 libopenblasp-r0.3.5.dev.dylib 0x000000010af14a3b blas_thread_server + 619
3 libsystem_pthread.dylib 0x00007fff5e09b2eb _pthread_body + 126
4 libsystem_pthread.dylib 0x00007fff5e09e249 _pthread_start + 66
5 libsystem_pthread.dylib 0x00007fff5e09a40d thread_start + 13

Thread 3:
0 libsystem_kernel.dylib 0x00007fff5dfe586a __psynch_cvwait + 10
1 libsystem_pthread.dylib 0x00007fff5e09e56e _pthread_cond_wait + 722
2 libopenblasp-r0.3.5.dev.dylib 0x000000010af14a3b blas_thread_server + 619
3 libsystem_pthread.dylib 0x00007fff5e09b2eb _pthread_body + 126
4 libsystem_pthread.dylib 0x00007fff5e09e249 _pthread_start + 66
5 libsystem_pthread.dylib 0x00007fff5e09a40d thread_start + 13

Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x000000010a75fd68 rcx: 0x000000010a75fd50 rdx: 0x00000001177329c0
rdi: 0x000000010a75fd80 rsi: 0x0000000000000000 rbp: 0x00007ffee58d49e0 rsp: 0x00007ffee58d49d0
r8: 0x1678a558da7df69e r9: 0x00007ffee58d4a90 r10: 0x00007fe0bb081a68 r11: 0x00007ffee58d4ae8
r12: 0x0000000116209a08 r13: 0x00000001161a8698 r14: 0x0000000116209a08 r15: 0x0000000000000000
rip: 0x0000000116045520 rfl: 0x0000000000010202 cr2: 0x0000000000000000

Logical CPU: 6
Error Code: 0x00000004
Trap Number: 14

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.7.3.final.0
python-bits: 64
OS: Darwin
OS-release: 18.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.2
pytest: None
pip: 19.0.3
setuptools: 40.8.0
Cython: None
numpy: 1.16.2
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.8.0
pytz: 2019.1
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

[jakebailey]

See also: scikit-learn/scikit-learn#13667

[jbrockmendel]

Yikes. To clarify, the working hypothesis is that this is a problem in cython that can be fixed by bumping the dependency version? If so, should be an easy PR.

[jakebailey]

I think so, yes, though I'm not familiar enough with what goes on during --no-binary to see if I'm getting a newer version than what's building and pushing to PyPI. When I do the --no-binary, I don't end up with cython in my environment, so I'm not sure if I'm effectively just building from the already generated C files downloaded, or a completely new set of generated files from a different cython version.

[jbrockmendel]

I just tried this with cython 0.29.6 and got the segfault on ubuntu but not on OSX (py36 on ubuntu, py37 on OSX).

cc: @scoder is this a known issue?

update reported an incorrect cython version on ubuntu, was actually 0.28.5. Just upgraded and will update this report when recompile is complete.

[jakebailey]

Another possible indication that this is a Cython thing:

If I do pip download pandas, extract the pandas wheel with unzip, then do strings pandas/_libs/algos.cpython-37m-darwin.so | grep cython, I see a string that says "_cython_0_28_2".

If I instead do pip download pandas --no-binary :all:, then go look at the generated source for _libs/algos.c, I can see that the Cython version there is 0.29.6.

[scoder]

Yes, Cython 0.29.6 prevents this crash.

[jbrockmendel]

Update: when actually using 0.29.7, there is no segfault on ubuntu.

@jakebailey if you want to make a PR with tests to ensure this doesn't show up elsewhere, it would be welcome. Possibly looping over many/all cython functions/attributes

(along with bumping the cython dependency)

[scoder]

(Ah, yes, use 0.29.7, definitely. It fixes another crash bug.)

[jakebailey]

Sure, I'll take a look. Thanks for confirming this!

[jakebailey]

My PR for this (#26140) was never merged, but the cython version was eventually updated past the broken version. I'll close this since I believe it's fixed at this point.