Python seg fault at startup while using scikit-learn

[RatkoJ]

Environment data

• VS Code version: 1.28.2
• Python Extension version (available under the Extensions sidebar): 2018.9.2
• Microsoft Python Language Server version 0.1.50.0
• OS and version: Mac OS 10.14
• Python version (& distribution if applicable, e.g. Anaconda): 3.6.2 and 3.7
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
• Relevant/affected Python packages and their versions: scikit-learn 0.20

Actual behavior

Mac OS throws an error "Python quit unexpectedly" upon startup of vscode and a project that is using sklearn. Posting this here because crash report says parent process is Microsoft Language Server.

Expected behavior

No error.

Steps to reproduce:

Make a file with the following:

from sklearn.cluster import KMeans

if __name__=="__main__":
    km = KMeans()

Open it in a fresh run of vscode
Wait 30 secs... Python process crashes but vscode is still running and then everything works normally afterwards.

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

none

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help)

none

Mac OS Crash Report (incomplete):

Process:               Python [97430]
Path:                  /Library/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python
Identifier:            Python
Version:               3.6.2 (3.6.2)
Code Type:             X86-64 (Native)
Parent Process:        Microsoft.Python.LanguageServer [96518]
Responsible:           Python [97430]
User ID:               501

Date/Time:             2018-11-02 14:15:18.028 +0100
OS Version:            Mac OS X 10.14 (18A391)
Report Version:        12
Anonymous UUID:        D66954B1-5A32-38C1-E910-41CD6726D5DF

Sleep/Wake UUID:       0F04EC0C-0277-4914-8CF4-DFBBE7ECC81F

Time Awake Since Boot: 570000 seconds
Time Since Wake:       220 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000008
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [97430]

VM Regions Near 0x8:
--> 
    __TEXT                 0000000100000000-0000000100001000 [    4K] r-x/rwx SM=COW  /Library/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   _k_means_elkan.cpython-36m-darwin.so	0x000000011cddb2bf __pyx_pf_7sklearn_7cluster_14_k_means_elkan_12__defaults__ + 47
1   sparsefuncs_fast.cpython-36m-darwin.so	0x0000000113ccb3c0 __Pyx_CyFunction_get_kwdefaults + 48
2   org.python.python             	0x0000000100767500 _PyObject_GenericGetAttrWithDict + 144
3   org.python.python             	0x00000001007e84bc builtin_getattr + 92
4   org.python.python             	0x00000001007637c8 _PyCFunction_FastCallDict + 552
5   org.python.python             	0x00000001007ec5e4 call_function + 612
6   org.python.python             	0x00000001007edfe4 _PyEval_EvalFrameDefault + 5604
7   org.python.python             	0x00000001007ec32e fast_function + 606
8   org.python.python             	0x00000001007ec5cb call_function + 587
9   org.python.python             	0x00000001007edfe4 _PyEval_EvalFrameDefault + 5604
10  org.python.python             	0x00000001007eb9a0 _PyEval_EvalCodeWithName + 2720
11  org.python.python             	0x00000001007ec1ab fast_function + 219
12  org.python.python             	0x00000001007ec5cb call_function + 587
13  org.python.python             	0x00000001007ee506 _PyEval_EvalFrameDefault + 6918

[jakebailey]

See also microsoft/vscode-python#2693. I can't seem to get macOS to show anything about an unexpected error, but I can see that the last thing in AnalysisLog.txt is the scraper inspecting _k_means_elkan, and that stubs we produce in the languageServer directory don't include the k-means stuff (which hints that it's not completing).

I'm not sure that is a problem with anything we've written. It's the library that's segfaulting, and it's only under us in the process tree because the language server sometimes needs to call out to python to inspect things. The scraper could be doing something wrong, since that k-means stuff is compiled, but we'd probably need a better stack trace to know where it came from.

$ tail AnalysisLog.txt
[00:00:26.9818900] ImportNotFound: distributed.get_client
[00:00:26.9839210] ImportNotFound: pytest
[00:00:26.9843510] ImportNotFound: sklearn.externals.joblib.pytest
[00:00:26.9856830] ImportNotFound: _pytest[00:00:26.9860870] ImportNotFound: sklearn.externals.joblib._pytest
[00:00:27.0034040] Scrape: /Users/jake/skseg/venv/bin/python, -B -E /Users/jake/.vscode/extensions/ms-python.python-2018.9.2/languageServer.0.1.53/scrape_module.py -u8 sklearn.utils.sparsefuncs_fast /Users/jake/skseg/venv/lib/python3.7/site-packages
[00:00:27.4519410] WriteCachedModule: /Users/jake/.vscode/extensions/ms-python.python-2018.9.2/languageServer.0.1.53/BsZ2KoVvO-7FXDPEIZGCp8UhjkLGaOh9fFNQ9UoaoF4=/sparsefuncs_fast.cpython-37m-darwin.pyi
[00:00:27.4709640] Scrape: /Users/jake/skseg/venv/bin/python, -B -E /Users/jake/.vscode/extensions/ms-python.python-2018.9.2/languageServer.0.1.53/scrape_module.py -u8 scipy.spatial.ckdtree /Users/jake/skseg/venv/lib/python3.7/site-packages
[00:00:27.8659110] WriteCachedModule: /Users/jake/.vscode/extensions/ms-python.python-2018.9.2/languageServer.0.1.53/SI_WNaKcgW6v9uGrg3UToUEqu-9Fp2F15HjG-GZGDC0=/ckdtree.cpython-37m-darwin.pyi
[00:00:27.9014190] Scrape: /Users/jake/skseg/venv/bin/python, -B -E /Users/jake/.vscode/extensions/ms-python.python-2018.9.2/languageServer.0.1.53/scrape_module.py -u8 sklearn.cluster._k_means_elkan /Users/jake/skseg/venv/lib/python3.7/site-packages

$ rg 'ckdtree'        
AnalysisLog.txt640:[00:00:27.4709640] Scrape: /Users/jake/skseg/venv/bin/python, -B -E /Users/jake/.vscode/extensions/ms-python.python-2018.9.2/languageServer.0.1.53/scrape_module.py -u8 scipy.spatial.ckdtree /Users/jake/skseg/venv/lib/python3.7/site-packages
641:[00:00:27.8659110] WriteCachedModule: /Users/jake/.vscode/extensions/ms-python.python-2018.9.2/languageServer.0.1.53/SI_WNaKcgW6v9uGrg3UToUEqu-9Fp2F15HjG-GZGDC0=/ckdtree.cpython-37m-darwin.pyi
SI_WNaKcgW6v9uGrg3UToUEqu-9Fp2F15HjG-GZGDC0=/ckdtree.cpython-37m-darwin.pyi
6:__file__ = '/Users/jake/skseg/venv/lib/python3.7/site-packages/scipy/spatial/ckdtree.cpython-37m-darwin.so'
7:__name__ = 'scipy.spatial.ckdtree'

$ rg 'elkan'  
AnalysisLog.txt
642:[00:00:27.9014190] Scrape: /Users/jake/skseg/venv/bin/python, -B -E /Users/jake/.vscode/extensions/ms-python.python-2018.9.2/languageServer.0.1.53/scrape_module.py -u8 sklearn.cluster._k_means_elkan /Users/jake/skseg/venv/lib/python3.7/site-packages %  ~/./e/m/languageServer.0.1.53  

[MikhailArkhipov]

Python crash, not LS. LS is managed code and does not normally run any Python except when to inspect Python types not available in source otherwise.

[jakebailey]

I can reproduce this and grab a stack trace with faulthandler:

$ python -X faulthandler -B -E /Users/jake/.vscode/extensions/ms-python.python-2018.9.2/languageServer.0.1.53/scrape_module.py -u8 sklearn.cluster._k_means_elkan /Users/jake/skseg/venv/lib/python3.7/site-packages
Fatal Python error: Segmentation fault

Current thread 0x00007fffab708380 (most recent call first):
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/inspect.py", line 1845 in _signature_is_functionlike
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/inspect.py", line 2274 in _signature_from_callable
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/inspect.py", line 1119 in getfullargspec
  File "/Users/jake/.vscode/extensions/ms-python.python-2018.9.2/languageServer.0.1.53/scrape_module.py", line 367 in _init_argspec_fromargspec
  File "/Users/jake/.vscode/extensions/ms-python.python-2018.9.2/languageServer.0.1.53/scrape_module.py", line 304 in __init__
  File "/Users/jake/.vscode/extensions/ms-python.python-2018.9.2/languageServer.0.1.53/scrape_module.py", line 708 in __init__
  File "/Users/jake/.vscode/extensions/ms-python.python-2018.9.2/languageServer.0.1.53/scrape_module.py", line 1002 in _collect_members
  File "/Users/jake/.vscode/extensions/ms-python.python-2018.9.2/languageServer.0.1.53/scrape_module.py", line 913 in collect_top_level_members
  File "/Users/jake/.vscode/extensions/ms-python.python-2018.9.2/languageServer.0.1.53/scrape_module.py", line 1490 in <module>
[1]    2061 segmentation fault  python -X faulthandler -B -E  -u8 sklearn.cluster._k_means_elkan

As far as I can tell, we're not doing anything bad in how we're calling into the inspect library. This seems like an upstream issue (I think something bad in scikit, based on the more-exact stack trace with the C libs in the original post).

At the very least, you shouldn't have trouble using the editor with incomplete data. In terms of preventing some macOS error prompt, I'm not sure we can do much. The language server already recognizes that the scraper crashed (look for ExitCode 139 in AnalysisLog.txt), and macOS displaying something extra is something separate.

[RatkoJ]

OK, understood.

But it is strange that when the language server restarts itself after this crash, it runs just fine the second time around and IntelliSense works for this package and KMeans object just fine. Not sure why the first time around a crash occurs.

Edit: just read the linked vscode-python issue. As a note, I've tried with two separate Python installs and two different versions of scikit so I think that reduces the chances of it being a corrupt install. But still could be in scikit.

[jakebailey]

Hmm, the language server itself shouldn't be crashing, and I didn't see that behavior on my machine. At worst, the python process it runs will crash, we see that there's an error code set, and move on. From your crash report, I think it's only saying that the parent process of the crashing process was the language server, not that the language server itself is crashing. (There should be a message in the Python output panel if the langauge server restarts, too.)

[RatkoJ]

Yes, you are correct. I just checked the output panel at trace level and the server never restarts. It is just the python process. I also confirmed that via the macOS Activity Monitor. There are a bunch of python processes showing up and disappearing but the language server is always there.

[jakebailey]

Yep, that's normal (besides the crash). For things we can't inspect directly (e.g. Python libraries written in C, even many standard library modules) we have to spawn a Python process to generate something we can use (stubs). This is definitely the case for many scientific/numerical libraries which implement things in C for speed, or call out to system libraries like GMP.

If the scraper crashes, then we won't have that type information and won't have accurate info in some cases, but it won't be fatal to the language server.