Segfault in scraper while inspecting some pyx-produced libraries (pandas, sklearn) #740
Comments
|
This is not LS, LS is .NET process, no native code. This looks like Python 3.7 itself. Would be useful to figure command line (i.e. which module we asked Python to introspect). However, no matter module, command like, script we passed Python should not be crashing. I'd recommend passing this to Python team as well. @brettcannon can suggest where to report. |
MikhailArkhipov
changed the title
|
I see this as the parent process: |
|
Yes, LS launched Python but it has no control of its behavior. We pass Python script and Python executable is crashing with segfault (aka null reference or buffer overflow) when executing it. First, no script should be able to crash Python itself since it is, in fact, a security hole. Consider possibility of SQL query crashing SQL server. Second, the script is unable to null ref itself as it is Python which is a managed environment and not down-to-metal C/C++. |
|
This is what I described in #713. |
|
#341 too, though we closed that without really solving it. A direct repro is there by executing |
|
https://bugs.python.org/issue36224 too (reported just last week), but was closed as not a Python problem. I have no idea what the solution to this would be; ignoring the root cause entirely, we can already technically catch crashes from the scraper, but the part where the OS (both macOS and Windows) show a visible message seems to be tied to the fact that those OSs know the processes are in a tree below a GUI application or similar. |
|
makes sense since I do use pandas and numpy in the project so I know there are some native libraries in there. Is it the case that maybe for libraries with native interfaces this doesn't work? |
|
Catching native crashes is a security no-no |
|
To be clear, by "catch" I meant "handle without crashing", i.e. we'd just get an incomplete result as the process ends early without writing all of the stub files. |
|
Yes, handling segfaults without crashing in native code is a security hole. |
The Python script that is crashing is the thing we use specifically to inspect compiled libraries, which often includes functions in numpy/scikit/etc. In this script, we don't directly access anything, but use Python's built-in inspect library to read information. That's what's crashing; Python is going down into the compiled code and breaking. Perhaps there's something in common with all of these functions which we could use to skip over them entirely. Since this is |
|
Chances are there is some extension module that isn't handling attribute access appropriately when the |
|
Unfortunately I've seen this in a wide range of libraries, numpy and scikit-learn included. |
|
Related to this: https://bugs.python.org/issue36224. This is seemingly from a .pyx (Cython) file. I'd guess this one: https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/algos.pyx. @mikebz, I'd try update your brew packages first. If that doesn't help, try using Anaconda distribution instead; probably a much safer build of pandas than what you would get from Homebrew. |
|
Yep, see: #740 (comment) |
|
Clarification on my setup. Homebrew is only installing base Python 3.7. Pandas and everything else is installed via PIP in VirtualEnv. If you guys want me to upgrade python? I already have the latest version: |
|
@mikebz make sure you are using the latest version of pandas, otherwise I would report the bug to pandas as this isn't a language server, VS Code, or Python bug. |
|
@mikebz I have encountered the same issue, and I got a solution: It seems that the pandas wheel is not compatible with Python 3.7.2 |
|
confirmation: I uninstalled pandas and numpy, I installed them with no-binary per @idaifish and so far I am not seeing any crashes. |
|
I'm going to reopen this; I want to do more investigation and see if we can work around potentially bad variables we don't need to look at. The large majority of people aren't going to be installing them |
|
I am facing the same bug, actually I can get expected result using debug, but the prompt still keeps remind me "python quit unexpectly", when I switch (From LanguageServer ) to Jedi, vscode can work fine. |
|
Has anyone who had run into this issue reported it upstream to pandas and/or Cython? |
|
A minimal repro (3.7 virtualenv on macOS, pip installed import inspect
from sklearn.cluster._k_means_elkan import k_means_elkan
inspect.getfullargspec(k_means_elkan) |
|
Similarly for: import inspect
from pandas._lib.algos import backfill
inspect.getfullargspec(backfill)
|
jakebailey
changed the title
jakebailey
changed the title
|
Narrowed it down even further: # This is wrong, see next comment.
# from sklearn.cluster._k_means_elkan import k_means_elkan
# getattr(k_means_elkan, '__code__', None) |
|
My mistake, should have been: from sklearn.cluster._k_means_elkan import k_means_elkan
getattr(k_means_elkan, '__kwdefaults__', None) # or hasattr(k_means_elkan, '__kwdefaults__')All of the accesses here except kwdefaults succeed, kwdefaults segfaults. The same for the pandas example. |
|
I have opened bugs on both pandas and scikit-learn:
This looks to me like cython/cython#1470. I'm going to work on some sort of blacklist for known-bad functions, for now. |
|
This was fixed in Cython 0.29.6 earlier this year. The two projects will probably use Cython 0.29.7 for their next releases to overcome this issue. |
|
I wrote a script to go through a library and figure out which values have inaccessible attributes. It was intended to be used for a blacklist in the scraper, but I found a somewhat okay way of detecting things (#995). But, here's that list for fun: 139 means it segfaulted and exited, 256 is some runtime error thrown when looking at the attributes. That's not so great either but the scraper is able to recover from those. |
I don't know if it is related, but I get exactly 3 "Python has stopped working" when I start VS Code and then open a folder containing python code. I don't use pandas or sklearn but it could be that my python 2.7.14 installation has other libraries which exhibit the same behavior. @jakebailey, The script to-go-through-a-library you mention have you shared that somewhere? Is it possible to see what my 3 crashes are related to, somewhere in a log or similar? |
|
Unfortunately I don't believe I have the script anymore; I think I lost it in a reimage of my Mac (where all of this was tested). Roughly, what the script did was import a library by name passed in an argument, then use inspect to iterate over its members. If successful, fork, then do a quick run over the next member's members and exit 0. If the subprocess was fine, then continue recursing. Then some bounds to prevent it from going back into the same thing over and over again which I can't recall after the 9 months (sorry!). |
|
You could enable trace logging and open it a few times to see which modules get scraped repeatedly. We cache the results, so if a compiled module is scraped again after a restart, then it's likely it failed. |
On my MacBook I continously get the crash below. I am not sure what causes. Seems like a background process.
The text was updated successfully, but these errors were encountered: