Stack overflow when importing GPy

[vmasrani]

Environment data

VS Code version: 1.37.1
Microsoft Python Language Server version: 0.3.66.0
Extension version (available under the Extensions sidebar):2019.8.30787
OS and version: MacOs Mojave 10.14.5 (18F132)
Python version (& distribution if applicable, e.g. Anaconda): Anaconda 3.7.3
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
Relevant/affected Python packages and their versions: GPy https://sheffieldml.github.io/GPy/
Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info microsoft/vscode-python#3977): python.jediEnabled：false

Expected behaviour

Language server doesn't hang on 'Analyzing in background'

Actual behaviour

CPU usage is not high but process keeps terminating due to StackOverflowException. Then the 'The Python Tools server crashed 5 times in the last 3 minutes. The server will not be restarted.' warning appears . I've isolated the problem to be due to importing the GPy package (problem goes away if I comment out the imports). I've also tried adding the '# noqa' tag after the imports and it doesn't work. Is it possible to allow users to specify packages for the language server to ignore?

Steps to reproduce:

Open a blank python file and import gpy

Logs

User belongs to experiment group 'AlwaysDisplayTestExplorer - experiment'
User belongs to experiment group 'ShowPlayIcon - control'
User belongs to experiment group 'ShowExtensionSurveyPrompt - control'
Starting Microsoft Python language server.

conda --version
pyenv root
python3.7 -c "import sys;print(sys.executable)"
python3.6 -c "import sys;print(sys.executable)"
python3 -c "import sys;print(sys.executable)"
python2 -c "import sys;print(sys.executable)"
python -c "import sys;print(sys.executable)"
~/miniconda3/envs/ml3/bin/python -c "import sys;print(sys.executable)"
conda info --json
conda env list
conda info --json
~/miniconda3/envs/ml3/bin/python -m pylint --errors-only --enable=W0614 "--generated-members=numpy.* ,torch.* ,cv2.* , cv." --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text ~/dev/workspace/test.py
cwd: ~/dev/workspace
~/miniconda3/envs/ml3/bin/python -m pylint --errors-only --enable=W0614 "--generated-members=numpy. ,torch.* ,cv2.* , cv." --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text ~/dev/workspace/test.py
cwd: ~/dev/workspace
conda env list
~/miniconda3/envs/ml3/bin/python -m pylint --errors-only --enable=W0614 "--generated-members=numpy. ,torch.* ,cv2.* , cv." --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text ~/dev/workspace/test.py
cwd: ~/dev/workspace
~/miniconda3/envs/ml3/bin/python -m pylint --errors-only --enable=W0614 "--generated-members=numpy. ,torch.* ,cv2.* , cv.*" --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text ~/dev/workspace/test.py
cwd: ~/dev/workspace
##########Linting Output - pylint##########
##########Linting Output - pylint##########
[Info - 2:51:34 PM] Analysis cache path: /Users/vmasrani/Library/Caches/Microsoft/Python Language Server
[Info - 2:51:34 PM] GetCurrentSearchPaths /Users/vmasrani/miniconda3/envs/ml3/bin/python
[Info - 2:51:34 PM] Interpreter search paths:
[Info - 2:51:34 PM] /private/var/folders/k4/mvr6htsd0kg3wvx5vzttctg00000gr/T/ymeysyn1.vjb
[Info - 2:51:34 PM] /Users/vmasrani/miniconda3/envs/ml3/lib/python3.7
[Info - 2:51:34 PM] /Users/vmasrani/miniconda3/envs/ml3/lib/python3.7/lib-dynload
[Info - 2:51:34 PM] /Users/vmasrani/.python
[Info - 2:51:34 PM] /Users/vmasrani/miniconda3/envs/ml3/lib/python3.7/site-packages
[Info - 2:51:34 PM] User search paths:
[Info - 2:51:38 PM] Microsoft Python Language Server version 0.3.66.0
[Info - 2:51:38 PM] Initializing for /Users/vmasrani/miniconda3/envs/ml3/bin/python
conda info --json
Process is terminating due to StackOverflowException.
[Info - 2:52:04 PM] Connection to server got closed. Server will restart.
[Info - 2:52:04 PM] Analysis cache path: /Users/vmasrani/Library/Caches/Microsoft/Python Language Server
[Info - 2:52:04 PM] GetCurrentSearchPaths /Users/vmasrani/miniconda3/envs/ml3/bin/python
[Info - 2:52:05 PM] Interpreter search paths:
[Info - 2:52:05 PM] /private/var/folders/k4/mvr6htsd0kg3wvx5vzttctg00000gr/T/zcsujvkk.oig
[Info - 2:52:05 PM] /Users/vmasrani/miniconda3/envs/ml3/lib/python3.7
[Info - 2:52:05 PM] /Users/vmasrani/miniconda3/envs/ml3/lib/python3.7/lib-dynload
[Info - 2:52:05 PM] /Users/vmasrani/.python
[Info - 2:52:05 PM] /Users/vmasrani/miniconda3/envs/ml3/lib/python3.7/site-packages
[Info - 2:52:05 PM] User search paths:
[Info - 2:52:08 PM] Microsoft Python Language Server version 0.3.66.0
[Info - 2:52:08 PM] Initializing for /Users/vmasrani/miniconda3/envs/ml3/bin/python
Process is terminating due to StackOverflowException.
[Info - 2:52:44 PM] Connection to server got closed. Server will restart.

[jakebailey]

noqa only affects the reporting of diagnostics, and does not otherwise affect the analysis. We don't have a way to specifically exclude modules, no. (Was #898 but that was refocused.)

[jdwestwood]

I encountered the StackOverflowException when using the Flask_WTF (0.14.2) package. In my case I was able to isolate the cause.

First, the basic structure of the code in the form.py module in Flask_WTF is:

from wtforms import Form

class FlaskForm(Form):
    # Class definition ...
    pass

# In prior versions of Flask_WTF, the class FlaskForm was called Form.
# flask_wtf.Form is defined for backward compatibility.  The symbol
# 'Form' from the import is overwritten by the definition of the
# 'Form' class here.
class Form(FlaskForm):
    """
    .. deprecated:: 0.13
        Renamed to :class:`~flask_wtf.FlaskForm`.
    """
    # Class definition ...
    pass

Next, my code (greatly simplified):

from flask_wtf import FlaskForm

class MyForm(FlaskForm):
    # The argument of the lambda function is a list, for example.
    _a = lambda x: x[0]
    # ('_a' is defined for convenience, and in my actual code is more complicated; it is
    # used multiple times in defining other class variables.  It is not intended to be a
    # class or static method).

Two things are required to generate the StackOverflowException:

1. The redefinition of Form after importing it in the flask_wtf.form module.
2. The particular lambda function in MyForm. Other functions such as lambda x: x do not cause the exception.

[jakebailey]

The first is definitely the same as what's been observed for GPy (and what the ongoing PR addresses).

[jdwestwood]

OK, but just to be clear, no. 1 (the redefinition of Form in the form.py module) by itself is not sufficient to cause the exception. Neither is the no. 2 (the particular form of the _a in a class definition). Both must be present.

[MikhailArkhipov]

Yes, the underlying cause is endless recursion in walking class bases since they are circular.