Re-analyzing whole project at every change

[thibault]

Hi,

A few days ago, I submitted issue #946. I'm glad to say that since latest updates, this issue is gone. A new issue appeared to take it's place, though…

Everytime I do the slightest change to any python file in the project (e.g it could be simply adding a space somewhere), the whole project is re-analyzed from scratch and I see the "analyzing in background" message flashing.

Now, the analysis is super quick, a few seconds vs. whole minutes a few weeks ago but since I'm constantly editing files, the whole project is analyzed over and over endlessly, and cpu usage is of course off the charts.

I could reproduce this issue with several different projects.

Environment data

Version: 1.33.1
Commit: 51b0b28134d51361cf996d2f0a1c698247aeabd8
Date: 2019-04-11T08:20:22.771Z
Electron: 3.1.6
Chrome: 66.0.3359.181
Node.js: 10.2.0
V8: 6.6.346.32
OS: Linux x64 4.15.0-46-generic

Python extension: 2019.3.6958
LS: 0.2.56.0

Starting output

Starting Microsoft Python language server.
[Info - 11:49:45] GetCurrentSearchPaths /home/thibault/.local/share/virtualenvs/aides-WKfJSWUb/bin/python
[Info - 11:49:45] Python search paths:
[Info - 11:49:45] /home/thibault/.local/share/virtualenvs/aides-WKfJSWUb/lib/python3.6
[Info - 11:49:45] /home/thibault/.local/share/virtualenvs/aides-WKfJSWUb/lib/python3.6/lib-dynload
[Info - 11:49:45] /usr/lib/python3.6
[Info - 11:49:45] /home/thibault/.local/share/virtualenvs/aides-WKfJSWUb/lib/python3.6/site-packages
[Info - 11:49:45] Configuration search paths:
[Info - 11:49:45] /home/thibault/.local/share/virtualenvs/aides-WKfJSWUb/lib/python36.zip
[Info - 11:49:45] /home/thibault/.local/share/virtualenvs/aides-WKfJSWUb/lib/python3.6
[Info - 11:49:45] /home/thibault/.local/share/virtualenvs/aides-WKfJSWUb/lib/python3.6/lib-dynload
[Info - 11:49:45] /usr/lib/python3.6
[Info - 11:49:45] /home/thibault/.local/share/virtualenvs/aides-WKfJSWUb/lib/python3.6/site-packages
[Info - 11:49:45] /home/thibault/Code/aides/src
[Info - 11:49:45] /home/thibault/.local/share/virtualenvs/aides-WKfJSWUb/bin
[Info - 11:49:46] Microsoft Python Language Server version 0.2.56.0
[Info - 11:49:46] Initializing for /home/thibault/.local/share/virtualenvs/aides-WKfJSWUb/bin/python
[Info - 11:49:49] Reloading modules...
[Info - 11:49:52] done.
[Info - 11:49:52] Analysis restarted.
[Info - 11:49:56] Reloading modules...
[Info - 11:49:59] done.
[Info - 11:49:59] Analysis restarted.

[CMLL]

I'm running into an issue that could be related to this so rather than opening a new issue going to pile on this one. Mine rather than being performance related is about syntax correctness.

Making a change triggers an analysis in the background and sometimes the syntax analysis will say that there are syntax errors that will linger until the analysis is complete, which in large projects can take a little while.

One example I was able to reproduce pretty constantly was if I wrote a text that the syntax analysis would consider as invalid, and delete it after the error mark would be set, the warning of an invalid variable would stay there if the code was deleted. It goes away as the triggered analysis finishes but sometimes when I would write a somewhat large piece of code the syntax errors would highlight stuff that had already been taken care of, or mid writing words that no longer exists.

I wonder if a possible solution to this would be limiting the scope of these rapid fire analysis that happen on every change to the single current document, and doing a large scale analysis either based on time or when the document gets saved. Don't know if either of those solutions are valid.

[MikhailArkhipov]

@thibault - Reloading modules is triggered when analysis engine thinks something has change in libraries. @jakebailey can comment more here - generally we take what Python reports as search paths that set from the root of the workspace in order to determine what is library and what is user code. I guess we miss something and your code is recognized as library so saving file triggers the reload.

@CMLL - this sounds like a different issue related to cancellation of running analysis after changes. @AlexanderSher may be interested.

[jakebailey]

This is just a general note, but the VSC extension recently updated and included some rewritten LS startup code; this should make the search paths behave much better (especially if you've used PYTHONPATH/extraPaths). Adding ./src or similar to one of those should now operate properly. Given this issue was originally a few weeks ago, it might be fixed if you're using the stable extension.

[MikhailArkhipov]

Folding into #1076