Symbol navigation via ctrl-t does not work when the microsoft language server is enabled (jedi disabled)

[ogrisel]

Environment data

• VS Code version: 1.27.2
• Extension version (available under the Extensions sidebar): 2018.8.0
• OS and version: Ubuntu 16.04
• Python version (& distribution if applicable, e.g. Anaconda): 3.6 or 3.7 built from source
• Type of virtual environment used: virtualenv / python -m venv
• Relevant/affected Python packages and their versions: N/A

Steps to reproduce:

Disable jedi in settings.json. Here the full settings.json I used to make sure that this issue is not caused by a problem in my config:

{
    "python.jediEnabled": false
}

Then try with the following mixed Python/Cython/C/C++ code base:

$ git clone https://github.com/scikit-learn/scikit-learn
$ cd scikit-learn
$ code .

Then hit "ctrl-t" and type "KMeans" for instance.

Actual behavior

The ctrl-t menu is never populated as expected (it stays empty). I have the same problem with other symbols defined in this project (e.g. LogisticRegression, check_array...).

The "Analysing workspace, 1 item remaining..." message never goes away.

Weirdly enough, hitting "F12" on a Python symbol defined in the workspace can make VS code navigate to it successfully. It's just the ctrl-t symbol search menu that's broken.

Expected behavior

Being able to open the k_means_.py file from the project folder in the workspace to navigate to the definition of the KMeans class.

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

Starting Microsoft Python language server.
Initializing for /volatile/ogrisel/.virtualenvs/py36/bin/python
##########Linting Output - pylint##########
[snip]

[Error - 09:52:35] Request workspace/symbol failed.
  Message: Object reference not set to an instance of an object.
  Code: -32000 
[object Object]
[Error - 09:52:36] Request workspace/symbol failed.
  Message: Object reference not set to an instance of an object.
  Code: -32000 
[object Object]
...

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help)

[nothing]

Also note that jedi has no problem with this workspace.

[brettcannon]

I will try to verify.

[brettcannon]

This is triggering an error, so I have reported it upstream.

[ogrisel]

@brettcannon I tried with today's release (2018.10.0) of the VSCode Python extension, I got the message telling me the language server was updated but I still cannot get ctrl-t to work when working on a local clone of the https://github.com/scikit-learn/scikit-learn repo.

With the new version update (.vscode/extensions/ms-python.python-2018.10.0/languageServer.0.1.60/) that arguably includes @MikhailArkhipov's fix from #137) I no longer the Code: -32000 error message but instead I can see the Microsoft.Python.LanguageServer process using 100% of one of my CPUs forever (in htop).

@MikhailArkhipov is there any command to launch from the command line that could help get a more verbose diagnostics on what's going on?

More importantly the Go to Symbol in Workspace menu is still never populated (even after waiting for 10 minutes).

[ogrisel]

Actually after a while I got the following error messages in the output panel of VS Code:

Starting Microsoft Python language server.
Microsoft Python Language Server version 0.1.60.0
Initializing for /opt/venvs/py37/bin/python
[Error - 12:01:32 PM] Request workspace/symbol failed.
  Message: Object reference not set to an instance of an object.
  Code: -32000 
   at Microsoft.Python.LanguageServer.Implementation.Server.<>c__DisplayClass133_0.<GetModuleVariables>b__4(ILocationInfo l)
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
   at Microsoft.Python.LanguageServer.Implementation.Server.<>c__DisplayClass133_0.<GetModuleVariables>b__2(IMemberResult m)
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at System.Linq.Enumerable.EnumerablePartition`1.ToList()
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Microsoft.Python.LanguageServer.Implementation.Server.GetModuleVariablesAsync(ProjectEntry entry, GetMemberOptions opts, String prefix, Int32 timeout, CancellationToken token)
   at Microsoft.Python.LanguageServer.Implementation.Server.WorkspaceSymbols(WorkspaceSymbolParams params, CancellationToken cancellationToken)
   at Microsoft.Python.LanguageServer.Implementation.LanguageServer.WorkspaceSymbols(JToken token, CancellationToken cancellationToken)

[same error repeated many times]

[ogrisel]

I opened microsoft/python-language-server#378 upstream.

[ogrisel]

Now that microsoft/python-language-server#378 (@jakebailey and @MikhailArkhipov) is closed, I have tried again today with Python extension 2018.11.0 and languageServer 0.1.65 and I no longer get the error message (or any languageServer output) in the output panel of vscode but the symbol navigation still does not work:

same behavior as before when I do a ctrl-T + name of a scikit-learn class such as KMeans or any other class or function name.

The languageServer is still using 100% of one or 2 CPUs in the background (for several minutes).

On the same git clone, jedi / ctags indexes the symbols much faster (less than a minute I would say) and navigation works as expected.

[jakebailey]

We fixed the bugs causing the errors, however we have not yet removed the big "wait until analysis is complete" blockers in those "find-something" calls (see microsoft/python-language-server#378 (comment)). Not being able to make those calls is for now still the implemented behavior.

You may be interested in watching microsoft/python-language-server#433, which aims to solve the problem a different way. I may still try removing those blocks to see what happens, though.

[bsandrow]

How long is it reasonable to "wait until analysis is complete" though? It looks like analysis is never complete on the codebase that I'm using this on. Same behaviour. 100% CPU and no error messages. Intellisense seems to work fine.

I want to switch away from jedi/ctags because that also has issues, but until Go to Symbol is working, this might be a deal-breaker for me. Is there anything that I can do to gain insight into what the Microsoft.Python.LanguageServer process is doing (e.g. debug logging, etc)?

[jakebailey]

You can increase the logging level and watch the Python output panel for the language server output:

"python.analysis.logLevel": "Trace"

When it's actually done, it'll say "analysis complete for ... <filename>". Additionally, there is a file named AnalysisLog.txt inside the language server folder in the extension's folder, which is even more verbose.

Intellisense will work because we don't wait for the full analysis to complete before returning results. This is not true for go-to-X/find-X, but is not the intended final behavior (see my previous comment).

[ogrisel]

@jakebailey thanks. I did that for the scikit-learn project and here is the output log:

https://gist.github.com/ogrisel/a49fc97204f6964dc3a6edf3dbf3d7bd

I just open vscode on the sklearn/cluster/k_means_.py file to trigger the start of the Python extension and the start of the MS Python language server process and then do nothing.

In the first couple seconds I get the output log linked above. But after that nothings happens: the Microsoft.Python.LanguageServer process uses 100% CPU but does not yield any new output.

The only analysis that can complete is the analysis of the toplevel setup.py file. Nothing else. And no error message either.

[ogrisel]

I have also tried to delete non-library folders like examples/, doc/, benchmarks/ and the top-level conftest,py file to let the language server focus on the main code of the library but the results are the same.

[jakebailey]

Just to link it here: microsoft/python-language-server#521 reworks how we work with symbols and movesit out of the analysis. Testing on 0.1.77 (which includes that PR), in scikit-learn with k_means_.py open, the outline appears just after the language server starts, and find symbol in file/workspace no longer block and are available immediately.

The CPU usage is another issue, tracked in microsoft/python-language-server#479 and microsoft/python-language-server#495, which are being addressed separately.

[ogrisel]

@jakebailey great news! How to get language server 0.1.77 in vscode? I only get the older 0.1.75 at the moment.

$ ls ~/.vscode/extensions/ms-python.python-2018.12.1/ | grep languageServer
languageServer.0.1.75

I have tried to reinstall the ms python extension and delete de languageServer folder but I still get 0.1.75 each time.

[jakebailey]

It's currently in our daily and beta channels, not the stable release (since it's brand new). You can switch to one of those to test a non-stable build by setting one of the following in your settings:

"python.analysis.downloadChannel": "daily"

"python.analysis.downloadChannel": "beta"

If you want to go back to stable, you can remove that line and delete the folders with names starting with languageServer inside of the extension's directory, as the extension will not rollback to an older version.

[ogrisel]

I confirm that the Go To Symbol feature works on scikit-learn when using the beta channel. Thank you very much.

Looking forward to the resolution for the CPU usage issue now :) The trace output shows that the language server never seems to get back analysis results for the majority of the files of the project: only the first queued file (in my cases setup.py) ever receives an analysis back from the language server. All the other queued files seem to be stalled and CPU usage stays at 100% forever without any output in the trace-level log.

Feel free to close this issue if you wish, as the symbol navigation part is fixed.

[bsandrow]

FYI: I switched to the beta channel, and Go To Symbol, Outline, etc are working now. 100% CPU issue is still there though. I set the log level to "TRACE", and it looks like it analyzes me entire codebase rather quickly, ending on analysis of setup.py, but then it spins up to 100% CPU and sits there with no more logging that gives insight into what is happening. Could this be an issue related to analysis of setup.py (since as that seems to be the only file that completes analysis in scikit-learn before hitting the 100% CPU issue?

Edit: Nvm. I read the logging output wrong. It was just all of my project's files being queue for analysis. setup.py was the only one that completed analysis. Removing setup.py from the project and restarting VS Code just has it do the same thing on (presumably) the next file in the queue (in my case manage.py).

[jakebailey]

Yes, that's the issue I was referring to above, which we're handling a different way (by reworking our analysis as a whole). The CPU usage will be high depending on the project, but completion/hover/etc should still be available, though may be incomplete depending on where the language server has gotten stuck.

[ogrisel]

Edit: Nvm. I read the logging output wrong. It was just all of my project's files being queue for analysis. setup.py was the only one that completed analysis. Removing setup.py from the project and restarting VS Code just has it do the same thing on (presumably) the next file in the queue (in my case manage.py).

I observe the same behavior on scikit-learn.

[ogrisel]

The CPU usage will be high depending on the project, but completion/hover/etc should still be available, though may be incomplete depending on where the language server has gotten stuck.

The problem is not so much that the CPU usage is high but instead that the analysis seems to get stuck and never progresses. The analysis process seems to be caught in some kind of infinite loop or something similar.

[brettcannon]

I'm closing this as per ogisel saying he was satisfied with the result (which is now out publicly as part of the January 2019 release of the extension).