mspyls cannot analyze symlinked modules

[ztlevi]

I start to work on a very big project. And mspyls couldn't cache the project. The caching animation in my modeline keeps going forever.

The lsp-log looks fine.

Click to expand!

Command "/home/ztlevi/.emacs.d/.local/etc/mspyls/Microsoft.Python.LanguageServer" is present on the path. Command "pyls" is not present on the path. Found the following clients for /home/ztlevi/av/detection/python/private/tools/tl_state_tracker_evaluation/groundtruth_provider.py: (server-id mspyls, priority 1) The following clients were selected based on priority: (server-id mspyls, priority 1) Analysis cache path: /home/ztlevi/.cache/Microsoft/Python Language Server GetCurrentSearchPaths /home/ztlevi/python/bin/python Interpreter search paths: /home/ztlevi/python/lib/python3.7 /home/ztlevi/python/lib/python3.7/lib-dynload /home/ztlevi/.local/lib/python3.7/site-packages /home/ztlevi/python/lib/python3.7/site-packages User search paths: Command "/home/ztlevi/.emacs.d/.local/etc/mspyls/Microsoft.Python.LanguageServer" is present on the path. Command "pyls" is not present on the path. Found the following clients for /home/ztlevi/Developer/mypy/setup.py: (server-id mspyls, priority 1) The following clients were selected based on priority: (server-id mspyls, priority 1) Analysis cache path: /home/ztlevi/.cache/Microsoft/Python Language Server GetCurrentSearchPaths /home/ztlevi/python/bin/python Interpreter search paths: /home/ztlevi/python/lib/python3.7 /home/ztlevi/python/lib/python3.7/lib-dynload /home/ztlevi/.local/lib/python3.7/site-packages /home/ztlevi/python/lib/python3.7/site-packages User search paths: Microsoft Python Language Server version 0.3.66.0 Initializing for /home/ztlevi/python/bin/python Command "/home/ztlevi/.emacs.d/.local/etc/mspyls/Microsoft.Python.LanguageServer" is present on the path. Command "pyls" is not present on the path. Found the following clients for /home/ztlevi/Developer/mypy/mypy/binder.py: (server-id mspyls, priority 1) The following clients were selected based on priority: (server-id mspyls, priority 1) Command "/home/ztlevi/.emacs.d/.local/etc/mspyls/Microsoft.Python.LanguageServer" is present on the path. Command "pyls" is not present on the path. Found the following clients for /home/ztlevi/Developer/mypy/mypy/nodes.py: (server-id mspyls, priority 1) The following clients were selected based on priority: (server-id mspyls, priority 1)

Every time I try to trigger lsp-ui-peek-find-definitions, it gives me error message lsp--send-request-async: No workspace could handle textDocument/definition.

But things works fine on smaller project, for example, https://github.com/python/mypy. I tried lsp-ui-peek-find-definitions in mypy project and everything works fine. It only breaks when I was in the large project.

Is there any way that we could add some extra parameter to ignore caching some directory? This might be the answer.

I did a fresh install today so everything should be updated. And my emacs version is 26.3.

[seagle0128]

According to the description, it seems the issue of the server configuration or itself. I didn't encounter such issue even with the large project (e.g. numpy, tensorflow). I suggest you to check if it exists in VSCode, and check the logs of Microsoft Python Language Server.

[ztlevi]

I'm opening a new issue cuz I found the problem is related to jedi

[ztlevi]

Sorry, I found it's still the problem that the project is too large to index. According to the instruction over here, I was able to resolve the popping message shows I need to increase the max watch.

But it's not the problem anymore. I found it's related to jedi. Because when I disable jedi in
vscode using "python.jediEnabled": false, I am literally no longer able to use features like jump
to definition. While as you can see from the vscode setting, jediEnabled is true by default.

Also, once I enabled jedi, I no longer has "$XDG_CACHE_HOME/Microsoft/Python Language Server" in my local machine (I deleted these caches couple times). Instead, I have jedi caches
locally like this ~/.cache/jedi/CPython-37-32/XXXXX-XXXX.pkl.

As far as I know, mspyls is an alternative for jedi. It's weird that Microsoft Python Language
Server use jedi as default.

So I don't konw how to track and slove this problem at this point. Is there anything that you can
think of to help me to solve my problem?

Thank you!

[seagle0128]

What happens if you uninstall jedi module?

[ztlevi]

I guess it comes with vscode python package and I couldn't uninstall it.

BTW, I found another issue might be the problem. The project has a python_root which contains some module's symbolic link. This might cause the issue that mspyls cannot resolve modules.

[ztlevi]

I can confirm mspyls cannot resolve symlinked modules but vscode with jedi enabled can.

Here is a repo I created for you to test https://github.com/ztlevi/python-test

[seagle0128]

If so, you should file an issue for mslyls server.

[ztlevi]

Yeah, maybe. But it seems work perfectly in vscode. Can you tell me to how inject couple setting into lsp-python-ms--extra-init-params? For example, how can we put "python.jediEnabled": false from vscode into Emacs?

[ztlevi]

I can also confirm neither anaconda-mode, pyls or mspyls work in the project I created here. Instructions can be found in previous comments.

I think it's a pretty common strategy to symlink some folder in project. For example, tensorflow symlinked a lot. Is there any solution or hack come across your mind that might solve the problem? Cuz if vscode works, mspyls should work as well.

[ztlevi]

Let me correct this. "python.jediEnabled": false won't break resolving symlinked modules. Actually, only when jedi is disabled, Microsoft.Python.LanguageServer will be used.

[ztlevi]

Posted a issue in here microsoft/python-language-server#1545

[ztlevi]

I might find the problem!

Does mspyls uses environment's PYTHONPATH?

The hack that I have in my project is I modified the PYTHONPATH like this export PYTHONPATH=$PYTHONPATH:{dir}/python-test/python_root. This is something unusual. Even if I can make Emacs has the proper PYTHONPATH, it doesn't really means Microsoft.Python.LanguageServe uses it.

Maybe lsp-python-ms-extra-paths?

Tested lsp-python-ms-extra-paths by adding .dir-locals.el like this. But no luck get it working.

((python-mode
  (lsp-python-ms-extra-paths . ("/Users/ztlevi/Developer/Github/python-test/python_root"))))

Here is the logs:

LSP :: Connected to [mspyls:97212 status:starting].
Python language server started
user-error: Not found for: test_func

Pycharm works

And also, like in Pycharm, I need to manually add the additional PYTHONPATH to the interperter. It turns out works perfectly for symlinked modules.