mspyls cannot analyze symlinked modules

[ztlevi]

I start to work on a very big project. And mspyls couldn't cache the project. The caching animation in my modeline keeps going forever.

The lsp-log looks fine.

Click to expand!

Command "/home/ztlevi/.emacs.d/.local/etc/mspyls/Microsoft.Python.LanguageServer" is present on the path. Command "pyls" is not present on the path. Found the following clients for /home/ztlevi/av/detection/python/private/tools/tl_state_tracker_evaluation/groundtruth_provider.py: (server-id mspyls, priority 1) The following clients were selected based on priority: (server-id mspyls, priority 1) Analysis cache path: /home/ztlevi/.cache/Microsoft/Python Language Server GetCurrentSearchPaths /home/ztlevi/python/bin/python Interpreter search paths: /home/ztlevi/python/lib/python3.7 /home/ztlevi/python/lib/python3.7/lib-dynload /home/ztlevi/.local/lib/python3.7/site-packages /home/ztlevi/python/lib/python3.7/site-packages User search paths: Command "/home/ztlevi/.emacs.d/.local/etc/mspyls/Microsoft.Python.LanguageServer" is present on the path. Command "pyls" is not present on the path. Found the following clients for /home/ztlevi/Developer/mypy/setup.py: (server-id mspyls, priority 1) The following clients were selected based on priority: (server-id mspyls, priority 1) Analysis cache path: /home/ztlevi/.cache/Microsoft/Python Language Server GetCurrentSearchPaths /home/ztlevi/python/bin/python Interpreter search paths: /home/ztlevi/python/lib/python3.7 /home/ztlevi/python/lib/python3.7/lib-dynload /home/ztlevi/.local/lib/python3.7/site-packages /home/ztlevi/python/lib/python3.7/site-packages User search paths: Microsoft Python Language Server version 0.3.66.0 Initializing for /home/ztlevi/python/bin/python Command "/home/ztlevi/.emacs.d/.local/etc/mspyls/Microsoft.Python.LanguageServer" is present on the path. Command "pyls" is not present on the path. Found the following clients for /home/ztlevi/Developer/mypy/mypy/binder.py: (server-id mspyls, priority 1) The following clients were selected based on priority: (server-id mspyls, priority 1) Command "/home/ztlevi/.emacs.d/.local/etc/mspyls/Microsoft.Python.LanguageServer" is present on the path. Command "pyls" is not present on the path. Found the following clients for /home/ztlevi/Developer/mypy/mypy/nodes.py: (server-id mspyls, priority 1) The following clients were selected based on priority: (server-id mspyls, priority 1)

Every time I try to trigger lsp-ui-peek-find-definitions, it gives me error message lsp--send-request-async: No workspace could handle textDocument/definition.

But things works fine on smaller project, for example, https://github.com/python/mypy. I tried lsp-ui-peek-find-definitions in mypy project and everything works fine. It only breaks when I was in the large project.

Is there any way that we could add some extra parameter to ignore caching some directory? This might be the answer.

I did a fresh install today so everything should be updated. And my emacs version is 26.3.

[seagle0128]

According to the description, it seems the issue of the server configuration or itself. I didn't encounter such issue even with the large project (e.g. numpy, tensorflow). I suggest you to check if it exists in VSCode, and check the logs of Microsoft Python Language Server.

[ztlevi]

I'm opening a new issue cuz I found the problem is related to jedi

[ztlevi]

Sorry, I found it's still the problem that the project is too large to index. According to the instruction over here, I was able to resolve the popping message shows I need to increase the max watch.

But it's not the problem anymore. I found it's related to jedi. Because when I disable jedi in
vscode using "python.jediEnabled": false, I am literally no longer able to use features like jump
to definition. While as you can see from the vscode setting, jediEnabled is true by default.

Also, once I enabled jedi, I no longer has "$XDG_CACHE_HOME/Microsoft/Python Language Server" in my local machine (I deleted these caches couple times). Instead, I have jedi caches
locally like this ~/.cache/jedi/CPython-37-32/XXXXX-XXXX.pkl.

As far as I know, mspyls is an alternative for jedi. It's weird that Microsoft Python Language
Server use jedi as default.

So I don't konw how to track and slove this problem at this point. Is there anything that you can
think of to help me to solve my problem?

Thank you!

[seagle0128]

What happens if you uninstall jedi module?

[ztlevi]

I guess it comes with vscode python package and I couldn't uninstall it.

BTW, I found another issue might be the problem. The project has a python_root which contains some module's symbolic link. This might cause the issue that mspyls cannot resolve modules.

[ztlevi]

I can confirm mspyls cannot resolve symlinked modules but vscode with jedi enabled can.

Here is a repo I created for you to test https://github.com/ztlevi/python-test

[seagle0128]

If so, you should file an issue for mslyls server.

[ztlevi]

Yeah, maybe. But it seems work perfectly in vscode. Can you tell me to how inject couple setting into lsp-python-ms--extra-init-params? For example, how can we put "python.jediEnabled": false from vscode into Emacs?

[ztlevi]

I can also confirm neither anaconda-mode, pyls or mspyls work in the project I created here. Instructions can be found in previous comments.

I think it's a pretty common strategy to symlink some folder in project. For example, tensorflow symlinked a lot. Is there any solution or hack come across your mind that might solve the problem? Cuz if vscode works, mspyls should work as well.

[ztlevi]

Let me correct this. "python.jediEnabled": false won't break resolving symlinked modules. Actually, only when jedi is disabled, Microsoft.Python.LanguageServer will be used.

[ztlevi]

Posted a issue in here microsoft/python-language-server#1545

[ztlevi]

I might find the problem!

Does mspyls uses environment's PYTHONPATH?

The hack that I have in my project is I modified the PYTHONPATH like this export PYTHONPATH=$PYTHONPATH:{dir}/python-test/python_root. This is something unusual. Even if I can make Emacs has the proper PYTHONPATH, it doesn't really means Microsoft.Python.LanguageServe uses it.

Maybe lsp-python-ms-extra-paths?

Tested lsp-python-ms-extra-paths by adding .dir-locals.el like this. But no luck get it working.

((python-mode
  (lsp-python-ms-extra-paths . ("/Users/ztlevi/Developer/Github/python-test/python_root"))))

Here is the logs:

LSP :: Connected to [mspyls:97212 status:starting].
Python language server started
user-error: Not found for: test_func

Pycharm works

And also, like in Pycharm, I need to manually add the additional PYTHONPATH to the interperter. It turns out works perfectly for symlinked modules.

[ztlevi]

@seagle0128 I figure out the solution! I tried to add lsp-python-ms-extra-paths before but it doesn't really work before because I put it in .dir-locals.el. mspyls probly starts before it reads .dir-locals.el variables. Once I set it as a const variable in my personal setting, it works.

This is something we need to document. And my suggestion would be utilizing PYTHONPATH by default which is what exactly vscode did when bootstrapping. It might be tricky for Emacs to do so because not every emacser has proper environment in their setting. But fortunately, exec-path-from-shell, spacemacs, doom-emacs now all have their own environment setup. And it's easier to get proper environment variables now.

[seagle0128]

So it's not a bug, right?

lsp-python-ms-extra-paths is not a local variable unless make-local-variable is used for it. It's a common sense to use defcustom as global vars. I don't understand why it's used in .dir-locals.el.

[ztlevi]

Ture. But my first insight is I thought extra paths is something specific to some project...

[seagle0128]

You can make it local to try.

[seagle0128]

@ztlevi Please test the latest version.

[phenix3443]

Ture. But my first insight is I thought extra paths is something specific to some project...

@ztlevi how to solve this problem? how to set extra path specific to some project ? same probleom to me .

[ztlevi]

@phenix3443 just use setq to set lsp-python-ms-extra-paths. It is not a local variable and need to be set before the lsp start

[phenix3443]

@ztlevi i guess i should set lsp-python-ms-extra-path before load the lsp-python-ms.el, because when the file regist lsp client , it use lsp-python-ms-extra-path, which i want to be set dynamiclly. I use the following code:

(use-package lsp-python-ms
  :ensure t
  ;需要延迟加载，因为脚本regist client时候依赖动态改变的lsp-python-ms-extra-paths
  :defer t
  :custom
  ;; lsp-python-executable-cmd, 用来解决在 PATH 里面存在 python2 和 python3,
  ;; 并且 python 默认指向 python2(例如 macos), 而平时使用都用 python3，
  ;; 此时不能引用python3 下面安装的包
  (lsp-python-ms-python-executable-cmd "python3")

  :config
  )

(use-package python-mode
  :ensure t
  :hook (
         (python-mode . lsp-deferred)
         (python-mode . (lambda ()
                          (setq lsp-python-ms-extra-paths (list (file-name-directory buffer-file-name)))
                          (require 'lsp-python-ms)
                          ))
         )
  :config
  )

key code is :deffer t and require, this make set lsp-python-ms-extra-path possible.