Python Language Server uses too much RAM / Memory

[nojvek]

Environment data

• VS Code version: Version 1.27.0 (1.27.0)

• Extension version (available under the Extensions sidebar): 2018.8.0

• OS and version: Mac Snow Leopard

• Python version (& distribution if applicable, e.g. Anaconda): Python2

Honestly there should just be a command that generates this

Actual behavior

Machine crawls to a halt. Using Python's language server using 30GB+ happens after a couple of hours of use.

Expected behavior

Sonic like universe. FAST!

Steps to reproduce:

1. Use vscode with python for a while. We have a somewhat large repo with 10k+ python files.

[ueoo]

I have the same issue with VS Code version 1.27.2

[ueoo]

Cause to VS Code not responding...

[tadly]

Same here and unlike #2553, I have jedi disabled.

[JPhillot]

Same here.

I use vscode as git editor and also as mergetool.
My git config:

# .gitconfig
[core]
    editor = code --wait
[difftool "vscode"]
    cmd = code --wait --diff $LOCAL $REMOTE
    keepBackup = false
    trustExitCode = true
[merge]
    tool = vscode
    conflictstyle = diff3
[mergetool "vscode"]
    cmd =  code --wait $MERGED
    keepBackup = false
    trustExitCode = true

After each merge conflict fixed in vscode, the server seems to not release the used RAM.

[blaylockbk]

The python extension is using nearly all my memory on my desktop computer, too. Task Manager says memory is eaten up by Microsoft.Python.LanguageServer

[krmozejko]

I have same issue on Linux Mint 18.3 with VS Code Version: 1.27.2 and extension 2018.8.0. After some time it uses all the RAM and starts flooding swap.

[brettcannon]

For those of you bumping up against this, a reminder that setting "python.jediEnabled": true in your settings will switch you back to Jedi and off the language server.

[brettcannon]

And if anyone has a repo that can reproduce the problem that they can share that would be greatly appreciated as debugging a potential cause is nearly impossible otherwise.

[tadly]

I already did switch back to jedi for the time being.

As for a share-able repo, I've mostly been working on dups, did switch between projects using the project manager extension though.
Not sure if this might have to do anything with it.

[brettcannon]

[ @nojvek I edited your ✉️ ⬆️ to come off as more 😃 and less 😠 ]

[MikhailArkhipov]

microsoft/python-language-server#109
microsoft/python-language-server#111

[gav-fyi]

I have the same issue using python3 on a small/new codebase (can’t share) where colleagues on same codebase works fine.

I also have the project manager plugin installed.

[nojvek]

Thank you @brettcannon , Apologies for the tone.

[brettcannon]

I'm going to leave this open, but hopefully we will have the fixes Mikhail and the language server team have made in the next release of the extension (at worst the one following).

@nojvek no problem, we all get cranky sometimes. 😄 And I'll also mention that #1119 covers your request for a command to gather system details we ask for.

[heyakyra]

@brettcannon are the releases on a regular schedule or just "when it's ready"?

[brettcannon]

@heyakyra monthly release schedule

[heyakyra]

Thanks, for me I have found that enabling jedi solves the issue, strangely enough.

[brettcannon]

@tadly you mentioned that you were switching between stuff using another extension, so I had not dived into trying the repo yet. Plus we are hoping to do a new release soon that will hopefully fix this.

[scorphus]

Hey, @brettcannon. Thanks for your reply! I think you can reproduce this behavior with thefuck's source code (at least this is the source code I'm having memory leak problems with). I'm on Linux 4.15 x86_64 and Python 3.7.0. Please let me know if you need any additional info. Also, I'd love to test any alpha/beta/RC.

[brettcannon]

@scorphus if you install a development build then you should be able to test the current beta version of the language server.

[hmnd]

@brettcannon Still happens on vscode insider, dev build of extension.

[xen]

I'm also have this issue with latest python (3.7), OSX (Mojave). Also, I have Visual Studio IntelliCode which require Language Server.

[NdagiStanley]

Same here

[heyakyra]

I first encountered this problem on vscode insiders (since stable doesn't work on Fedora 29)

[MikhailArkhipov]

@scorphus, @NdagiStanley, @xen, @hmnd

Can you try setting python.analysis.watchSearchPaths to false and see if it helps.

[hmnd]

@MikhailArkhipov That config setting apparently doesn't exist.

[MikhailArkhipov]

@hmnd - it does, it is simply not currently public (i.e. not listed in package.json). There are some development settings that rarely matter to regular users.

[hmnd]

@MikhailArkhipov Looks like ram usage remains incredibly high but it doesn't grow as quickly as before...

[xen]

@MikhailArkhipov doesn't help. I'm tried to restart VS Code after added this flag, but now it started to consume all resources again:

If I kill process I'm getting this error on Output tab:

[Info  - 3:47:40 PM] Connection to server got closed. Server will restart.
[Error - 3:47:40 PM] Request textDocument/documentSymbol failed.
Error: Connection got disposed.
	at Object.dispose (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/main.js:825:25)
	at Object.dispose (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-languageclient/lib/client.js:57:35)
	at LanguageClient.handleConnectionClosed (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-languageclient/lib/client.js:2036:42)
	at LanguageClient.handleConnectionClosed (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-languageclient/lib/main.js:127:15)
	at closeHandler (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-languageclient/lib/client.js:2023:18)
	at CallbackList.invoke (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/events.js:71:39)
	at Emitter.fire (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/events.js:135:36)
	at closeHandler (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/main.js:221:26)
	at CallbackList.invoke (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/events.js:71:39)
	at Emitter.fire (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/events.js:135:36)
	at StreamMessageReader.AbstractMessageReader.fireClose (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/messageReader.js:135:27)
	at Socket.<anonymous> (/Users/xen/.vscode/extensions/ms-python.python-2018.8.0/node_modules/vscode-jsonrpc/lib/messageReader.js:188:62)
	at emitOne (events.js:121:20)
	at Socket.emit (events.js:211:7)
	at Pipe._handle.close [as _onclose] (net.js:554:12)
Initializing for /Users/xen/.local/share/virtualenvs/notify-1dEpo5ac/bin/python

Upd. In status line also shown spinning arrows circle with text "Analyzing workspace, 14 items remaining..."

[scorphus]

@MikhailArkhipov, thanks for your message.

I tried with "python.analysis.watchSearchPaths": false as you instructed but the behaviour persists, memory usage keeps increasing.

Maybe worth to mention, I'm running with only 8G of RAM and I don't know if memory usage stops increasing at some point because the system runs out of memory if I don't quit VSCode.

This is the output of grep python ~/.config/Code/User/settings.json:

    "python.linting.enabled": true,
    "python.linting.lintOnSave": true,
    "python.linting.pylintEnabled": false,
    "python.linting.pylintCategorySeverity.convention": "Hint",
    "python.linting.pylintCategorySeverity.warning": "Error",
    "python.analysis.watchSearchPaths": false,
    "python.linting.flake8Enabled": true,

After re-downloading language server, I see the following in the output panel:

Starting Microsoft Python language server.
Initializing for /.../.pyenv/versions/thefuck-370/bin/python
##########Linting Output - flake8##########
##########Linting Output - flake8##########
[Error - 3:10:43 PM] System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.Collections.Generic.HashSet`1.Enumerator.MoveNext()
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.OfTypeIterator[TResult](IEnumerable source)+MoveNext()
   at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
   at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
   at Microsoft.PythonTools.Analysis.PythonAnalyzer.GetEntriesThatImportModule(String moduleName, Boolean includeUnresolved)
   at Microsoft.Python.LanguageServer.Implementation.Server.OnDocumentChangeProcessingCompleteAsync(IDocument doc, VersionCookie vc, Boolean enqueueForAnalysis, AnalysisPriority priority, IDisposable disposeWhenEnqueued)

I hope this helps. I'm still looking forward to a chance (aka free time) to test the development build, as @brettcannon suggested. I'll do so ASAP.

Thank you all for your time and support! 👍

[MikhailArkhipov]

Most probably roots to

microsoft/python-language-server#193
microsoft/python-language-server#173

[obrotowy]

Same on macOS 10.14 (18A391), VSCode 1.27.2 and vscode-python 2018.8.0. It used 22GB of RAM on my machine xD

[DonJayamanne]

@nojvek @tadly @SynneK1337 @scorphus @xen @hmnd @heyakyra @ahter @blaylockbk @JPhillot

We believe this issue has been resolved.
We just released the latest version of the Python Extension, please update to this new version 2018.9.0 and give it a go.

[hmnd]

Looks like it's fixed for me @DonJayamanne

[xen]

I tried and its works. Thank you very much for this changes and for your amazing work.

[nojvek]

Thanks. Y’all so 😎.

I’m just curious what the technical issue was that caused such a collosal memory conquest?

[brettcannon]

@nojvek If I remember correctly it was from the language server constantly re-processing modules as they changed but not freeing the memory from the old analyses.