High memory usage

[ebak]

Description

I have recently tried out the stable tool-chain (installed with swiftly on Manjaro Linux) with VSCode and its Swift plugin.
The memory usage of sourcekit-lsp easily freezes out the machine.

The machine has 16 CPU threads (8 core with 2 HW threads), but only 8GB RAM.
As I suspect the number of sourcekit-lsp threads scales up based on the number of CPU threads.

Is it possible to explicitly limit the sourcekit-lsp threads?
(Linux allows to turn off some cores, but that is not too convenient.)

Or would it be possible to also consider available RAM too when spawning sourcekit-lsp threads?
E.g. for 8GB of RAM only spawn 3 threads, even if the machine have 256 CPUs.

[ahoppen]

Synced to Apple’s issue tracker as rdar://130844901

[ahoppen]

Hey @ebak,

A couple of questions:

• How much memory does sourcekit-lsp use on your machine? I would be expecting something <500MB on start up, fairly independent of your CPU core count.
• Does the memory usage of SourceKit-LSP grow over time or does it go up immediately after launching
• Does this also reproduce with a Swift 6 toolchain. We have made significant changes to SourceKit-LSP between 5.10 and 6.0, so the issue might already be fixed.
• Does the memory usage go up on every project that you open or only on specific ones? If so, would you be able to share the project?

[ebak]

Hi,

Now I don't have possibility to try it on the machine with 8 CPUs, maybe next week.

How much memory does sourcekit-lsp use on your machine? I would be expecting something <500MB on start up, fairly independent of your CPU core count.

I attach screenshots of htop output, as I see at startup it only consumes 2.8% (I don't know what htop means about this percentage.)

Does the memory usage of SourceKit-LSP grow over time or does it go up immediately after launching

Grows over time. As I see build and source browsing is required for it.

Does this also reproduce with a Swift 6 toolchain. We have made significant changes to SourceKit-LSP between 5.10 and 6.0, so the issue might already be fixed.

Not yet tried out, maybe I'll do when I have possibility to play it around with the system with 8 CPUs.

Does the memory usage go up on every project that you open or only on specific ones? If so, would you be able to share the project?

Recently I was working with this project (QtTest):
QtTest.zip

[ebak]

Just by editing the same code, the memory usage quickly goes up to 45% (3.6GB maybe):

Maybe unrelated but VSCode usually forgets about the dependent packages:

The dependencies (Qlift, Math) are sometimes shown up there, sometimes not.

Sad the 8GB RAM is hardly enough for IDE assisted development.
In my case the opened apps almost use up all free memory (Gnome, 2 terminals, 2 file managers, a light weight editor (gedit), an ebook reader for the Swift book, VSCode with Swift instrumentation).
Opening up a Firefox with 1-2 tabs would easily freeze out the system.

[ahoppen]

I have been able to reproduce the memory growth by working on the SourceKit-LSP codebase in VS Code on Ubuntu 22.04.

[ebak]

Thanks! With the 6.0 or the 5.10 toolchain? Maybe if this issue doesn't exist in 6.0, this issue could be closed and wait for 6.0 to get stable. (Currently I cannot test.)

[ahoppen]

I saw the memory issue with a 6.0 toolchain.

[iMostfa]

i think that this issue is happening more after updating to Xcode 16.
it keeps growing until the whole device get freezed.

[iMostfa]

i downloaded Xcode 15, copied the sourceKit inside it, then placed it inside Xcode 16, and it appears to be working, and fixed my issue [until now]

[ahoppen]

Oh, that’s interesting that you’re also seeing excessive memory usage on macOS. I haven't seen that before.

Could you try reproducing the problem, run leaks --outputGraph=/tmp sourcekit-lsp and then share the .memgraph file that is being generated the command? I’m hoping that will show us where the memory allocations are from.

[iMostfa]

@ahoppen i will try, when it happens i'm running against time to kill SourceKit-lsp before my device is freezed 🥶

i can also confirm that since i started using sourcekit-lsp from Xcode 15 the issue isn't appearing

[ahoppen]

I think it would be valuable to see the graph when SourceKit-LSP is at ~2GB. That’s already more than it should take and should contain the culprit who’s using the memory.

[iMostfa]

@ahoppen
i just shared it with u over email, thanks for u

[ahoppen]

Thanks for sharing the memgraph, @iMostfa 🙏🏽 With it, it was obvious that the failure cause is the same one as #1720.

[ahoppen]

I tracked down the memory leak to a bug in libdispatch. The issue should be fixed by swiftlang/swift-corelibs-libdispatch#850.

[m-buczek]

This error prevents me from working on Xcode 16.2.

@ahoppen Is there a chance to fix this bug in Xcode 16.x or do we have to wait for Xcode 17?

I can confirm that the workaround provided by @iMostfa works (thanks a lot!).

More detailed instructions:

1. Disable System Integrity Protection: Apple Documentation
2. Get Xcode 15.4: Download Link (requires login to a developer account) and unzip archive.
3. Replace sourcekit-lsp binary:

sudo cp /<PATH_TO_XCODE_15.4>/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/sourcekit-lsp /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/

Please note that Xcode will only work with System Integrity Protection disabled.

[iMostfa]

@m-buczek it's interesting that you needed SIP to be disabled, for me it worked without disabling it.
btw @ahoppen can you confirm to us if the changes to sourcekit-lsp were merged to Xcode[or swift which later merged to Xcode]

[ahoppen]

Thanks for also confirming that my PR above fixes the issue. The fix will be included in Swift 6.1. I can’t speak for Xcode’s release schedule.