x/tools/gopls: high memory consumption

[orlovm]

What version of Go are you using (go version)?

$ go version
go version go1.17 linux/amd64

Does this issue reproduce with the latest release?

Yes. I have the latest versions of gopls and go.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mikhail/.cache/go-build"
GOENV="/home/mikhail/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/mikhail/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mikhail/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4125386380=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Opened cockroachDB repository in vim(same thing with neovim) with vim-go plugin and gopls enabled.
I've removed everything but vim-go and gopls from the .vimrc and disabled all other plugins. But the result is the same.

What did you expect to see?

Normal memory consumption. For example, Jetbrains Goland uses ~2GB when this project is open

What did you see instead?

gopls consumes about 9GB of memory, sometimes more than 14GB.

gopls.30806-8GiB-withnames.zip

gopls.3081-14GiB-withnames.zip

[heschi]

In general gopls can be fairly memory hungry on large projects. This is an ongoing area of research. cc @findleyr.

FWIW, glancing at the attached zip, github.com/cockroachdb/cockroach/pkg/workload/geospatial is probably costing you 500+ MB all by itself. That package contains a very large composite literal that is very expensive to parse and type check. Removing that composite literal from gopls' scope, perhaps by changing its representation or excluding it from the build with a build tag, might be well worth the effort.

[orlovm]

@heschi I've added a new archive. Please take a look at it.
gopls consumes more memory during use.
Isn't there any memory leak?

[heschi]

No. I don't see any signs of a memory leak.

If you want, you can try setting https://github.com/golang/tools/blob/master/gopls/doc/settings.md#memorymode-enum. Use it at your own risk.

[orlovm]

So this is normal behavior?
Why is there such a big difference in memory consumption compared to JB Goland?

[hyangah]

I'd say the high memory usage is expected currently - Goland and gopls take different approaches.

Gopls dev team is currently working on improvement in this area. The "DegradeClosed" memory mode @heschi shared is one of the directions we are exploring. Please give it a try (warning: still experimental). Feedback and bug reports for the DegradeClosed mode are greatly appreciated!

[karlismea]

In general gopls can be fairly memory hungry on large projects.

I thought all source needs to be in /src directory - it means hundreds of imports and all my projects! Happy if someone can explains how to remove this "virus" from VS Code.

[578559967]

How to enable "DegradeClosed" memory mode in vim?

I'd say the high memory usage is expected currently - Goland and gopls take different approaches.

Gopls dev team is currently working on improvement in this area. The "DegradeClosed" memory mode @heschi shared is one of the directions we are exploring. Please give it a try (warning: still experimental). Feedback and bug reports for the DegradeClosed mode are greatly appreciated!

[adonovan]

The next release of gopls (expected in March) will include major reductions in memory usage, so if you're on the verge of giving up, consider holding on just a few more weeks.

[n-r-w]

[findleyr]

@n-r-w which version of gopls are you using?

We're still smoothing out some edge cases and so have not yet cut a prerelease for [email protected], but I expect memory usage to be significantly reduced once we do.

[n-r-w]

@n-r-w which version of gopls are you using?

v0.11.0

[n-r-w]

We're still smoothing out some edge cases and so have not yet cut a prerelease for [email protected], but I expect memory usage to be significantly reduced once we do.

It's great to hear that. Because right now my gopls is using 34 Gb of memory :)

[findleyr]

Hi all, we just cut the first prerelease version of gopls to include the new architecture:

go install golang.org/x/tools/[email protected]

More details in the release notes.

This is a prerelease version: there are several known regressions that we're still working on. However, it should use significantly less memory. I'd love to hear from users currently struggling with gopls' memory usage, whether this release improves their experience. If it doesn't, there may be more we can do given additional details.

Thanks!

[garymoon]

Early impressions are very positive for me: Working with the Gitea codebase in VSCode, after letting things settle, my gopls RSS has almost halved, from 1,044,976 to 568,124, and in the time it's taken me to write this has somehow fallen further to 478,228. You've made a cranky old infra engineer very happy, thank you all very much! 💙

[findleyr]

Thanks for the feedback, @garymoon!

in the time it's taken me to write this has somehow fallen further to 478,228

FWIW this is likely due to a background GC cycle, and/or the timing of background work that gopls is doing. To get a more detailed picture of memory usage, you can start gopls with -debug=localhost:8080 and visit http://localhost:8080/memory.

[findleyr]

Hi folks,

We just released the second prerelease, which fixes several bugs reported since -pre.1 and smooths out performance a bit:

go install golang.org/x/tools/[email protected]

At this point, gopls' memory footprint has fundamentally changed: gopls requires very little data in memory other than file content. To reduce its steady-state memory further would require dropping file content, and we don't current plan to do that (doing so would introduce a lot of complexity). There may be additional issues related to memory: for example, gopls could use less memory during large operations such as rename. However, those would be specific problems that deserve a separate github issue.

Therefore, I am going to close this issue as fixed. Thanks everyone for your feedback and patience.