staticcheck and gopls nuke my computer #51829

bschaatsbergen · 2022-03-20T19:46:46Z

What version of Go are you using (`go version`)?

bruno@pop-os ~> go version                                                                                                                                                                                                                                                  (base) 
go version go1.17.8 linux/amd64

Does this issue reproduce with the latest release?

Open a large code base in VS code.

What operating system and processor architecture are you using (`go env`)?

go env Output

bruno@pop-os ~> go env                                                                                                                                                                                                                                                      (base) 
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/bruno/.cache/go-build"
GOENV="/home/bruno/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/bruno/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/bruno/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/linuxbrew/.linuxbrew/Cellar/go/1.17.8/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/linuxbrew/.linuxbrew/Cellar/go/1.17.8/libexec/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.8"
GCCGO="gccgo"
AR="ar"
CC="gcc-5"
CXX="g++-5"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1077551622=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Open visual studio code, to be more specific aws-sdk-go project or the hashicorp/terraform-provider-aws project

What did you expect to see?

My project

What did you see instead?

I get logged out because my machine is having a hard time

The text was updated successfully, but these errors were encountered:

dominikh · 2022-03-20T20:20:38Z

gopls doesn't invoke the Staticcheck binary. Even with Staticcheck enabled in gopls, the checks would execute directly in the gopls process. So this doesn't seem like a gopls issue.

findleyr · 2022-03-21T13:43:18Z

This may be a vscode-go issue, however.

CC @hyangah

hyangah · 2022-03-21T15:55:59Z

Agree that it's likely the vscode-go issue rather than gopls or go - @bschaatsbergen can you verify staticchecks are the children of a VSCode process?
One workaround is to disable staticcheck invocation by vscode-go, but delegate to gopls by enabling
"gopls": { "ui.diagnostics.staticcheck": true }

ElanHasson · 2022-03-24T18:04:46Z

I'm seeing similar here. downgrading to 0.8.0 seems to work with setting "go.toolsManagement.autoUpdate": false, in vscode settings

findleyr · 2022-03-25T16:51:44Z

Hi, fwiw I don't think [email protected] should make a difference relative to [email protected]. @ElanHasson are you sure this matters?

ElanHasson · 2022-03-25T20:02:12Z

Several internal folks here have reported gopls as being slow for them and eating memory. As you mentioned this may be a vscode-go issue.

Downgrading seems to have resolved the issue for us. I did a quick search in vscode issues list and couldn't find anything relevant.

What is the best way to gather data on this to help diagnose the issue (or rule out gopls)?

findleyr · 2022-03-25T20:05:56Z

Gopls does use a lot of memory, but I'd be a bit surprised if it had a regression in v0.8.1, as that was mostly a bugfix release.

If you could run [email protected] on your codebase and report its RSS once it has finished loading, then compare with [email protected], that would be very helpful.

Which version of Go are you using, and do you use go workspace? go.work?

ElanHasson · 2022-04-01T23:17:44Z

❯ ps -o pid,vsz,rss,%mem,command -p $(ps -m -o pid=28227 | head -n1)
  PID    VSZ   RSS %MEM COMMAND
28227 3781784 1705460 10.5 /home/elan/go/bin/gopls -mode=stdio
❯ go version
go version go1.18 linux/amd64


❯ ps -o pid,vsz,rss,%mem,command -p $(ps -m -o pid=23130 | head -n1)
  PID    VSZ   RSS %MEM COMMAND
23130 14398852 12438852 76.5 /home/elan/go/bin/gopls -mode=stdio
❯ gopls version
golang.org/x/tools/gopls v0.8.1
    golang.org/x/tools/[email protected] h1:q5nDpRopYrnF4DN/1o8ZQ7Oar4Yd4I5OtGMx5RyV2/8=

  PID    VSZ   RSS %MEM COMMAND
13322 13516860 10040104 61.8 /home/elan/go/bin/gopls -mode=stdio
❯ gopls version
golang.org/x/tools/gopls v0.8.2
    golang.org/x/tools/[email protected] h1:+b7X2FDwmP7vxyYNOHe/ZFkecN5X8OtAgkNtuCr86wk=

I'd say this should be closed as the sluggishness has gone away.

findleyr · 2022-04-05T13:48:05Z

Thanks for following up!

ElanHasson · 2022-04-12T21:18:37Z

so, hate to be "that guy"...

Output from VSCode:

Starting linting the current package at /home/elan/repos/cthulhu/docode/src/do/teams/insights/executioner/pkg/execute
/home/elan/repos/cthulhu/docode/src/do/teams/insights/executioner/pkg/execute>Finished running tool: /home/elan/go/bin/staticcheck

Failed while handling 'onDidChangeTextDocument': TypeError: Cannot read properties of undefined (reading 'children')

findleyr · 2022-04-12T21:26:08Z

@ElanHasson your screenshot shows a lot of gopls ~~processes~~threads (using a ton of memory), but your output relates to staticcheck.

Can you clarify what you're pointing out? Do you know why you have so many gopls ~~processes~~threads?

EDIT: threads, not processes

ElanHasson · 2022-04-12T21:36:01Z

Hi @findleyr, I'm not sure-- I imagine it's a "feature" of VSCode.

I'm not sure if this is a VSCode problem (the staticcheck or multiple gopls). I figured I'd add details here for discoverability.

The previous RSS numbers I dropped were very low compared to these above. I'm unfamiliar with the internals of gopls or staticcheck nor how VSCode uses them.

I'm going to log an issue in https://github.com/microsoft/vscode and link to this for reference and operate under the impression it's a VSCode issue first.

I'd still say leave this closed unless the VSCode folks come back and say something otherwise.

findleyr · 2022-04-12T21:41:42Z

I don't think this has anything to do with microsoft/vscode. It's either gopls or vscode-go (both of which are us :) )

Reopening to continue investigation:

can you please check which process started all those gopls instances?
can you try killing them? Are they all restarted?
how many vscode workspaces do you have open?
which directory did you open as your workspace? gopls can use a lot of memory if opened to e.g. your home directory

ElanHasson · 2022-04-12T21:54:41Z

Thanks for clarifying ownership, much appreciated.

Versions:

❯ gopls version
golang.org/x/tools/gopls v0.8.3
    golang.org/x/tools/[email protected] h1:Mxm94ix8oSARQ6svioO6SxKEYWT/VCP54/448LOHzrk=

❯ /home/elan/go/bin/staticcheck  -version
staticcheck 2022.1 (v0.3.0)

❯ go version
go version go1.18 linux/amd64

[golang.Go ](https://marketplace.visualstudio.com/items?itemName=golang.Go) = v0.32.0

Visual Studio Code:

Version: 1.66.1 (user setup)
Commit: 8dfae7a5cd50421d10cd99cb873990460525a898
Date: 2022-04-06T14:50:12.141Z (6 days ago)
Electron: 17.2.0
Chromium: 98.0.4758.109
Node.js: 16.13.0
V8: 9.8.177.11-electron.0
OS: Windows_NT x64 10.0.22000

I'm using VSCode WSL-Remote on Windows 11.

Seems to be vscode and then gopls started the rest:

killall -9 gopls kills them all and they are restarted, memory slowly rising over the course of a few minutes
I had multiple open VSCodes open, closed them and now I only have one workspace with multiple folders/repos (some folders in same monorepo)
My .code-workspace is in ~/repos/workspaces and my others are in ~/repos ( ../ = ~/repos):

{
	"folders": [
		{
			"path": "../cthulhu/docode/src/do/teams/paas/apps"
		},
		{
			"path": "../godo"
		},
		{
			"path": "../product-docs"
		},
		{
			"path": "../cthulhu/docode/src/do/teams/serverless"
		},
		{
			"path": "../openapi"
		},
		{
			"path": "../workspaces"
		},
		{
			"path": "../cthulhu/docode/src/do/teams/insights"
		},
		{
			"path": "../edge-loadbalancers"
		},
		{
			"path": "../dographql"
		}
	],

dominikh · 2022-04-12T22:14:10Z

Can you clarify what you're pointing out? Do you know why you have so many gopls processes?

These are threads, not processes. And the memory statistics for threads mirror that of the owning process. So that's a single gopls process, with several threads, and the gopls process uses 9.6 GiB in total.

ElanHasson · 2022-04-12T22:17:17Z

Thanks for pointing that out.

After some time it becomes impossible to save any files and it hangs during the static check (at least it seems to).

When that happens I can't use any of the go behavior within vscode. For example, finding references, and I get red squiggly lines on valid code that I just added.

findleyr · 2022-04-12T22:21:51Z

These are threads, not processes.

🤦, thanks, I should have noticed the RSS was all the same...

I'm afraid 9.6 GiB is not unheard of for large workspaces. For reference gopls uses 3-6gb for kubernetes, depending on whether staticcheck is enabled. I recommend reducing the size of your workspace to modules that you are actively working on, if possible.

ElanHasson · 2022-04-12T22:24:50Z

its worth noting the code in my workspace isn't all go code and it is definitely less code than k8s.

does gopls have the ability to dump any runtime info?

findleyr · 2022-04-12T22:35:53Z

its worth noting the code in my workspace isn't all go code and it is definitely less code than k8s.

There may well be a bug related to that. Can you share a sense of the size of the workspace, in terms of:

number of total files in workspace folders
number of go files in the workspace folders

does gopls have the ability to dump any runtime info?

Gopls automatically dumps heap to files of the form /tmp/gopls.<PID>-NGiB-(with|no)names.zip, which you can use to investigate. (upon unzipping, you can look at the heap profile with go tool pprof and open the html file, which contains estimates of relative memory usage by type-checked package. In the latter, the units are meaningless but it can be useful to see whether there are packages in there that should not be there.

ElanHasson · 2022-04-12T23:02:31Z

thanks @findleyr will capture data when i hit the issue again.

It's going to be difficult to know what should be loaded and not due to that monorepo life.

ElanHasson · 2022-04-12T23:41:44Z

aaand i hit it again :D

Sometimes I'm able to run the test despite the red squiggles, other times not. Killing gopls and retrying allows me to run again:

File Counts

/home/elan/repos/cthulhu/docode/src/do/teams/paas/apps
+ find . -type f
+ wc -l
1402
+ find . -type f -name '*.go'
+ wc -l
971

/home/elan/repos/cthulhu/docode/src/do/teams/serverless
+ find . -type f
+ wc -l
497
+ find . -type f -name '*.go'
+ wc -l
264

/home/elan/repos/godo
+ wc -l
+ find . -type f
202
+ wc -l
+ find . -type f -name '*.go'
82

/home/elan/repos/product-docs
+ find . -type f
+ wc -l
8634
+ find . -type f -name '*.go'
+ wc -l
0

/home/elan/repos/edge-loadbalancers
+ find . -type f
+ wc -l
260
+ find . -type f -name '*.go'
+ wc -l
0

/home/elan/repos/dographql
+ find . -type f
+ wc -l
1408
+ find . -type f -name '*.go'
+ wc -l
0

/home/elan/repos/cthulhu/docode/src/do/teams/insights
+ find . -type f
+ wc -l
1416
+ find . -type f -name '*.go'
+ wc -l
825

/home/elan/repos/dographql
+ find . -type f
+ wc -l
1408
+ find . -type f -name '*.go'
+ wc -l
0

/home/elan/repos/openapi
+ find . -type f
+ wc -l
1379
+ find . -type f -name '*.go'
+ wc -l
0

ElanHasson · 2022-04-13T00:10:06Z

memory usage aside.. I can run the test and the IDE isn't getting updated static checking (?) and the red squiggles still appear.

I'm not really sure how to determine what should and should not be there.. there are loads of transitive dependencies.

I've never used go tool pprof before, but running web results in powershell being executed? I'm running from Windows Terminal in Ubuntu, not in a WSL windows mount... so I think it thinks i'm on windows?

Interesting... I killed gopls and vscode didn't like it and dumped its stack:

Why are those repos (living in ~/repos) even being noticed by gopls, this may be the reason why it's using so much memory, it's loading lots of unrelated folders?

/home/elan/repos
+ find . -type f
+ wc -l
264795
+ find . -type f -name '*.go'
+ wc -l
46962

Update: Had a go.mod in ~/repos, not sure if that matters

ElanHasson · 2022-04-29T02:30:27Z

I'm at DigitalOcean and that this is effectively a continuation of the work @bhcleek started with the team on
#45363.

findleyr · 2022-04-29T13:04:58Z

Thank you for the data, and sorry for the slow turnaround.

Are you able to:

Upgrade to Go 1.18
Use a go.work that activates only the repos you care about? I.e. go work init <list of directories you want to use> from ~/repos, and open the ~/repos directory?

Can you share your gopls settings? Do you have experimentalWorkspaceModule: true?

ElanHasson · 2022-04-30T04:07:10Z

Thanks @findleyr, no worries about the turnaround, between holidays, family, and life in general we all get busy 😃

We're on Go 1.18.1.

I'll investigate the the go.work stuff, it is new to me.

I personally do not have experimentalWorkspaceModule set.

findleyr · 2022-05-20T13:10:07Z

Just checking in -- did using go.work help?

We're working on redesigning the core of gopls to finally address some of these scaling problems, but in the meantime go.work can help limit the scope of your workspace.

ElanHasson · 2022-05-23T10:44:33Z

Hi,

Someone on the team is experimenting with go.work, I'll check in with them and respond here.

For me specifically, I increased the CPU to 16 cores and the problem went away. I watch the CPU spike when I use the show references functionality, sometimes for 10 seconds across all cores. Gopls taking up 50-60% of the 32GB ram.

This is using go 1.18.2 and latest versions of gopls and friends.

barrettj12 · 2022-06-17T01:56:27Z

I've had this issue too, and in the end vscode-go became completely unusable for me. Now I've switched to GoLand and have experienced no performance issues at all.

findleyr · 2022-06-28T13:42:47Z

@barrettj12 I'm sorry to hear that vscode-go became unusable for you. Unfortunately, there are aspects of gopls that scale with the size of the workspace, and so it will (currently) be unusable on large codebases.

However, we're working hard on performance improvements, and have landed several for the v0.9.0 release. Right now, these performance improvements are focused on CPU usages while typing, as well as optimizing workspaces that are narrowed using a go.work file. @ElanHasson I'd be very curious to hear if these help (go install golang.org/x/tools/[email protected]).

These optimizations still do not tackle the problem of type-checking the entire workspace. That will take longer to solve, though the "degradeClosed" memoryMode may help.

ElanHasson · 2022-06-28T15:32:07Z

@findleyr, just installed, will monitor.

I will say after working for ~a month with the increased RAM and CPUs has been radically different.

I just installed the v0.9.0-pre.2 and I can say after a little usage, EVERYTHING is much snappier! The hover-over types/functions responds faster than I've ever seen, so does jumping to references.

Memory Usage is:
VIRT: 18G
RES: 15.6G
Relatively low CPU usage once everything is loaded.

Will update accordingly if anything changes.

ElanHasson · 2022-07-12T02:44:33Z

@findleyr I can report back that v9.0-pre.2 seems to have fixed the issue. Every so often (maybe a few days) I have to restart VSCode because the language service (maybe ?) stops working.

findleyr · 2022-07-12T22:15:03Z

@ElanHasson that's great news, thanks for reporting back.

We've got more improvements in the pipeline: nothing yet that will make an order-of-magnitude improvement, but several things that will reduce a significant fraction of load and memory.

I'm going to close this issue now since it's not specifically actionable, though of course performance remains a significant concern for gopls.

ElanHasson · 2022-07-12T22:19:36Z

@findleyr thank you for your work on this, it is appreciated!

gopherbot added the gopls Issues related to the Go language server, gopls. label Mar 20, 2022

seankhliao added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 20, 2022

findleyr closed this as completed Apr 5, 2022

findleyr reopened this Apr 12, 2022

hyangah mentioned this issue Apr 27, 2022

gopls: automated issue report (crash) - runtime: out of memory golang/vscode-go#2218

Closed

findleyr added this to the gopls/v0.9.1 milestone Jul 12, 2022

findleyr removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jul 12, 2022

findleyr closed this as completed Jul 12, 2022

findleyr modified the milestones: gopls/v0.9.1, gopls/v0.9.2 Jul 13, 2022

MadeBaruna mentioned this issue May 13, 2023

x/tools/gopls: high CPU & memory for minutes in "Saving foo.go: Getting code actions from "Go" ..." #57707

Closed

golang locked and limited conversation to collaborators Jul 13, 2023

gopherbot added the FrozenDueToAge label Jul 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

staticcheck and gopls nuke my computer #51829

staticcheck and gopls nuke my computer #51829

bschaatsbergen commented Mar 20, 2022 •

edited

Loading

dominikh commented Mar 20, 2022

findleyr commented Mar 21, 2022

hyangah commented Mar 21, 2022

ElanHasson commented Mar 24, 2022

findleyr commented Mar 25, 2022

ElanHasson commented Mar 25, 2022

findleyr commented Mar 25, 2022

ElanHasson commented Apr 1, 2022 •

edited

Loading

findleyr commented Apr 5, 2022

ElanHasson commented Apr 12, 2022 •

edited

Loading

findleyr commented Apr 12, 2022 •

edited

Loading

ElanHasson commented Apr 12, 2022

findleyr commented Apr 12, 2022

ElanHasson commented Apr 12, 2022

dominikh commented Apr 12, 2022

ElanHasson commented Apr 12, 2022 •

edited

Loading

findleyr commented Apr 12, 2022

ElanHasson commented Apr 12, 2022 •

edited

Loading

findleyr commented Apr 12, 2022 •

edited

Loading

ElanHasson commented Apr 12, 2022

ElanHasson commented Apr 12, 2022

ElanHasson commented Apr 13, 2022 •

edited

Loading

ElanHasson commented Apr 29, 2022

findleyr commented Apr 29, 2022

ElanHasson commented Apr 30, 2022

findleyr commented May 20, 2022

ElanHasson commented May 23, 2022

barrettj12 commented Jun 17, 2022

findleyr commented Jun 28, 2022

ElanHasson commented Jun 28, 2022 •

edited

Loading

ElanHasson commented Jul 12, 2022

findleyr commented Jul 12, 2022

ElanHasson commented Jul 12, 2022

staticcheck and gopls nuke my computer #51829

staticcheck and gopls nuke my computer #51829

Comments

bschaatsbergen commented Mar 20, 2022 • edited Loading

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

dominikh commented Mar 20, 2022

findleyr commented Mar 21, 2022

hyangah commented Mar 21, 2022

ElanHasson commented Mar 24, 2022

findleyr commented Mar 25, 2022

ElanHasson commented Mar 25, 2022

findleyr commented Mar 25, 2022

ElanHasson commented Apr 1, 2022 • edited Loading

findleyr commented Apr 5, 2022

ElanHasson commented Apr 12, 2022 • edited Loading

findleyr commented Apr 12, 2022 • edited Loading

ElanHasson commented Apr 12, 2022

findleyr commented Apr 12, 2022

ElanHasson commented Apr 12, 2022

dominikh commented Apr 12, 2022

ElanHasson commented Apr 12, 2022 • edited Loading

findleyr commented Apr 12, 2022

ElanHasson commented Apr 12, 2022 • edited Loading

findleyr commented Apr 12, 2022 • edited Loading

ElanHasson commented Apr 12, 2022

ElanHasson commented Apr 12, 2022

File Counts

ElanHasson commented Apr 13, 2022 • edited Loading

ElanHasson commented Apr 29, 2022

findleyr commented Apr 29, 2022

ElanHasson commented Apr 30, 2022

findleyr commented May 20, 2022

ElanHasson commented May 23, 2022

barrettj12 commented Jun 17, 2022

findleyr commented Jun 28, 2022

ElanHasson commented Jun 28, 2022 • edited Loading

ElanHasson commented Jul 12, 2022

findleyr commented Jul 12, 2022

ElanHasson commented Jul 12, 2022

bschaatsbergen commented Mar 20, 2022 •

edited

Loading

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

ElanHasson commented Apr 1, 2022 •

edited

Loading

ElanHasson commented Apr 12, 2022 •

edited

Loading

findleyr commented Apr 12, 2022 •

edited

Loading

ElanHasson commented Apr 12, 2022 •

edited

Loading

ElanHasson commented Apr 12, 2022 •

edited

Loading

findleyr commented Apr 12, 2022 •

edited

Loading

ElanHasson commented Apr 13, 2022 •

edited

Loading

ElanHasson commented Jun 28, 2022 •

edited

Loading