x/tools/gopls: high memory consumption

[codisms]

gopls version

Build info
----------
golang.org/x/tools/gopls v0.9.1
    golang.org/x/tools/gopls@v0.9.1 h1:SigsTL4Hpv3a6b/a7oPCLRv5QUeSM6PZNdta1oKY4B0=
    github.com/google/go-cmp@v0.5.7 h1:81/ik6ipDQS2aGcBfIN5dHDB36BwrStyeAQquSYCV4o=
    github.com/sergi/go-diff@v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
    golang.org/x/mod@v0.6.0-dev.0.20220419223038-86c51ed26bb4 h1:6zppjxzCulZykYSLyVDYbneBfbaBIQPYMevg0bEwv2s=
    golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c h1:5KslGYwFpkhGh+Q16bwMP3cOontH8FOep7tGV86Y7SQ=
    golang.org/x/sys@v0.0.0-20220209214540-3681064d5158 h1:rm+CHSpPEEW2IsXUib1ThaHIjuBVZjxNgSKmBLFfD4c=
    golang.org/x/text@v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
    golang.org/x/tools@v0.1.12-0.20220713141851-7464a5a40219 h1:Ljlba2fVWOA1049JjsKii44g8nZN2GjpxMlzVc8AnQM=
    mvdan.cc/gofumpt@v0.3.0 h1:kTojdZo9AcEYbQYhGuLf/zszYthRdhDNDUi2JKTxas4=
    mvdan.cc/xurls/v2@v2.4.0 h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc=
go: go1.16.15

go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ubuntu/.cache/go-build"
GOENV="/home/ubuntu/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ubuntu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ubuntu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.15"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3575468840=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Editing Go files with vim and vim-go.

What did you expect to see?

Lower memory consumption.

What did you see instead?

Over 5GB of memory consumption; thrashing swap file.

Editor and settings

VIM - Vi IMproved 9.0 (2022 Jun 28, compiled Aug  1 2022 05:17:14)
Included patches: 1-124
Compiled by ubuntu@dev-veritone
Huge version without GUI.  Features included (+) or not (-):
+acl               +file_in_path      +mouse_urxvt       -tag_any_white
+arabic            +find_in_path      +mouse_xterm       -tcl
+autocmd           +float             +multi_byte        +termguicolors
+autochdir         +folding           +multi_lang        +terminal
-autoservername    -footer            -mzscheme          +terminfo
-balloon_eval      +fork()            +netbeans_intg     +termresponse
+balloon_eval_term +gettext           +num64             +textobjects
-browse            -hangul_input      +packages          +textprop
++builtin_terms    +iconv             +path_extra        +timers
+byte_offset       +insert_expand     +perl/dyn          +title
+channel           +ipv6              +persistent_undo   -toolbar
+cindent           +job               +popupwin          +user_commands
-clientserver      +jumplist          +postscript        +vartabs
-clipboard         +keymap            +printer           +vertsplit
+cmdline_compl     +lambda            +profile           +vim9script
+cmdline_hist      +langmap           -python            +viminfo
+cmdline_info      +libcall           +python3/dyn       +virtualedit
+comments          +linebreak         +quickfix          +visual
+conceal           +lispindent        +reltime           +visualextra
+cryptv            +listcmds          +rightleft         +vreplace
+cscope            +localmap          +ruby/dyn          +wildignore
+cursorbind        -lua               +scrollbind        +wildmenu
+cursorshape       +menu              +signs             +windows
+dialog_con        +mksession         +smartindent       +writebackup
+diff              +modify_fname      -sodium            -X11
+digraphs          +mouse             -sound             -xfontset
-dnd               -mouseshape        +spell             -xim
-ebcdic            +mouse_dec         +startuptime       -xpm
+emacs_tags        -mouse_gpm         +statusline        -xsmp
+eval              -mouse_jsbterm     -sun_workshop      -xterm_clipboard
+ex_extra          +mouse_netterm     +syntax            -xterm_save
+extra_search      +mouse_sgr         +tag_binary
-farsi             -mouse_sysmouse    -tag_old_static
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H -g -O2 -D_REENTRANT -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: gcc -L. -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,-E -L/usr/local/lib -Wl,--as-needed -o vim -lm -ltinfo -lrt -ldl -Wl,-E -fstack-protector-strong -L/usr/local/lib -L/usr/lib/x86_64-linux-gnu/perl/5.30/CORE -lperl -ldl -lm -lpthread -lcrypt

Logs

gopls.3782930-5GiB-nonames.zip

[findleyr]

That gopls uses a lot of memory is a known (and difficult) problem. See for example #47855.

However, 5gb is a lot, and we should check whether that makes sense based on your usage. Can you tell me:

• Approximately how many .go files are in your workspace (this depends on which folder was resolved as your workspace folder).
• Do you experience this memory usage immediately, or does it take a while to build up? There was a large memory leak that will be fixed in the next gopls release (v0.9.2), so I'm curious if you may be experiencing that.

Unfortunately, at the moment 5gb is not unheard-of memory usage when working on large projects, such as Kubernetes. We're working on this, but it is a long-term project.

[codisms]

Unfortunately, I don't have a great way to answer the first question. The number is hard to know because of the libraries being referenced, but the project itself has 3500 Go files, though not all referenced in all parts of the code. One thing I can say is that we have a single file generated by gqlgen that has 256k lines in it and is probably a big culprit.

Initially loading the project causes the machine to spin for a good five minutes or so and it'll spike to 3-4 GB pretty easily. It'll settle around 2-3 after a while but spikes as the files are edited. Multiple files could be changed at the same time with multiple vim windows going on. I do have the gopls server running for quite a long time though.

[findleyr]

Thanks.

I expect you'll see it stay in the the ~3gb range after gopls@v0.9.2 is released. I'll take a look at the heap dump you've provided to see if I can understand this better.

[findleyr]

Putting this in the next milestone for additional triage.