x/tools/gopls: semanticTokensProvider response uses nulls incorrectly

[mvdan]

What did you do?

Trying a new editor, https://helix-editor.com/, which has built-in support for LSPs like gopls.

I'm using gopls master, just like I usually do with neovim.

What did you expect to see?

It should work out of the box, just like it does with neovim's native lsp support. In fact, I had tried a previous release of Helix with a previous version of gopls and they did work together, so this breakage is new.

What did you see instead?

Helix apparently does not connect to gopls properly. I ended up filing a helix bug at helix-editor/helix#6545, although they seem to have spotted a bug in gopls instead.

I am quoting the relevant comment for reference:

The important lines in the log:

2023-04-01T23:23:46.013 helix_lsp::transport [INFO] <- {"jsonrpc":"2.0","result":{"capabilities":{"textDocumentSync":{"openClose":true,"change":2,"save":{}},"completionProvider":{"triggerCharacters":["."]},"hoverProvider":true,"signatureHelpProvider":{"triggerCharacters":["(",","]},"definitionProvider":true,"typeDefinitionProvider":true,"implementationProvider":true,"referencesProvider":true,"documentHighlightProvider":true,"documentSymbolProvider":true,"codeActionProvider":{"codeActionKinds":["quickfix","refactor.extract","refactor.rewrite","source.fixAll","source.organizeImports"]},"codeLensProvider":{},"documentLinkProvider":{},"workspaceSymbolProvider":true,"documentFormattingProvider":true,"renameProvider":{"prepareProvider":true},"foldingRangeProvider":true,"selectionRangeProvider":true,"executeCommandProvider":{"commands":["gopls.add_dependency","gopls.add_import","gopls.apply_fix","gopls.check_upgrades","gopls.edit_go_directive","gopls.fetch_vulncheck_result","gopls.gc_details","gopls.generate","gopls.go_get_package","gopls.list_imports","gopls.list_known_packages","gopls.mem_stats","gopls.regenerate_cgo","gopls.remove_dependency","gopls.reset_go_mod_diagnostics","gopls.run_govulncheck","gopls.run_tests","gopls.start_debugging","gopls.test","gopls.tidy","gopls.toggle_gc_details","gopls.update_go_sum","gopls.upgrade_dependency","gopls.vendor"]},"callHierarchyProvider":true,"semanticTokensProvider":{"legend":{"tokenTypes":null,"tokenModifiers":null},"range":true,"full":true},"inlayHintProvider":{},"workspace":{"workspaceFolders":{"supported":true,"changeNotifications":"workspace/didChangeWorkspaceFolders"}}},"serverInfo":{"name":"gopls","version":"{\"GoVersion\":\"devel go1.21-bb44c2b54e Fri Mar 31 20:01:17 2023 +0000\",\"Path\":\"golang.org/x/tools/gopls\",\"Main\":{\"Path\":\"golang.org/x/tools/gopls\",\"Version\":\"v0.0.0-20230331191349-65208701901c\",\"Sum\":\"h1:fWtx3BldeTdC8pjqg6KVSfzSW7sDtaRiCZ3B1y/PZk8=\",\"Replace\":null},\"Deps\":[{\"Path\":\"github.com/BurntSushi/toml\",\"Version\":\"v1.2.1\",\"Sum\":\"h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak=\",\"Replace\":null},{\"Path\":\"github.com/google/go-cmp\",\"Version\":\"v0.5.9\",\"Sum\":\"h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=\",\"Replace\":null},{\"Path\":\"github.com/sergi/go-diff\",\"Version\":\"v1.1.0\",\"Sum\":\"h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=\",\"Replace\":null},{\"Path\":\"golang.org/x/exp\",\"Version\":\"v0.0.0-20220722155223-a9213eeb770e\",\"Sum\":\"h1:+WEEuIdZHnUeJJmEUjyYC2gfUMj69yZXw17EnHg/otA=\",\"Replace\":null},{\"Path\":\"golang.org/x/exp/typeparams\",\"Version\":\"v0.0.0-20221212164502-fae10dda9338\",\"Sum\":\"h1:2O2DON6y3XMJiQRAS1UWU+54aec2uopH3x7MAiqGW6Y=\",\"Replace\":null},{\"Path\":\"golang.org/x/mod\",\"Version\":\"v0.9.0\",\"Sum\":\"h1:KENHtAZL2y3NLMYZeHY9DW8HW8V+kQyJsY/V9JlKvCs=\",\"Replace\":null},{\"Path\":\"golang.org/x/sync\",\"Version\":\"v0.1.0\",\"Sum\":\"h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o=\",\"Replace\":null},{\"Path\":\"golang.org/x/sys\",\"Version\":\"v0.6.0\",\"Sum\":\"h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ=\",\"Replace\":null},{\"Path\":\"golang.org/x/text\",\"Version\":\"v0.8.0\",\"Sum\":\"h1:57P1ETyNKtuIjB4SRd15iJxuhj8Gc416Y78H3qgMh68=\",\"Replace\":null},{\"Path\":\"golang.org/x/tools\",\"Version\":\"v0.7.1-0.20230331191349-65208701901c\",\"Sum\":\"h1:KRo5CL18+P9McX3TfKZwKggqIwa4ACpt7LYLoeOla00=\",\"Replace\":null},{\"Path\":\"golang.org/x/vuln\",\"Version\":\"v0.0.0-20230110180137-6ad3e3d07815\",\"Sum\":\"h1:A9kONVi4+AnuOr1dopsibH6hLi1Huy54cbeJxnq4vmU=\",\"Replace\":null},{\"Path\":\"honnef.co/go/tools\",\"Version\":\"v0.4.2\",\"Sum\":\"h1:6qXr+R5w+ktL5UkwEbPp+fEvfyoMPche6GkOpGHZcLc=\",\"Replace\":null},{\"Path\":\"mvdan.cc/gofumpt\",\"Version\":\"v0.4.0\",\"Sum\":\"h1:JVf4NN1mIpHogBj7ABpgOyZc65/UUOkKQFkoURsz4MM=\",\"Replace\":null},{\"Path\":\"mvdan.cc/xurls/v2\",\"Version\":\"v2.4.0\",\"Sum\":\"h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc=\",\"Replace\":null}],\"Settings\":[{\"Key\":\"-buildmode\",\"Value\":\"exe\"},{\"Key\":\"-compiler\",\"Value\":\"gc\"},{\"Key\":\"-ldflags\",\"Value\":\"-w -s\"},{\"Key\":\"CGO_ENABLED\",\"Value\":\"1\"},{\"Key\":\"CGO_CFLAGS\",\"Value\":\"\"},{\"Key\":\"CGO_CPPFLAGS\",\"Value\":\"\"},{\"Key\":\"CGO_CXXFLAGS\",\"Value\":\"\"},{\"Key\":\"CGO_LDFLAGS\",\"Value\":\"\"},{\"Key\":\"GOARCH\",\"Value\":\"amd64\"},{\"Key\":\"GOOS\",\"Value\":\"linux\"},{\"Key\":\"GOAMD64\",\"Value\":\"v3\"}],\"Version\":\"master\"}"}},"id":0}
2023-04-01T23:23:46.014 helix_lsp [ERROR] failed to initialize language server: failed to parse: data did not match any variant of untagged enum SemanticTokensServerCapabilities

This is an upstream issue with gopls: it's sending

{
  "semanticTokensProvider": {
    "legend": {"tokenTypes": null, "tokenModifiers": null},
    "range": true,
    "full": true
  }
}

in the server capabilities in response to the initialize request.

The nulls are invalid according to the spec: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#semanticTokensLegend. tokenTypes and tokenModifiers are non-nullable. Instead gopls should use an empty list.

Build info

golang.org/x/tools/gopls master
    golang.org/x/tools/gopls@v0.0.0-20230406191632-45ef82945d2c h1:aYohqGdfnEEd5B90Zxl+KfrOo/u+8GrnGkPq5Vn27EQ=
    github.com/BurntSushi/toml@v1.2.1 h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak=
    github.com/google/go-cmp@v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
    github.com/sergi/go-diff@v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
    golang.org/x/exp@v0.0.0-20220722155223-a9213eeb770e h1:+WEEuIdZHnUeJJmEUjyYC2gfUMj69yZXw17EnHg/otA=
    golang.org/x/exp/typeparams@v0.0.0-20221212164502-fae10dda9338 h1:2O2DON6y3XMJiQRAS1UWU+54aec2uopH3x7MAiqGW6Y=
    golang.org/x/mod@v0.10.0 h1:lFO9qtOdlre5W1jxS3r/4szv2/6iXxScdzjoBMXNhYk=
    golang.org/x/sync@v0.1.0 h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o=
    golang.org/x/sys@v0.7.0 h1:3jlCCIQZPdOYu1h8BkNvLz8Kgwtae2cagcG/VamtZRU=
    golang.org/x/text@v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE=
    golang.org/x/tools@v0.8.1-0.20230406191632-45ef82945d2c h1:6FcxpuzgR8bx6R5nFoULRCVtztRxhuBSTH3e5ZGcaec=
    golang.org/x/vuln@v0.0.0-20230110180137-6ad3e3d07815 h1:A9kONVi4+AnuOr1dopsibH6hLi1Huy54cbeJxnq4vmU=
    honnef.co/go/tools@v0.4.2 h1:6qXr+R5w+ktL5UkwEbPp+fEvfyoMPche6GkOpGHZcLc=
    mvdan.cc/gofumpt@v0.4.0 h1:JVf4NN1mIpHogBj7ABpgOyZc65/UUOkKQFkoURsz4MM=
    mvdan.cc/xurls/v2@v2.4.0 h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc=
go: devel go1.21-9be533a8ee Fri Apr 7 10:09:11 2023 +0000

[pjweinb]

Thanks for the bug report. I'll try to fix it. (I presume the client in SemanticTokensClientCapabilities is not sending any tokenTypes.)

[mvdan]

Out of curiosity, are there any checks that the JSON encoded by gopls adheres to the LSP spec, perhaps if they publish a JSONSchema? Presumably that sort of check or static analysis could have caught this bug, and there might be other similar bugs lurking. I only found this one because Helix's decoder for LSP types appears to be fairly strict, whereas Neovim's appears to be more forgiving.

[pjweinb]

They do publish a JSON schema, and the Go code is derived from it. But the translation from LSP types (which are Typescript types) into Go types is not straightforward. For instance, Go structs do not have optional fields, nor does Go have union types. Further, and relevant to this bug, x in var x struct { A []string } is serialized as '{"A": null}' so the error is not a type error in the Go code. Fortunately the error can be fixed with x.A = make([]string,0), and then x serializes as `{"A": []}` I would be delighted to learn of some sort of analysis, static or otherwise, that would have found this bug.

…

On Fri, Apr 7, 2023 at 12:48 PM Daniel Martí ***@***.***> wrote: Out of curiosity, are there any checks that the JSON encoded by gopls adheres to the LSP spec, perhaps if they publish a JSONSchema? Presumably that sort of check or static analysis could have caught this bug, and there might be other similar bugs lurking. I only found this one because Helix's decoder for LSP types appears to be fairly strict, whereas Neovim's appears to be more forgiving. — Reply to this email directly, view it on GitHub <#59479 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJIAIYF4RFIAOCMNSKIAUTXABAN7ANCNFSM6AAAAAAWWRJNF4> . You are receiving this because you were assigned.Message ID: ***@***.***>

[mvdan]

Interesting, thanks for the context. You might like to hear that with @dsnet and others we've been looking at a new encoding/json design, and it no longer encodes nil slices as null: https://pkg.go.dev/github.com/go-json-experiment/json#readme-behavior-changes

So that would at least solve this one quirk.

With optional fields, could you not use pointers and omitempty?

With union/sum types, I agree that that's simply a limitation of the Go type system. You could potentially still do something involving interfaces and unexported methods, like go/ast does, though that would add a bit more boilerplate.

[gopherbot]

Change https://go.dev/cl/483216 mentions this issue: gopls: strict LSP compliance for empty slices

[mvdan]

Confirming that the integration with Helix is fixed :)