gopls/doc/design: rewrite the architecture overview

Change-Id: Ice7f00c2109f999bcc3619acb979ae265caadf28
Reviewed-on: https://go-review.googlesource.com/c/tools/+/555977
Auto-Submit: Alan Donovan <[email protected]>
Reviewed-by: Robert Findley <[email protected]>
LUCI-TryBot-Result: Go LUCI <[email protected]>

Author: adonovan

gopls/doc/design/architecture.svg

@@ -1,46 +1,173 @@
-# gopls implementation documentation
 
-This is not intended as a complete description of the implementation, for the most the part the package godoc, code comments and the code itself hold that.
-Instead this is meant to be a guide into finding parts of the implementation, and understanding some core concepts used throughout the implementation.
+# Gopls architecture
 
-## View/Session/Cache
+Last major update: Jan 16 2024
 
-Throughout the code there are references to these three concepts, and they build on each other.
+This doc presents a high-level overview of the structure of gopls to
+help new contributors find their way. It is not intended to be a
+complete description of the implementation, nor even of any key
+components; for that, the package documentation (linked below) and
+other comments within the code are a better guide.
 
-At the base is the *Cache*. This is the level at which we hold information that is global in nature, for instance information about the file system and its contents.
+The diagram below shows selected components of the gopls module and
+their relationship to each other according to the Go import graph.
+Tests and test infrastructure are not shown, nor are utility packages,
+nor packages from the [x/tools] module. For brevity, packages are
+referred to by their last segment, which is usually unambiguous.
 
-Above that is the *Session*, which holds information for a connection to an editor. This layer hold things like the edited files (referred to as overlays).
+The height of each blob corresponds loosely to its technical depth.
+Some blocks are wide and shallow, such as [protocol], which declares
+Go types for the entire LSP protocol. Others are deep, such as [cache]
+and [source], as they contain a lot of dense logic and algorithms.
 
-The top layer is called the *View*. This holds the configuration, and the mapping to configured packages.
+<!-- Source: https://docs.google.com/drawings/d/1CK6YSLt7G3svRoZf7skJI-lxRol2VI90YOxHcYS0DP4 -->
+![Gopls architecture](architecture.svg)
 
-The purpose of this layering is to allow a single editor session to have multiple views active whilst still sharing as much information as possible for efficiency.
-In theory if only the View layer existed, the results would be identical, but slower and using more memory.
+Starting from the bottom, we'll describe the various components.
 
-## Code location
+The lowest layer defines the request and response types of the
+Language Server Protocol:
 
-gopls will be developed in the [x/tools] Go repository; the core packages are in [internal/lsp], and the binary and integration tests are located in [gopls].
+- The [protocol] package defines the standard protocol; it is mostly
+  generated mechanically from the schema definition provided by
+  Microsoft.
+  The most important type is DocumentURI, which represents a `file:`
+  URL that identifies a client editor document. It also provides
+  `Mapper`, which maps between the different coordinate systems used
+  for source positions: UTF-8, UTF-16, and token.Pos.
 
-Below is a list of the core packages of gopls, and their primary purpose:
+- The [command] package defines Gopls's non-standard commands, which
+  are all invoked through the `workspace/executeCommand` extension
+  mechanism. These commands are typically returned by the server as
+  continuations of Code Actions or Code Lenses; most clients do not
+  construct calls to them directly.
 
-Package | Description
---- | ---
-[gopls] | the main binary, plugins and integration tests
-[internal/lsp] | the core message handling package
-[internal/lsp/cache] | the cache layer
-[internal/cmd] | the gopls command line layer
-[internal/debug] | features to aid in debugging gopls
-[internal/lsp/protocol] | the types of LSP request and response messages
-[internal/lsp/source] | the core feature implementations
-[internal/memoize] | a function invocation cache used to reduce the work done
-[internal/jsonrpc2] | an implementation of the JSON RPC2 specification
+The next layer defines a number of important and very widely used data structures:
 
-[gopls]: https://github.com/golang/tools/tree/master/gopls
-[internal/jsonrpc2]: https://github.com/golang/tools/tree/master/internal/jsonrpc2
-[internal/lsp]: https://github.com/golang/tools/tree/master/gopls/internal/lsp
-[internal/lsp/cache]: https://github.com/golang/tools/tree/master/gopls/internal/lsp/cache
-[internal/cmd]: https://github.com/golang/tools/tree/master/gopls/internal/cmd
-[internal/debug]: https://github.com/golang/tools/tree/master/gopls/internal/lsp/debug
-[internal/lsp/source]: https://github.com/golang/tools/tree/master/gopls/internal/lsp/source
-[internal/memoize]: https://github.com/golang/tools/tree/master/internal/memoize
-[internal/lsp/protocol]: https://github.com/golang/tools/tree/master/gopls/internal/lsp/protocol
-[x/tools]: https://github.com/golang/tools
+- The [file] package defines the primary abstractions of a client
+  file: its `Identity` (URI and content hash), and its `Handle` (which
+  additionally provides the version and content of a particular
+  snapshot of the file.
+
+- The [parsego] package defines `File`, the parsed form of a Go source
+  file, including its content, syntax tree, and coordinary mappings
+  (Mapper and token.File). The package performs various kinds of tree
+  repair to work around error-recovery shortcomings of the Go parser.
+
+- The [metadata] package defines `Package`, an abstraction of the
+  metadata of a Go package, similar to the output of `go list -json`.
+  Metadata is produced from [go/packages], which takes
+  care of invoking `go list`. (Users report that it works to some extent
+  with a GOPACKAGESDRIVER for Bazel, though we maintain no tests for this
+  scenario.)
+
+  The package also provides `Graph`, the complete import graph for a
+  workspace; each graph node is a `Package`.
+
+The [settings] layer defines the data structure (effectively a large
+tree) for gopls configuration options, along with its JSON encoding.
+
+The [cache] layer is the largest and most complex component of gopls.
+It is concerned with state management, dependency analysis, and invalidation:
+the `Session` of communication with the client;
+the `Folder`s that the client has opened;
+the `View` of a particular workspace tree with particular build
+options;
+the `Snapshot` of the state of all files in the workspace after a
+particular edit operation;
+the contents of all files, whether saved to disk (`DiskFile`) or
+edited and unsaved (`Overlay`);
+the `Cache` of in-memory memoized computations,
+such as parsing go.mod files or build the symbol index;
+and the `Package`, which holds the results of type checking a package
+from Go syntax.
+
+The cache layer depends on various auxiliary packages, including:
+
+- The [filecache] package, which manages gopls' persistent, transactional,
+  file-based key/value store.
+  
+- The [xrefs], [methodsets], and [typerefs] packages define algorithms
+  for constructing indexes of information derived from type-checking,
+  and for encoding and decoding these serializable indexes in the file
+  cache.
+
+  Together these packages enable the fast restart, reduced memory
+  consumption, and synergy across processes that were delivered by the
+  v0.12 redesign and described in ["Scaling gopls for the growing Go
+  ecosystem"](https://go.dev/blog/gopls-scalability).
+
+The cache also defines gopls's [go/analysis] driver, which runs
+modular analysis (similar to `go vet`) across the workspace.
+Gopls also includes a number of analysis passes that are not part of vet.
+
+The next layer defines four packages, each for handling files in a
+particular language:
+[mod] for go.mod files;
+[work] for go.work files;
+[template] for files in `text/template` syntax; and
+[source], for files in Go itself.
+This package, by far the largest, provides the main features of gopls:
+navigation, analysis, and refactoring of Go code.
+As most users imagine it, this package _is_ gopls.
+
+The [server] package defines the LSP service implementation, with one
+handler method per LSP request type. Each handler switches on the type
+of the file and dispatches to one of the four language-specific
+packages.
+
+The [lsprpc] package connects the service interface to our [JSON RPC](jsonrpc2)
+server.
+
+Bear in mind that the diagram is a dependency graph, a "static"
+viewpoint of the program's structure. A more dynamic viewpoint would
+order the packages based on the sequence in which they are encountered
+during processing of a particular request; in such a view, the bottom
+layer would represent the "wire" (protocol and command), the next
+layer up would hold the RPC-related packages (lsprpc and server), and
+features (e.g. source, mod, work, template) would be at the top.
+
+<!--
+A dynamic view would be an interesting topic for another article.
+This slide deck [requires Google network]
+The Life of a (gopls) Query (Oct 2021)
+https://docs.google.com/presentation/d/1c8XJaIldzii-F3YvEOPWHK_MQJ_o8ua5Bct1yDa3ZlU
+provides useful (if somewhat out of date) information.
+-->
+
+The [cmd] package defines the command-line interface of the `gopls`
+command, around which gopls's main package is just a trivial wrapper.
+It is usually run without arguments, causing it to start a server and
+listen indefinitely.
+It also provides a number of subcommands that start a server, make a
+single request to it, and exit, providing traditional batch-command
+access to server functionality. These subcommands are primarily
+provided as a debugging aid (but see
+[#63693](https://github.com/golang/go/issues/63693)).
+
+[cache]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/cache
+[cmd]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/cmd
+[command]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/command
+[debug]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/debug
+[file]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/file
+[filecache]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/filecache
+[go/analysis]: https://pkg.go.dev/golang.org/x/tools@master/go/analysis
+[go/packages]: https://pkg.go.dev/golang.org/x/tools@master/go/packages
+[gopls]: https://pkg.go.dev/golang.org/x/tools/gopls@master
+[jsonrpc2]: https://pkg.go.dev/golang.org/x/tools@master/internal/jsonrpc2
+[lsp]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp
+[lsprpc]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/lsprpc
+[memoize]: https://github.com/golang/tools/tree/master/internal/memoize
+[metadata]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/cache/metadata
+[methodsets]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/cache/methodsets
+[mod]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/mod
+[parsego]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/cache/parsego
+[protocol]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/protocol
+[server]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/server
+[settings]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/settings
+[source]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/source
+[template]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/template
+[typerefs]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/cache/typerefs
+[work]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/work
+[x/tools]: https://github.com/golang/tools@master
+[xrefs]: https://pkg.go.dev/golang.org/x/tools/gopls@master/internal/lsp/cache/xrefs