cmd/go: add GOVCS setting to control version control usage

The go command runs commands like git and hg to download modules.
In the past, we have had problems with security bugs in version
control systems becoming security bugs in “go get”.

The original modules draft design removed use of these commands
entirely, saying:

> We want to move away from invoking version control tools such as bzr,
> fossil, git, hg, and svn to download source code. These fragment the
> ecosystem: packages developed using Bazaar or Fossil, for example, are
> effectively unavailable to users who cannot or choose not to install
> these tools. The version control tools have also been a source of
> exciting security problems. It would be good to move them outside the
> security perimeter.

The removal of these commands was not possible in the end: being able
to fetch directly from Git repos is too important, especially for
closed source. But the security exposure has not gone away.
We remain vulnerable to problems in VCS systems, especially the less
scrutinized ones.

This change adds a GOVCS setting to let users control which version
control systems are allowed by default.

It also changes the default allowed version control systems to git and hg
for public code and any version control system for private code
(import path or module path matched by the GOPRIVATE setting).

See the changes in alldocs.go for detailed documentation.
See #41730 for proposal and discussion.

Fixes #41730.

Change-Id: I1999ddf7445b36a7572965be5897c7a1ff7f4265
Reviewed-on: https://go-review.googlesource.com/c/go/+/266420
Trust: Russ Cox <[email protected]>
Run-TryBot: Russ Cox <[email protected]>
TryBot-Result: Go Bot <[email protected]>
Reviewed-by: Jay Conrod <[email protected]>

Author: rsc

src/cmd/go/alldocs.go

@@ -268,6 +268,7 @@ var (
 	GONOPROXY  = envOr("GONOPROXY", GOPRIVATE)
 	GONOSUMDB  = envOr("GONOSUMDB", GOPRIVATE)
 	GOINSECURE = Getenv("GOINSECURE")
+	GOVCS      = Getenv("GOVCS")
 )
 
 var SumdbDir = gopathDir("pkg/sumdb")

src/cmd/go/internal/cfg/cfg.go

@@ -87,6 +87,7 @@ func MkEnv() []cfg.EnvVar {
 		{Name: "GOSUMDB", Value: cfg.GOSUMDB},
 		{Name: "GOTMPDIR", Value: cfg.Getenv("GOTMPDIR")},
 		{Name: "GOTOOLDIR", Value: base.ToolDir},
+		{Name: "GOVCS", Value: cfg.GOVCS},
 	}
 
 	if work.GccgoBin != "" {

src/cmd/go/internal/envcmd/env.go

@@ -526,7 +526,7 @@ General-purpose environment variables:
 		Comma-separated list of glob patterns (in the syntax of Go's path.Match)
 		of module path prefixes that should always be fetched directly
 		or that should not be compared against the checksum database.
-		See 'go help module-private'.
+		See 'go help private'.
 	GOROOT
 		The root of the go tree.
 	GOSUMDB

src/cmd/go/internal/help/helpdoc.go

@@ -848,16 +848,16 @@ the checksum database is not consulted, and all unrecognized modules are
 accepted, at the cost of giving up the security guarantee of verified repeatable
 downloads for all modules. A better way to bypass the checksum database
 for specific modules is to use the GOPRIVATE or GONOSUMDB environment
-variables. See 'go help module-private' for details.
+variables. See 'go help private' for details.
 
 The 'go env -w' command (see 'go help env') can be used to set these variables
 for future go command invocations.
 `,
 }
 
-var HelpModulePrivate = &base.Command{
-	UsageLine: "module-private",
-	Short:     "module configuration for non-public modules",
+var HelpPrivate = &base.Command{
+	UsageLine: "private",
+	Short:     "configuration for downloading non-public code",
 	Long: `
 The go command defaults to downloading modules from the public Go module
 mirror at proxy.golang.org. It also defaults to validating downloaded modules,
@@ -898,6 +898,11 @@ be used for downloading both public and private modules, because
 GONOPROXY has been set to a pattern that won't match any modules,
 overriding GOPRIVATE.
 
+The GOPRIVATE variable is also used to define the "public" and "private"
+patterns for the GOVCS variable; see 'go help vcs'. For that usage,
+GOPRIVATE applies even in GOPATH mode. In that case, it matches import paths
+instead of module paths.
+
 The 'go env -w' command (see 'go help env') can be used to set these variables
 for future go command invocations.
 `,

src/cmd/go/internal/modfetch/fetch.go

@@ -176,6 +176,82 @@ Usage: ` + CmdGet.UsageLine + `
 ` + CmdGet.Long,
 }
 
+var HelpVCS = &base.Command{
+	UsageLine: "vcs",
+	Short:     "controlling version control with GOVCS",
+	Long: `
+The 'go get' command can run version control commands like git
+to download imported code. This functionality is critical to the decentralized
+Go package ecosystem, in which code can be imported from any server,
+but it is also a potential security problem, if a malicious server finds a
+way to cause the invoked version control command to run unintended code.
+
+To balance the functionality and security concerns, the 'go get' command
+by default will only use git and hg to download code from public servers.
+But it will use any known version control system (bzr, fossil, git, hg, svn)
+to download code from private servers, defined as those hosting packages
+matching the GOPRIVATE variable (see 'go help private'). The rationale behind
+allowing only Git and Mercurial is that these two systems have had the most
+attention to issues of being run as clients of untrusted servers. In contrast,
+Bazaar, Fossil, and Subversion have primarily been used in trusted,
+authenticated environments and are not as well scrutinized as attack surfaces.
+
+The version control command restrictions only apply when using direct version
+control access to download code. When downloading modules from a proxy,
+'go get' uses the proxy protocol instead, which is always permitted.
+By default, the 'go get' command uses the Go module mirror (proxy.golang.org)
+for public packages and only falls back to version control for private
+packages or when the mirror refuses to serve a public package (typically for
+legal reasons). Therefore, clients can still access public code served from
+Bazaar, Fossil, or Subversion repositories by default, because those downloads
+use the Go module mirror, which takes on the security risk of running the
+version control commands, using a custom sandbox.
+
+The GOVCS variable can be used to change the allowed version control systems
+for specific packages (identified by a module or import path).
+The GOVCS variable applies both when using modules and when using GOPATH.
+When using modules, the patterns match against the module path.
+When using GOPATH, the patterns match against the import path
+corresponding to the root of the version control repository.
+
+The general form of the GOVCS setting is a comma-separated list of
+pattern:vcslist rules. The pattern is a glob pattern that must match
+one or more leading elements of the module or import path. The vcslist
+is a pipe-separated list of allowed version control commands, or "all"
+to allow use of any known command, or "off" to allow nothing.
+The earliest matching pattern in the list applies, even if later patterns
+might also match.
+
+For example, consider:
+
+	GOVCS=github.com:git,evil.com:off,*:git|hg
+
+With this setting, code with an module or import path beginning with
+github.com/ can only use git; paths on evil.com cannot use any version
+control command, and all other paths (* matches everything) can use
+only git or hg.
+
+The special patterns "public" and "private" match public and private
+module or import paths. A path is private if it matches the GOPRIVATE
+variable; otherwise it is public.
+
+If no rules in the GOVCS variable match a particular module or import path,
+the 'go get' command applies its default rule, which can now be summarized
+in GOVCS notation as 'public:git|hg,private:all'.
+
+To allow unfettered use of any version control system for any package, use:
+
+	GOVCS=*:all
+
+To disable all use of version control, use:
+
+	GOVCS=*:off
+
+The 'go env -w' command (see 'go help env') can be used to set the GOVCS
+variable for future go command invocations.
+`,
+}
+
 var (
 	getD   = CmdGet.Flag.Bool("d", false, "")
 	getF   = CmdGet.Flag.Bool("f", false, "")

src/cmd/go/internal/modget/get.go

@@ -365,7 +365,7 @@ list if the error is a 404 or 410 HTTP response or if the current proxy is
 followed by a pipe character, indicating it is safe to fall back on any error.
 
 The GOPRIVATE and GONOPROXY environment variables allow bypassing
-the proxy for selected modules. See 'go help module-private' for details.
+the proxy for selected modules. See 'go help private' for details.
 
 No matter the source of the modules, the go command checks downloads against
 known checksums, to detect unexpected changes in the content of any specific