cmd/go: go mod download breaks on 1.21.0 due to empty GOPROXY

[Snaipe]

What version of Go are you using (go version)?

$ go version
1.21.0

Does this issue reproduce with the latest release?

Yes, and it only happens with the latest 1.21.0. Things were working by default on the previous release.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/snaipe/.local/var/cache/go-build'
GOENV='/home/snaipe/.local/etc/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/snaipe/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/snaipe/go'
GOPRIVATE=''
GOPROXY=''
GOROOT='/tmp/go'
GOSUMDB=''
GOTMPDIR=''
GOTOOLCHAIN=''
GOTOOLDIR='/tmp/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='0'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1578046652=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Run go mod download -x github.com/xdg/scram@v1.0.5

What did you expect to see?

It should have worked, and output something like this:

# get https://proxy.golang.org/github.com/xdg/scram/@v/v1.0.5.info
# get https://proxy.golang.org/github.com/xdg/scram/@v/v1.0.5.info: 200 OK (0.086s)
# get https://proxy.golang.org/github.com/xdg/scram/@v/v1.0.5.mod
# get https://proxy.golang.org/github.com/xdg/scram/@v/v1.0.5.mod: 200 OK (0.005s)
# get https://proxy.golang.org/sumdb/sum.golang.org/supported
# get https://proxy.golang.org/sumdb/sum.golang.org/supported: 404 Not Found (0.005s)
# get https://sum.golang.org/lookup/github.com/xdg/scram@v1.0.5
# get https://sum.golang.org/lookup/github.com/xdg/scram@v1.0.5: 200 OK (0.281s)
# get https://sum.golang.org/tile/8/1/288
# get https://sum.golang.org/tile/8/1/288: 200 OK (0.008s)
# get https://sum.golang.org/tile/8/0/x073/886
# get https://sum.golang.org/tile/8/2/001.p/33
# get https://sum.golang.org/tile/8/0/x074/141.p/165
# get https://sum.golang.org/tile/8/1/289.p/157
# get https://sum.golang.org/tile/8/2/001.p/33: 200 OK (0.010s)
# get https://sum.golang.org/tile/8/1/289.p/157: 200 OK (0.010s)
# get https://sum.golang.org/tile/8/0/x073/886: 200 OK (0.035s)
# get https://sum.golang.org/tile/8/0/x074/141.p/165: 200 OK (0.039s)
# get https://sum.golang.org/tile/8/0/x034/273
# get https://sum.golang.org/tile/8/0/x034/273: 200 OK (0.014s)
# get https://proxy.golang.org/github.com/xdg/scram/@v/v1.0.5.zip
# get https://proxy.golang.org/github.com/xdg/scram/@v/v1.0.5.zip: 200 OK (0.008s)

What did you see instead?

The command fails with this error:

go: github.com/xdg/scram@v1.0.5: GOPROXY list is not the empty string, but contains no entries

---

A simple way to reproduce the issue:

$ git clone https://github.com/golang/go --single-branch -b go1.21.0 --depth=1 /tmp/go/src
$ export GOROOT=/tmp/go/src
$ export GOROOT_BOOTSTRAP=/usr/lib/go
$ export GOROOT_FINAL=/tmp/go/out
$ export CGO_ENABLED=0 
$ cd /tmp/go/src/src
$ bash ./make.bash
$ cd /tmp/go
$ cp -rf src out
$ cd out
$ ./bin/go version
go version go1.21.0 linux/amd64
$ ./bin/go env GOPROXY
$ ./bin/go mod download -x github.com/xdg/scram@v1.0.5
go: github.com/xdg/scram@v1.0.5: GOPROXY list is not the empty string, but contains no entries

For comparison, using my locally installed Go 1.20.1:

$ go version
go version go1.20.1 linux/amd64
$ go env GOPROXY
https://proxy.golang.org,direct
$ go mod download -x github.com/xdg/scram@v1.0.5
# get https://proxy.golang.org/github.com/xdg/scram/@v/v1.0.5.info
# get https://proxy.golang.org/github.com/xdg/scram/@v/v1.0.5.info: 200 OK (0.086s)
# get https://proxy.golang.org/github.com/xdg/scram/@v/v1.0.5.mod
# get https://proxy.golang.org/github.com/xdg/scram/@v/v1.0.5.mod: 200 OK (0.005s)
# get https://proxy.golang.org/sumdb/sum.golang.org/supported
# get https://proxy.golang.org/sumdb/sum.golang.org/supported: 404 Not Found (0.005s)
# get https://sum.golang.org/lookup/github.com/xdg/scram@v1.0.5
# get https://sum.golang.org/lookup/github.com/xdg/scram@v1.0.5: 200 OK (0.281s)
# get https://sum.golang.org/tile/8/1/288
# get https://sum.golang.org/tile/8/1/288: 200 OK (0.008s)
# get https://sum.golang.org/tile/8/0/x073/886
# get https://sum.golang.org/tile/8/2/001.p/33
# get https://sum.golang.org/tile/8/0/x074/141.p/165
# get https://sum.golang.org/tile/8/1/289.p/157
# get https://sum.golang.org/tile/8/2/001.p/33: 200 OK (0.010s)
# get https://sum.golang.org/tile/8/1/289.p/157: 200 OK (0.010s)
# get https://sum.golang.org/tile/8/0/x073/886: 200 OK (0.035s)
# get https://sum.golang.org/tile/8/0/x074/141.p/165: 200 OK (0.039s)
# get https://sum.golang.org/tile/8/0/x034/273
# get https://sum.golang.org/tile/8/0/x034/273: 200 OK (0.014s)
# get https://proxy.golang.org/github.com/xdg/scram/@v/v1.0.5.zip
# get https://proxy.golang.org/github.com/xdg/scram/@v/v1.0.5.zip: 200 OK (0.008s)

[bcmills]

The default GOPROXY configuration is now read from GOROOT/go.env.

What do go env GOROOT and cat $(go env GOROOT)/go.env show when you run those commands?

[bcmills]

And, on a related note: what are the contents of /home/snaipe/.local/etc/go/env?

[Snaipe]

In the freshly built Go root dir, go.env contains the following:

# This file contains the initial defaults for go command configuration.
# Values set by 'go env -w' and written to the user's go/env file override these.
# The environment overrides everything else.

# Use the Go module mirror and checksum database by default.
# See https://proxy.golang.org for details.
GOPROXY=https://proxy.golang.org,direct
GOSUMDB=sum.golang.org

# Automatically download newer toolchains as directed by go.mod files.
# See https://go.dev/doc/toolchain for details.
GOTOOLCHAIN=auto

So these values look expected. My /home/snaipe/.local/etc/go/env contains:

GOFLAGS=

I don't quite remember why I set this before, so I removed the file.

The behavior stays the same on different environments. If I build Go from source from a minimal Alpine Linux container, which should avoid any weird user config, the error remains.

[Snaipe]

I edited the initial reproducer; I realized upon rereading that I failed to transcribe it correctly.

[Snaipe]

I straced the newly built Go process and the path it tries to open go.env at seems very wrong:

LC_ALL=C strace -fe open,openat ./bin/go mod download github.com/xdg/scram@v1.0.5
openat(AT_FDCWD, "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = 3
strace: Process 3928185 attached
strace: Process 3928186 attached
strace: Process 3928187 attached
[pid 3928184] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=3928184, si_uid=1000} ---
strace: Process 3928188 attached
strace: Process 3928189 attached
[pid 3928184] openat(AT_FDCWD, "/home/snaipe/.local/etc/go/env", O_RDONLY|O_CLOEXEC) = 3
[pid 3928184] openat(AT_FDCWD, "/tmp/go/go.env", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
go: github.com/xdg/scram@v1.0.5: GOPROXY list is not the empty string, but contains no entries
[pid 3928189] +++ exited with 1 +++
[pid 3928188] +++ exited with 1 +++
[pid 3928187] +++ exited with 1 +++
[pid 3928186] +++ exited with 1 +++
[pid 3928185] +++ exited with 1 +++
+++ exited with 1 +++

And sure enough, as you suspected, GOROOT seems wrong:

$ ./bin/go env GOROOT
/tmp/go

(it should have been /tmp/go/out)

Is GOROOT_FINAL no longer respected for building Go?

[Snaipe]

Nevermind, the problem was our build toolchain; I think GOROOT was incorrectly still set to the source directory. Let me investigate a bit to make sure, but it's likely not a problem with Go.

[bcmills]

Is GOROOT_FINAL no longer respected for building Go?

Ah, see #61921.

[Snaipe]

After more investigation, the "reproducer" I had wasn't accurately representing what was happening on our end. It turns out our build instructions had a bit of a footgun that caused the go.env file to not get copied. I can confirm that things are now working as expected.

Thanks for the pointers about go.env, I don't think I'd have been able to figure what was wrong that quickly otherwise.