x/pkgsite: panic when running in direct_proxy mode

[catzkorn]

What version of Go are you using (go version)?

1.15.5

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/charlotte/Library/Caches/go-build"
GOENV="/Users/charlotte/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/charlotte/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/charlotte/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.15.5/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.15.5/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/charlotte/src/pkgsite/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/9g/lm8bfm3170d5zbpfxk68hgl80000gn/T/go-build289031567=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I started the frontend using

$ go run ./cmd/frontend -direct_proxy

Navigated to:
http://localhost:8080/github.com/golang-migrate/migrate/[email protected]

After waiting for it to load, I reloaded the page again and the server paniced.

panic Output

2020/11/30 11:49:53 Info: 0 /github.com/golang-migrate/migrate/[email protected] map[requestType:request start]
2020/11/30 11:49:53 http: panic serving 127.0.0.1:58802: runtime error: invalid memory address or nil pointer dereference
goroutine 2284 [running]:
net/http.(*conn).serve.func1(0xc002331c20)
        /usr/local/Cellar/go/1.15.5/libexec/src/net/http/server.go:1801 +0x147
panic(0x2171b80, 0x399a260)
        /usr/local/Cellar/go/1.15.5/libexec/src/runtime/panic.go:975 +0x47a
golang.org/x/pkgsite/internal/proxydatasource.(*DataSource).GetUnitMeta(0xc000cacf50, 0x3075a80, 0xc0012cff50, 0xc001dc5950, 0x24, 0xc001dc5920, 0x24, 0x2374b30, 0x6, 0x0, ...)
        /REDACTED/src/pkgsite/internal/proxydatasource/details.go:55 +0x2eb
golang.org/x/pkgsite/internal/frontend.latestMinorVersion(0x3075a80, 0xc0012cff50, 0x3076080, 0xc000cacf50, 0xc001dc5950, 0x24, 0xc001dc5920, 0x24, 0xc002c24b00, 0x7, ...)
        /REDACTED/src/pkgsite/internal/frontend/latest_version.go:57 +0x1f2
golang.org/x/pkgsite/internal/frontend.(*Server).GetLatestMinorVersion(0xc00161a210, 0x3075a80, 0xc0012cff50, 0xc001dc5950, 0x24, 0xc001dc5920, 0x24, 0xc002c24b00, 0x7, 0x7, ...)
        /REDACTED/src/pkgsite/internal/frontend/latest_version.go:37 +0xc5
golang.org/x/pkgsite/internal/middleware.LatestVersions.func1.1(0x306a400, 0xc0012dc2c0, 0xc001ead400)
        /REDACTED/src/pkgsite/internal/middleware/latestversion.go:52 +0x3c2
net/http.HandlerFunc.ServeHTTP(0xc001bc9140, 0x306a400, 0xc0012dc2c0, 0xc001ead400)
        /usr/local/Cellar/go/1.15.5/libexec/src/net/http/server.go:2042 +0x44
golang.org/x/pkgsite/internal/middleware.SecureHeaders.func1.1(0x306a400, 0xc0012dc2c0, 0xc001ead400)
        /REDACTED/src/pkgsite/internal/middleware/secureheaders.go:57 +0x370
net/http.HandlerFunc.ServeHTTP(0xc001bd4040, 0x306a400, 0xc0012dc2c0, 0xc001ead400)
        /usr/local/Cellar/go/1.15.5/libexec/src/net/http/server.go:2042 +0x44
golang.org/x/pkgsite/internal/middleware.GodocURL.func1.1(0x306a440, 0xc002210cc0, 0xc001ead400)
        /REDACTED/src/pkgsite/internal/middleware/godoc.go:65 +0x215
net/http.HandlerFunc.ServeHTTP(0xc001bd4060, 0x306a440, 0xc002210cc0, 0xc001ead400)
        /usr/local/Cellar/go/1.15.5/libexec/src/net/http/server.go:2042 +0x44
golang.org/x/pkgsite/internal/middleware.Quota.func1.1(0x306a440, 0xc002210cc0, 0xc001ead400)
        /REDACTED/src/pkgsite/internal/middleware/quota.go:95 +0x2ad
net/http.HandlerFunc.ServeHTTP(0xc001bb8240, 0x306a440, 0xc002210cc0, 0xc001ead400)
        /usr/local/Cellar/go/1.15.5/libexec/src/net/http/server.go:2042 +0x44
golang.org/x/pkgsite/internal/middleware.AcceptRequests.func1.1(0x306a440, 0xc002210cc0, 0xc001ead400)
        /REDACTED/src/pkgsite/internal/middleware/accept_requests.go:25 +0x16a
net/http.HandlerFunc.ServeHTTP(0xc001bc9170, 0x306a440, 0xc002210cc0, 0xc001ead400)
        /usr/local/Cellar/go/1.15.5/libexec/src/net/http/server.go:2042 +0x44
golang.org/x/pkgsite/internal/middleware.(*handler).ServeHTTP(0xc001bd4080, 0x306b980, 0xc001cd3a40, 0xc001ead300)
        /REDACTED/src/pkgsite/internal/middleware/requestlog.go:75 +0x435
net/http.serverHandler.ServeHTTP(0xc000d62540, 0x306b980, 0xc001cd3a40, 0xc001ead300)
        /usr/local/Cellar/go/1.15.5/libexec/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc002331c20, 0x30759c0, 0xc0012dc140)
        /usr/local/Cellar/go/1.15.5/libexec/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /usr/local/Cellar/go/1.15.5/libexec/src/net/http/server.go:2969 +0x36c

What did you expect to see?

I expected the page to reload.

What did you see instead?

The server paniced and the browser displayed a 500 internal error.

[julieqiu]

Thanks for filing the issue, @catzkorn! I wasn't able to reproduce the panic when visiting http://localhost:8080/github.com/golang-migrate/migrate/[email protected].

Here are the steps I took to start the frontend:

$ git fetch
$ git reset --hard origin/master
$ go run ./cmd/frontend -direct_proxy

This is the page that I saw at that URL:

[sudo-sturbia]

I was able to recreate the panic using http://localhost:8080/golang.org/x/pkgsite/ by interrupting the page while loading.

These are the steps i used:

1. Run the frontend server using $ go run ./cmd/frontend -direct_proxy
2. Open http://localhost:8080/golang.org/x/pkgsite/
3. Wait until a log message similar to Debug: memory after fetch of golang.org/x/[email protected]: 49M appears and then immediately interrupt loading.
4. Try to reload the page.

This seems to happen because modules are cached before packages are processed. I would like to submit a CL, if it's no problem!

[julieqiu]

@sudo-sturbia - please do, thank you!

[gopherbot]

Change https://golang.org/cl/274152 mentions this issue: internal/proxydatasource: delay caching of fetched modules