cmd/internal/moddeps: test TestAllDependenciesVendored/golang.org/x/tools fails

[dmitshur]

What version of Go are you using (go version)?

$ go version                        
go version devel +6412750f32 Wed Mar 18 01:03:36 2020 +0000 darwin/amd64

Does this issue reproduce with the latest release?

No, the cmd/internal/moddeps package is new to master and doesn't exist in Go 1.14.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/dmitshur/Library/Caches/go-build"
GOENV="/Users/dmitshur/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/dmitshur/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/dmitshur/gotip"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/dmitshur/gotip/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/dmitshur/gotip/src/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/zb/5p8cwfhj29gf_m8vdy8ylmlr00jwcj/T/go-build665868502=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I ran all.bash or go test cmd/internal/moddeps on clean tip:

gotip $ git rev-parse HEAD
6412750f32224f6820f781d09b2092c5c358dddc
gotip $ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

What did you expect to see?

Test TestAllDependenciesVendored passing.

What did you see instead?

Test TestAllDependenciesVendored fails:

src $ ./all.bash
[...]
ok  	cmd/internal/dwarf	0.352s
ok  	cmd/internal/edit	0.711s
ok  	cmd/internal/goobj	2.582s
--- FAIL: TestAllDependenciesVendored (8.66s)
    --- FAIL: TestAllDependenciesVendored/golang.org/x/tools (0.17s)
        moddeps_test.go:126: /Users/dmitshur/gotip/bin/go list -mod=mod -m all: exit status 1
            go: updating go.mod: open /Users/dmitshur/gotip/pkg/obj/gopath/pkg/mod/golang.org/x/tools@v0.0.0-20190325161752-5a8dccf5b48a/go.mod: permission denied
FAIL
FAIL	cmd/internal/moddeps	8.928s
ok  	cmd/internal/obj	0.347s
ok  	cmd/internal/obj/arm64	0.903s
ok  	cmd/internal/obj/riscv	1.430s
[...]
ok  	cmd/pack	3.467s
ok  	cmd/trace	0.331s
ok  	cmd/vet	13.507s
FAIL
go tool dist: Failed: exit status 1

$ go test cmd/internal/moddeps 
--- FAIL: TestAllDependenciesVendored (1.22s)
    --- FAIL: TestAllDependenciesVendored/golang.org/x/tools (0.01s)
        moddeps_test.go:126: /Users/dmitshur/gotip/bin/go list -mod=mod -m all: exit status 1
            go: updating go.mod: open /Users/dmitshur/gotip/pkg/obj/gopath/pkg/mod/golang.org/x/tools@v0.0.0-20190325161752-5a8dccf5b48a/go.mod: permission denied
FAIL
FAIL	cmd/internal/moddeps	1.529s
FAIL

This may be related to #37749.

It fails 100% reproducibly for me. I don't know why it's not caught on builders. If it doesn't reproduce easily for others, let me know, and I'll provide more information about my environment.

/cc @bcmills @matloob @jayconrod

[bcmills]

What does sudo ls -al /Users/dmitshur/gotip/pkg/obj/gopath/pkg/mod/golang.org/x/tools@v0.0.0-20190325161752-5a8dccf5b48a/go.mod report?

[dmitshur]

$ sudo ls -al /Users/dmitshur/gotip/pkg/obj/gopath/pkg/mod/golang.org/x/tools@v0.0.0-20190325161752-5a8dccf5b48a/go.mod
-r--r--r--  1 dmitshur  primarygroup  87 Mar 25  2019 /Users/dmitshur/gotip/pkg/obj/gopath/pkg/mod/golang.org/x/tools@v0.0.0-20190325161752-5a8dccf5b48a/go.mod

[dmitshur]

I'll try making a new temporary directory, clone https://go.googlesource.com/go, and see if it reproduces there.

[bcmills]

Hang on, how did you end up with a module cache in GOROOT/pkg/obj/gopath?

pkg is .gitignored, which is why it isn't being caught on the builders: they're not doing whatever step you did that generated that directory.

go/.gitignore

Line 31 in 6197104

/pkg/

[dmitshur]

It doesn't reproduce with a fresh checkout:

ok  	cmd/internal/moddeps	8.331s

$HOME/gotip is my normal checkout of the main Go repo for development. I constantly switch branches and do various things in it, and rely on clean git status and running all.bash to bring me to a clean state. I can't easily find out what it was that caused that file to exist there.

I think it's reasonable to close this for now, but if it continues to re-occur, we should change .gitignore or tests to avoid this problem from continuing to happen.

[gopherbot]

Change https://golang.org/cl/223924 mentions this issue: cmd/internal/moddeps: skip GOROOT/pkg when locating modules

[bcmills]

Given that we know the mechanism, let's just change the test. Skipping pkg should make it marginally more efficient anyway.