cmd/go: GOPROXY default can make 'get -u' lag upstream repository

[mvdan]

What version of Go are you using (go version)?

$ go version
go version devel +623d653db7 Sat Jun 29 13:17:15 2019 +0000 linux/amd64

Does this issue reproduce with the latest release?

No, because Go 1.12 doesn't have a GOPROXY set up by default.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN="/home/mvdan/go/bin"
GOCACHE="/home/mvdan/go/cache"
GOENV="/home/mvdan/.config/go/env"
GOEXE=""
GOFLAGS="-ldflags=-w"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mvdan/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org"
GOROOT="/home/mvdan/tip"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/mvdan/tip/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build558320792=/tmp/go-build -gno-record-gcc-switches"
GOROOT/bin/go version: go version devel +623d653db7 Sat Jun 29 13:17:15 2019 +0000 linux/amd64
GOROOT/bin/go tool compile -V: compile version devel +623d653db7 Sat Jun 29 13:17:15 2019 +0000
uname -sr: Linux 5.1.15-arch1-1-ARCH
/usr/lib/libc.so.6: GNU C Library (GNU libc) stable release version 2.29.
gdb --version: GNU gdb (GDB) 8.3

What did you do?

Module A and B live in separate repositories, and they are both v0. A depends on B.

I push a commit to B, and then run go get -u in A to update the dependency to that new master commit. Alternatively, tried go get B@master.

What did you expect to see?

A's go.mod updated to show B's latest pseudo-version with the master commit just pushed.

What did you see instead?

B's version staying at an older version, pushed hours or days before. Presumably because proxy.golang.org caches the version for @latest and @master.

This went unnoticed for a while and someone was scratching their head until they realised go get -u hadn't done what they were used to.

GOPROXY=direct go get -u had to be used to work around this issue in the end. The other option was to manually copy the commit hash to force the proxy to fetch that git ref, which is a bit cumbersome.

In the end, I worry that the new GOPROXY default in 1.13 is going to confuse people who are used to pushing to multiple repos that depend on each other. I understand why the proxy server can't fetch from git repeatedly every single time, but I wonder if there's something we can do to not silently break this.

/cc @bcmills @heschik @jayconrod

[myitcv]

@mvdan @rogpeppe and I were discussing this offline including various alternatives to the GOPROXY=direct "solution".

I think there is a change in workflow here that might affect a not-insignificant number of users. That's just a feeling, because I don't have specific numbers.

In 1.12, go get -u works in the scenario described.

In 1.13, are we saying GOPROXY=direct go get -u is only guaranteed way to deal with the outlined scenario?

[mewmew]

I think there is a change in workflow here that might affect a not-insignificant number of users. That's just a feeling, because I don't have specific numbers.

Not that this adds much to the discussion, but I'd be one of those users. Thanks for raising this issue.

[bcmills]

One option would be for the proxy not to serve branch names at all. It could serve a 404 or 410 for the .info file for the branch, and then the default fallback to direct would resolve that to the correct version.

We would lose some of the efficiency benefits of the proxy that way, but those benefits are most important for transitive dependencies — and transitive dependencies should always be pre-resolved to proper versions or pseudo-versions anyway.

CC @hyangah @katiehockman

[myitcv]

One option would be for the proxy not to serve branch names at all.

This would presumably include @latest so that also falls back to direct?

[hyangah]

@bcmills That is an interesting idea. Will the fallback to 'direct' mode be permanent? It seems like go commands are starting to depend on the fallback more and more.
What do you think about tags which don't look like semver or incompatible?
If we decide to take this path, maybe the go command even stops sending requests to proxy.golang.org for such requests to avoid extra network round trips.

@myitcv Is @latest version meant the one from the go command's perspective (the default when version is not specified), or the proxy endpoint /@latest for pseudo versions?
The go command currently picks up the latest version from the proxy's /@v/list endpoints and if there is no released version, the go command hits /@latest version to find any usable latest pseudo version.

If you meant the former, most go users will start fetching modules, packages, programs without version specified and the go command will treat them (and some of the dependencies) as @latest. As a result, the users will still face the problems proxy wanted to solve. (e.g. left-pad issues, vcs client requirements, ...).

--
Someone suggested a GONOPROXY trick for the repos I am actively working and testing on.

That will cause the module I am working on to be fetched directly (without any roundtrip to the proxy) but does verification against sum.golang.org.

We are also planning to improve data freshness by utilizing the new 1.13 feature (#32239) to detect the latest pseudo version.

[tbpg]

If we decide to take this path, maybe the go command even stops sending requests to proxy.golang.org for such requests to avoid extra network round trips.

I really like this idea. I'm slightly concerned about different versions of go returning different versions for @latest. The true latest is probably better than the proxy's latest, though.

[myitcv]

@hyangah

... for pseudo versions?

I'm not sure this issue is specific to pseudo versions is it? Because it occurs whenever a VCS releases a new version that the proxy has not yet seen, pseudo-version or otherwise.

The issue (at least the aspect I'm worried about) is that in in 1.13, we are saying users either have to:

• GOPROXY=direct go get -u
• or, as you suggested, set GONOPROXY

to avoid getting a stale latest version.

Both solutions work, but both represent a change to what is, I suspect, a common workflow.

I just want to be sure a) my understanding is correct and b) we are comfortable with pushing this change in workflow.

[hyangah]

@myitcv For non-pseudo versions, proxy.golang.org includes it in the /@v/list results as soon as it observes the new version. The module owners can teach proxy.golang.org about their new released version by simply running go mod download or go get with the newly released version. I hope that becomes part of the best practice for module release. The future go release tool may be helpful.

Once the semver-tagged module lands in the proxy.golang.org, it should be available to other users almost immediately (a couple of minutes in worst case). Otherwise, please file a bug.

[bcmills]

@hyangah

Will the fallback to 'direct' mode be permanent? It seems like go commands are starting to depend on the fallback more and more.

I'm not sure. I certainly expect it to be around for a while, at least.

What do you think about tags which don't look like semver or incompatible?

It seems fairly harmless to let those drop to direct too, since the go command will resolve them to canonical versions or pseudo-versions in the steady state.

If we decide to take this path, maybe the go command even stops sending requests to proxy.golang.org for such requests to avoid extra network round trips.

That would be fine for proxy.golang.org, but not for proxies in general: for private proxies we still want to give the proxy the opportunity to reject the module outright (for security, privacy, or other policy reasons).

[bcmills]

This would presumably include @latest so that also falls back to direct?

Probably not, for the reasons @hyangah mentioned. Plus, @latest is not as rare as, say, a branch-name or non-canonical tag: if someone runs go get -u they'll end up fetching the latest version of potentially a very large number of modules, so it's more important to get the efficiency boost from the proxy.

[hyangah]

@bcmills my personal preference is also for proxy.golang.org not to accept /@v/*.info queries with vcs branch names, non-semver tags, commit hashes or incompatible semver tags (whatever that is not resolved to the identical version). That is too vcs-centric. :-)

[myitcv]

The module owners can teach proxy.golang.org about their new released version by simply running go mod download or go get with the newly released version

Ah, now I think I understand your reference to #32239 (I couldn't understand why that field would be useful to the user, but it is to the proxy).

So just to clarify. Under your proposal:

• go get $package@$version against the proxy will, in the case of a "cache miss" on the proxy, cause the proxy to synchronously go get $package@$version against the VCS before returning to the caller
• if the version pulled back is latest according to VCS, the proxy can then update its version of @latest for that module

Is that about right?

[hyangah]

@myitcv Yes. right. For non-pseudo versions, proxy.golang.org "already" includes the newly found version in its results even without the fix for #32239.

[myitcv]

Thanks for confirming @hyangah. This is, if memory serves, very much along the lines of a conversation we had before, so great to hear it's going live!

So, assuming we are post the release of a proxy fix that includes #32239, the change in workflow here is: when a new (pseudo) version is released, someone/something (i.e. go release) needs to pull the version through the proxy.

That's definitely more understandable, but still worth flagging up in the release notes I suspect.

(On a related note: given that many (most?) people release new versions via one of the GitHub interfaces (e.g. "Draft new release") it might be worth kicking off a conversation with GitHub folks about integrating some sort of hook with the proxy?)

[FiloSottile]

I hit this multiple times in my personal infrastructure. I usually push changes to a repository, and then rebuild a Docker container with go get.

RUN go get github.com/FiloSottile/mostly-harmless/covfefe/cmd/webfefe@master

The proxy cache means that I can't deploy more than once every cache period.

My temporary solution is to set GONOPROXY for the main module, which still gets me most of the performance advantage of the proxy.

# https://github.com/golang/go/issues/32870
ENV GONOPROXY github.com/FiloSottile/mostly-harmless

[matloob]

We have a section on proxy.golang.org that documents that users should explicitly get a version after pushing it: it's under the Q&A for "I committed a new change (or released a new version) to a repository, why isn't it showing up when I run go get -u or go list -m --versions?".