Go checksum server responds with 410 Gone despite tag existing

[mholt]

What version of Go are you using (go version)?

$ go version
go version go1.15.2 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/matt/.cache/go-build"
GOENV="/home/matt/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/matt/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/matt/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/matt/Dev/caddyserver/caddy/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build115081725=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ go get github.com/caddyserver/caddy/[email protected]

What did you expect to see?

Successful pull:

go: finding github.com/caddyserver/caddy/v2 v2.2.0

What did you see instead?

verifying github.com/caddyserver/caddy/[email protected]/go.mod: github.com/caddyserver/caddy/[email protected]/go.mod: reading https://sum.golang.org/lookup/github.com/caddyserver/caddy/[email protected]: 410 Gone

According to seankhliao on Gophers Slack, right now proxy responds gone, fallbacks to direct, then it errors out because the sum server responds gone.

And according to @bcmills:

It seems to work fine using GOPRIVATE=github.com/caddyserver, so this certainly looks like something specific to the proxy, and my prior is that “something weird having to do with the proxy” is almost always the cache TTL.

But this is the first pull of that module and tag. (We've been using this release flow for about 6 months with no problems.)

Bryan also suggested issue #38065 might be related, but I am skeptical, since again, I have no clue as to why this would be cached already. Nor does it make sense to cache a non-entry. (Bug in the cache/proxy?)

This is blocking the v2.2.0 release of Caddy: https://github.com/caddyserver/caddy/runs/1161405190

Edit: After about 20 minutes it is working now. But this is still a bit unsettling because it blocks releases. What causes this, and how can this be avoided in the future?

[heschi]

This is the exact same situation you reported in #36183. I checked the logs again, and the first request we've seen for v2.2.0 came in today, not too long before you filed this issue. I think that the most likely explanation by far is that some part of the release process requested it early. I suppose it is possible that someone took the trouble to delay the release half an hour.

Do you have a specific question that was not answered in #36183? I already pointed to why it's necessary to have negative caching.

[mholt]

The thing is, this keeps happening, and although I'm pretty sure I didn't make a mistake last time, I'm 100% sure there wasn't a mistake (by me) this time, since our release process is automated and has been working fine, except occasionally the module cache has for some reason cached a non-entry. Maybe somebody set up a client to poll for the new tag or something. Whatever it is, it's getting in the way.

I believe that the ability for any random person on the Internet to break or delay releases -- especially security releases (not this one, fortunately) -- is something that needs to be fixed.

[heschi]

As I said, nobody has been polling for the new release: the first request came in shortly before you filed this issue. It's either a huge coincidence or a consequence of the release process.

proxy/sum.golang.org perform no authentication, so everyone is a "random person" to them, and that is not likely to change. Without that, there isn't going to be a good way to expire the negative cache entry. The TTL is less than an hour, which should be a tolerable, though admittedly unfortunate, amount of time to wait.

I can leave this open if you like but I don't think it's likely we'll change anything without evidence that it's causing widespread problems. I have seen almost none in the ~year since you filed the last issue.

@hyangah @katiehockman

[mholt]

It's either a huge coincidence or a consequence of the release process.

The release process is triggered by a push of the tag, so it's not a consequence of it. It's the other way around.

it's likely we'll change anything without evidence

Well. That is unfortunate (and disappointing, for something relied upon so heavily).

[leaanthony]

I'm with Matt on this. Nothing preventing a bad actor just scripting a get for a whole range of unpublished versions every 45 mins to prevent that module ever being downloaded by anyone using the public proxy. Why cache misses?

[cespare]

@leaanthony it's important for performance. #34370 (comment)

[heschi]

@leaanthony How would someone prevent a module from ever being downloaded? They can "poison" the cache as often as they want, the first time the cache expires and the module exists it will be downloaded.

[leaanthony]

You're entirely right, it's only a problem for the duration of the TTL.