builders: linux-ppc64-buildlet is failing saying "Stale runtime.a"

[ianlancetaylor]

There have been several recent failures on linux-ppc64-buildlet in which the runtime package fails saying

--- FAIL: TestCrashHandler (0.06s)
	crash_test.go:171: Stale runtime.a. Run 'go install runtime'.

The error is repeated many times.

For example:
https://build.golang.org/log/3c1890fd20e980d864f36c0fa3eabbcd83d2d1f0
https://build.golang.org/log/e6dd0a7c7399a1858c20e64efbf79b82adf31992

This error is basically impossible as the runtime package was just built and installed, and can not be stale. This suggests some sort of misconfiguration or disk error on the buildlet.

[ianlancetaylor]

Also happening on linux-ppc64le-buildlet:
https://build.golang.org/log/8e0531dbb1211817f46f1b88c765731c4a27e04c
https://build.golang.org/log/3238a5ec0d5faa6e831951fa698c898001852ed4

CC @laboger

[bradfitz]

I really don't want to debug this again on my own. I seem to recall fighting this with the Mac builders for a while to no avail. But the suspicion at the time was because the build was sharded and shipped around, I figured there was clock skew between the machines. But with ppc64 and ppc64le there's no sharding, so I have no clue what the problem is.

Can I just disable that stale check somehow?

[ianlancetaylor]

The test would be easy to disable. It's in checkStaleRuntime in runtime/crash_test.go.

I'll see if I can recreate it using gomote.

[laboger]

I am not familiar with what can cause it to decide that a package is stale. It might be helpful to have the error message in the test print out the contents of the 'out' variable and the staleReason.

[ianlancetaylor]

I have not been able to recreate this using gomote.

[gopherbot]

CL https://golang.org/cl/36918 mentions this issue.

[laboger]

StaleReason is what I was hoping to see.

I'm trying to see if I can reproduce it somewhere. There is a definite commit where it started happening recently.

[bradfitz]

Oh, yes, it should be StaleReason.

[laboger]

I'm not able to reproduce this so far. What is the exact command or script that is used to invoke these builds?

[gopherbot]

CL https://golang.org/cl/36934 mentions this issue.