runtime: fatal error: throwOnGCWork (and general builder instability)

[eliasnaur]

Seen on several builders:

linux/amd64: https://build.golang.org/log/2ce33c8fa9981d37f3d5a81e384285c0be6df37a
android/arm: https://build.golang.org/log/34a763d60d9908f39856dd7380948081bef8fff9 (although it looks like the error is from the host running vet)

I noticed the errors because the Android builder has started to time out a lot:
https://build.golang.org/log/8b9d9127e958601e4a0a3aa71243463d814bd92e
https://build.golang.org/log/8aa5d8614243f1aea5354ba3598e66143de3249c
https://build.golang.org/log/80435e9f9573e9eec360dcb0a69772ca4ccf991a
https://build.golang.org/log/24e04fb21e1383cd002c3a575398f38cf15c7532
For this run I had to manually send a SIGQUIT signal to a vet process that hung for hours:
https://build.golang.org/log/994e945a2a0f4e4ac2efdbabf9c7fedcf402553e

Timeouts also happen on several builders unrelated to mobile:
linux/amd64: https://build.golang.org/log/f1d308e0e68a514f98c242fce22f83b790cc8304
linux/amd64: https://build.golang.org/log/bebe0b5a38b0218a6814e60769645435e0a48a90
darwin/amd64: https://build.golang.org/log/75959bdf3602be17a3247ce5f1735a323392794d

[ianlancetaylor]

CC @aclements

[aclements]

linux/amd64: https://build.golang.org/log/2ce33c8fa9981d37f3d5a81e384285c0be6df37a
android/arm: https://build.golang.org/log/34a763d60d9908f39856dd7380948081bef8fff9

Dup of #27993. (Definitely a release-blocker. I'm actively working on this one.)

https://build.golang.org/log/8b9d9127e958601e4a0a3aa71243463d814bd92e
https://build.golang.org/log/8aa5d8614243f1aea5354ba3598e66143de3249c
https://build.golang.org/log/80435e9f9573e9eec360dcb0a69772ca4ccf991a
https://build.golang.org/log/24e04fb21e1383cd002c3a575398f38cf15c7532

Dup of #25519. Only affects debug call injection, so not high-priority. This isn't technically Android-specific, though it seems to happen a lot there, so maybe we should just disable that test on Android.

https://build.golang.org/log/994e945a2a0f4e4ac2efdbabf9c7fedcf402553e

Stuck doing a syscall.Open?

[eliasnaur]

https://build.golang.org/log/994e945a2a0f4e4ac2efdbabf9c7fedcf402553e

Stuck doing a syscall.Open?

I believe the vet process used ~100% cpu when I sent it SIGQUIT. So an infinite loop somehow.

[gopherbot]

Change https://golang.org/cl/154112 mentions this issue: runtime: fix hangs in TestDebugCall*

[aclements]

Closing as a dup of #27993 and #25519.