runtime: widespread deadlocks involving subprocesses

[bcmills]

Ever since CL 204800, which may or may not be directly related, the linux-arm64-packet builder has been failing consistently.

The failures affect various tests; the only consistent pattern I can see among them is that the tests are blocked waiting for some child process to exit, generally in os.(*Process).blockUntilWaitable.

CC @ianlancetaylor @mknyszek

[bcmills]

linux-386: https://build.golang.org/log/ca4e429817cd924771991d08cb82d0540708b93b
linux-amd64: https://build.golang.org/log/07a9ef5a59aa3a726def455678d1b5e3ed61d4c7

[bcmills]

os.(*Process).wait on darwin is suspiciously similar.

darwin-amd64-10_14: https://build.golang.org/log/7917b648a72c359e8e63d8bbfefcb23d5c514516
darwin-amd64-10_12: https://build.golang.org/log/3eefa6817bbc7f42c3551955ad29075ee7773447

[bcmills]

And even windows-amd64-2016: https://build.golang.org/log/3f64d9f90c37c55dce95f426a8ec7fdcf7dde9ed

goroutine 41 [syscall, 5 minutes, locked to thread]:
syscall.Syscall(0x7ffe15cc42c0, 0x2, 0x224, 0xffffffff, 0x0, 0x0, 0x0, 0x0)
	C:/workdir/go/src/runtime/syscall_windows.go:188 +0xe9
syscall.WaitForSingleObject(0x224, 0xc0ffffffff, 0x0, 0x0, 0x5)
	C:/workdir/go/src/syscall/zsyscall_windows.go:738 +0x6b
os.(*Process).wait(0xc0002e7a70, 0x0, 0x0, 0x0)
	C:/workdir/go/src/os/exec_windows.go:18 +0x87
os.(*Process).Wait(...)
	C:/workdir/go/src/os/exec.go:125

[ianlancetaylor]

I don't see any particular reason to think this is a deadlock in (*Process).Wait itself. It seems rather more likely to be a problem causing the child process to deadlock or otherwise fail to exit in time.

[bcmills]

Fair enough. From the variety of tests I had thought that some of the subprocesses were non-Go programs, but now I don't think I actually had any evidence to support that assumption.

[gopherbot]

Change https://golang.org/cl/205558 mentions this issue: runtime: don't hold scheduler lock when calling timeSleepUntil