runtime: "goroutine stack exceeds 250000000-byte limit" on linux-arm

[bcmills]

We're seeing stack overflows this week in various tests on the linux-arm builder.

This may be related to #35349, but the stack traces on linux-arm are more diverse.

CC @ianlancetaylor @aclements @mknyszek @cherrymui

2019-11-08T19:24:30-e6c12c3/linux-arm
2019-11-07T19:20:35-ceca99b/linux-arm
2019-11-07T18:39:03-05aa4a7/linux-arm
2019-11-07T16:13:31-0bf2eb5/linux-arm
2019-11-05T20:56:05-81559af/linux-arm
2019-11-05T17:19:16-1b3a1db/linux-arm

[ianlancetaylor]

These stacks are not large at all, so the problem is not actual stack overflow, but a false report of stack overflow.

[cherrymui]

I can imagine a possibility: if there are both a synchronous preemption request (by clobbering the stack guard) and an asynchronous one (by signal), and the goroutine in a function prologue first sees the clobbered stack guard, so it will call morestack. If the signal lands after the CMP instruction but before the call to morestack, it will be asynchronously preempted, enter the scheduler. When it is resumed, the scheduler clears the preemption request, unclobbers the stack guard. But the resumed goroutine will still call morestack (as it has passed the CMP instruction). morestack will, as there is no preemption request, double the stack unnecessarily. If this happens multiple times, the stack may grow too big, although only a small amount is actually used.

I let it print the current stack bounds in the stack-too-large error message, and the stack is indeed quite large, with only a small amount used:

runtime: goroutine stack exceeds 250000000-byte limit
	sp=0x1847dad8 stack=[0x10480000, 0x18480000]
fatal error: stack overflow

In theory this can happen on other platforms. Not sure why this is only seen on the ARM builder.

[cherrymui]

Maybe we want to disable async preemption in function prologue between the CMP instruction and the call to morestack? As it will call morestack, it will be preempted anyway.

[randall77]

@cherrymui That sounds like a good idea.

I think we might have to start at the load of the stack guard, as the CMP result is predestined at that point. But then maybe we need to only prevent async preemption if that loaded value is in fact the preempted guard.

Tricky.

[gopherbot]

Change https://golang.org/cl/207350 mentions this issue: cmd/internal/obj: mark split-stack prologue nonpreemptible

[gopherbot]

Change https://golang.org/cl/207351 mentions this issue: runtime: print more information on stack overflow

[gopherbot]

Change https://golang.org/cl/207349 mentions this issue: cmd/internal/obj, runtime: use register map to mark unsafe points

[ianlancetaylor]

See also #35784.

[aclements]

Should we also close #35784?

[cherrymui]

Yeah, I think we can close that.