runtime/trace: segfault in runtime.fpTracebackPCs during deferred call after recovering from panic #61766
Comments
gopherbot
added
the
compiler/runtime
|
Here's what I have so far. Some lldb output: The crash is from this line: Line 1014 in a4b6685 Register For what it's worth, the exact return address from Since the last good PC points to |
|
Change https://go.dev/cl/516157 mentions this issue: |
mknyszek
added
the
NeedsFix
|
My hunch was right. When a goroutine is recovering from a panic, the |
github-project-automation
bot
moved this from All-But-Submitted
to Done
in Go Compiler / Runtime
|
@gopherbot Please open up a backport issue for Go 1.21. This causes crashes that while rare (a fairly specific situation is required) are difficult to work around. The only option is turning off FP unwinding for the tracer, but that means a fairly severe regression in overheads, which can be a problem if you've already committed to tracing more frequently at scale. The fix is also small and fairly safe. (This issue also does not apply to Go 1.20.) |
|
Backport issue(s) opened: #62046 (for 1.21). Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases. |
|
Change https://go.dev/cl/519656 mentions this issue: |
|
Change https://go.dev/cl/523697 mentions this issue: |
|
Change https://go.dev/cl/523698 mentions this issue: |
Previously, the frame pointer wouldn't be restored at all, which could cause panics during frame pointer unwinding. As of CL 516157, the frame pointer is restored, but it's restored incorrectly on arm64: on arm64, the frame pointer points one word below SP, but here it's one below panic.fp which is the stack pointer of the caller's frame (nothing to do with the architectural bp). For #61766. Change-Id: I86504b85a4d741df5939b51c914d9e7c8d6edaad Reviewed-on: https://go-review.googlesource.com/c/go/+/523697 TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Cherry Mui <[email protected]> Run-TryBot: Michael Knyszek <[email protected]> LUCI-TryBot-Result: Go LUCI <[email protected]>
|
Change https://go.dev/cl/524315 mentions this issue: |
…recovering from panic When recovering from a panic, restore the caller's frame pointer before returning control to the caller. Otherwise, if the function proceeds to run more deferred calls before returning, the deferred functions will get invalid frame pointers pointing to an address lower in the stack. This can cause frame pointer unwinding to crash, such as if an execution trace event is recorded during the deferred call on architectures which support frame pointer unwinding. Original CL by Nick Ripley, includes fix from CL 523697, and includes a test update from CL 524315. This CL also deviates from the original fix by doing some extra computation to figure out the fp from the sp, since we don't have the fp immediately available to us in `recovery` on the Go 1.21 branch, and it would probably be complicated to plumb that through its caller. For #61766 Fixes #62046 Change-Id: I5a99ca4f909f6b6e209a330d595d1c99987d4359 Reviewed-on: https://go-review.googlesource.com/c/go/+/523698 LUCI-TryBot-Result: Go LUCI <[email protected]> Auto-Submit: Dmitri Shuralyov <[email protected]> Reviewed-by: Cherry Mui <[email protected]>
|
Change https://go.dev/cl/526256 mentions this issue: |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
No, it does not reproduce with Go 1.20.7.
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
Ran the following program, which records an execution trace event during a deferred call after recovering from a panic:
This is inspired by
TestCallersAfterRecovery, which failed for me while I was attempting to use frame pointer unwinding inruntime.Callers.What did you expect to see?
I expected the program to exit successfully.
What did you see instead?
The program crashed due to a segfault during frame pointer unwinding. It does not crash if I disable frame pointer unwinding for tracing by setting
GODEBUG=tracefpunwindoff=1.Here is an example output:
Crash output
cc @felixge @mknyszek
The text was updated successfully, but these errors were encountered: