runtime: fatal error: unknown caller pc

[howardjohn]

What version of Go are you using (go version)?

$ go version
go version go1.15.6 linux/amd64

Does this issue reproduce with the latest release?

Not sure, we don't control this (oss-fuzz)

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/root/.go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/root/.go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build470550485=/tmp/go-build -gno-record-gcc-switches"

What did you do?

runtime: unexpected return pc for runtime.chanrecv called from 0x0
  | stack: frame={sp:0x10c00007a6d0, fp:0x10c00007a760} stack=[0x10c00007a000,0x10c00007a800)
  | 000010c00007a5d0:  00007f0bf6f523b0  0000000000000008
  | 000010c00007a5e0:  0000000000000008  00000000005860cc <runtime.wbBufFlush+108>
  | 000010c00007a5f0:  000010c00007a600  0000000000243000
  | 000010c00007a600:  00000000005b7f20 <runtime.wbBufFlush.func1+0>  00000000005860cc <runtime.wbBufFlush+108>
  | 000010c00007a610:  000010c00007a620  000010c00007a628
  | 000010c00007a620:  00000000005b7f20 <runtime.wbBufFlush.func1+0>  000010c000001d08
  | 000010c00007a630:  000010c00007a6b8  000010c000001c80
  | 000010c00007a640:  000010c0004e1c00  000010c00007a6c0
  | 000010c00007a650:  00000000005c07e7 <runtime.gcWriteBarrierBX+7>  000000000058e896 <runtime.acquireSudog+630>
  | 000010c00007a660:  000010c000001c80  000010c000001d08
  | 000010c00007a670:  000010c00007a6b8  000010c000001db8
  | 000010c00007a680:  000010c00009a000  000010c000001c80
  | 000010c00007a690:  000010c0004e1c00  000010c00007a6c0
  | 000010c00007a6a0:  000010c0004e0000  000000000058e546 <runtime.gopark+230>
  | 000010c00007a6b0:  0000000002aa37e0  000010c00006a000
  | 000010c00007a6c0:  000010c00007a750  000000000055b8cf <runtime.chanrecv+879>
  | 000010c00007a6d0: <0000000002aa35e0  000010c00009b378
  | 000010c00007a6e0:  000000000000170e  0000000000000002
  | 000010c00007a6f0:  0000000000560ee5 <runtime.libfuzzerTraceConstCmp4+69>  0000000000000000
  | 000010c00007a700:  0000000000000000  000010c00009a000
  | 000010c00007a710:  000010c00007a750  000010c00009b378
  | 000010c00007a720:  0000000000034215  000090c0005ee000
  | 000010c00007a730:  000010c00007ad20  00007f0bfa4fe390
  | 000010c00007a740:  0000000000000007  0000000000000000
  | 000010c00007a750:  00007f0bf6f89000 !0000000000000000
  | 000010c00007a760: >0000000000008000  00007f0bf65fed90
  | 000010c00007a770:  00007ffdee1aa0b8  0000000000000000
  | 000010c00007a780:  00000000008c36b8 <k8s.io/klog.(*loggingT).header+120>  0000000000000007
  | 000010c00007a790:  0000000000000000  000090c0005ee000
  | 000010c00007a7a0:  000090c0005f9fff  00007ffdee1aa080
  | 000010c00007a7b0:  0000000009c83ee1  000010c00007ad20
  | 000010c00007a7c0:  0000000000034215  0000000000000000
  | 000010c00007a7d0:  00007f0bcadd86d8  0000000000000000
  | 000010c00007a7e0:  000010c00007ad00  0000000000583a8e <runtime.mProf_FlushLocked+110>
  | 000010c00007a7f0:  0000000000000246  002b000000000033
  | fatal error: unknown caller pc
  |  
  | runtime stack:
  | runtime.throw(0x211cee9, 0x11)
  | runtime/panic.go:1116 +0x74
  | runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0x10c000001c80, 0x0, 0x0, 0x7fffffff, 0x7f0bcba4ac80, 0x0, 0x0, ...)
  | runtime/traceback.go:273 +0x1aec
  | runtime.scanstack(0x10c000001c80, 0x10c00006b698)
  | runtime/mgcmark.go:844 +0x17c
  | runtime.markroot.func1()
  | runtime/mgcmark.go:245 +0xc6
  | runtime.markroot(0x10c00006b698, 0x10c00000002b)
  | runtime/mgcmark.go:218 +0x310
  | runtime.gcDrain(0x10c00006b698, 0xb)
  | runtime/mgcmark.go:1109 +0x138
  | runtime.gcBgMarkWorker.func2()
  | runtime/mgc.go:1979 +0x190
  | runtime.systemstack(0x0)
  | runtime/asm_amd64.s:370 +0x63
  | runtime.mstart()
  | runtime/proc.go:1116
  |  
  | goroutine 7 [GC worker (idle)]:
  | runtime.systemstack_switch()
  | runtime/asm_amd64.s:330 fp=0x10c00007af60 sp=0x10c00007af58 pc=0x5be8c0
  | runtime.gcBgMarkWorker(0x10c00006a000)
  | runtime/mgc.go:1945 +0x1ce fp=0x10c00007afd8 sp=0x10c00007af60 pc=0x5726ae
  | runtime.goexit()
  | runtime/asm_amd64.s:1374 +0x1 fp=0x10c00007afe0 sp=0x10c00007afd8 pc=0x5c0681
  | created by runtime.gcBgMarkStartWorkers
  | runtime/mgc.go:1839 +0x79
  |  
  | goroutine 17 [runnable, locked to thread]:
  | runtime.goexit()
  | runtime/asm_amd64.s:1374 +0x1
  |  
  | goroutine 6 [chan receive (scan)]:
  | AddressSanitizer:DEADLYSIGNAL
  | =================================================================
  | ==974==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000118 (pc 0x0000005af87e bp 0x7f0bcba4a490 sp 0x7f0bcba4a1d0 T6)
  | ==974==The signal is caused by a READ memory access.
  | ==974==Hint: address points to the zero page.
  | SCARINESS: 10 (null-deref)
  | #0 0x5af87e in runtime.gentraceback runtime/traceback.go:261
  |  
  | AddressSanitizer can not provide additional info.
  | SUMMARY: AddressSanitizer: SEGV (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds_istio_3e2f8dbe6bc54615af91b66627c4bba3ef4cffdb/revisions/fuzz_parse_inputs+0x5af87e)
  | Thread T6 created by T1 here:
  | #0 0x50c37a in pthread_create /src/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:214:3
  | #1 0x554d90 in _cgo_try_pthread_create /_/runtime/cgo/gcc_libinit.c:100:9
  | #2 0x591e5c in runtime.newm runtime/proc.go:1822
  | #3 0x5924e8 in runtime.startm runtime/proc.go:1979
  | #4 0x5926d4 in runtime.handoffp runtime/proc.go:2007
  | #5 0x592b84 in runtime.stoplockedm runtime/proc.go:2081
  | #6 0x59496b in runtime.schedule runtime/proc.go:2617
  | #7 0x594f7d in runtime.goschedImpl runtime/proc.go:2866
  | #8 0x5951f5 in runtime.gopreempt_m runtime/proc.go:2894
  | #9 0x5a481b in runtime.newstack runtime/stack.go:1047
  | #10 0x5bea03 in runtime.morestack runtime/asm_amd64.s:449
  |  
  | Thread T1 created by T0 here:
  | #0 0x50c37a in pthread_create /src/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:214:3
  | #1 0x554ce0 in _cgo_try_pthread_create /_/runtime/cgo/gcc_libinit.c:100:9
  | #2 0x554ce0 in x_cgo_sys_thread_create /_/runtime/cgo/gcc_libinit.c:27:12
  | #3 0x20f10ec in __libc_csu_init
 

What did you expect to see?

No crash

What did you see instead?

Crash

[randall77]

That stack frame does look pretty trashed. Not sure how that would happen.
Can you reproduce? Any chance we could reproduce it ourselves?

Does it still fail with 1.16?

@mdempsky Might be a libfuzzer issue.

[mdempsky]

I won't rule out libfuzzer, but I don't immediately see anything that libfuzzer is doing wrong. The compiler instrumentation is fairly simple (just inserting some global counter increments, or calls to runtime helpers), and the runtime helper code (runtime/libfuzzer.go, runtime/libfuzzer_{amd,arm}64.s) doesn't appear to have diverged any from the race-detector code it was based on. (At least for Go 1.16; I do see a few dev.regabi-related CLs that only updated race_amd64.s, but not libfuzzer_amd64.s; /cc @cherrymui.)

It could maybe be related to the open-coded defer issues (#43882, etc; /cc @danscales)? Those failures sometimes manifested as "unknown caller pc" crashes. If so, it would be helpful to know if the issue still reproduces with Go 1.16, since that release includes fixes for the cases that are known to affect end user code (a5a5e2c).

[cherrymui]

(At least for Go 1.16; I do see a few dev.regabi-related CLs that only updated race_amd64.s, but not libfuzzer_amd64.s; /cc @cherrymui.)

That code should only have effect if GOEXPERIMENT=regabi is enabled (of course there could be bugs in the changes). I may need to update libfuzzer_amd64.s, at some point. Thanks for bringing this up.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)