runtime: unexpected return pc for runtime.gopark called from 0x4

[neo-hu]

What version of Go are you using (go version)?

$ go version
go version go1.16.2

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

crash report

Mar 16 19:54:41 netpm bgpd: runtime: unexpected return pc for runtime.gopark called from 0x4
Mar 16 19:54:41 netpm bgpd: stack: frame={sp:0xc00a686dc0, fp:0xc00a686de0} stack=[0xc00a686800,0xc00a687000)
Mar 16 19:54:41 netpm bgpd: 000000c00a686cc0: 000000000179a560 0000000000485425 
Mar 16 19:54:41 netpm bgpd: 000000c00a686cd0: 0000000000000004 01000000004432ca
Mar 16 19:54:41 netpm bgpd: 000000c00a686ce0: 00007f6e341365a0 0000000000485425 
Mar 16 19:54:41 netpm bgpd: 000000c00a686cf0: 0000000000000004 00007f6e65a8e3c8
Mar 16 19:54:41 netpm bgpd: 000000c00a686d00: 0000000000000030 000000c00510ed30
Mar 16 19:54:41 netpm bgpd: 000000c00a686d10: 000000000048505c  00000000017347c0
Mar 16 19:54:41 netpm bgpd: 000000c00a686d20: 000000c000404200 0000000000000004
Mar 16 19:54:41 netpm bgpd: 000000c00a686d30: 000000c00510ed58 00000000004e9431 
Mar 16 19:54:41 netpm bgpd: 000000c00a686d40: 00000000017347c0 0000000000ecb6c0
Mar 16 19:54:41 netpm bgpd: 000000c00a686d50: 000000c0096fa0d0 000000c00510edb0
Mar 16 19:54:41 netpm bgpd: 000000c00a686d60: 00000000004e998b  000000c0096fa0d0
Mar 16 19:54:41 netpm bgpd: 000000c00a686d70: 000000c008a9c040 0000000000000021
Mar 16 19:54:41 netpm bgpd: 000000c00a686d80: 000000c00b0c21b0 00000000004f7e99 
Mar 16 19:54:41 netpm bgpd: 000000c00a686d90: 000000c000498135 0000000000ee85b3
Mar 16 19:54:41 netpm bgpd: 000000c00a686da0: 0000000000000004 0000000000000004
Mar 16 19:54:41 netpm bgpd: 000000c00a686db0: 000000c00510ee18 0000000000538353 
Mar 16 19:54:41 netpm bgpd: 000000c00a686dc0: <000000c0013ee090 0000000000ee85b3
Mar 16 19:54:41 netpm bgpd: 000000c00a686dd0: 0000000000000004 !0000000000000004
Mar 16 19:54:41 netpm bgpd: 000000c00a686de0: >0000000000000000 0000000000000000
Mar 16 19:54:41 netpm bgpd: 000000c00a686df0: 00000000004f7e99  0000000000000004
Mar 16 19:54:41 netpm bgpd: 000000c00a686e00: 0000000000ee85b3 0000000000000005
Mar 16 19:54:41 netpm bgpd: 000000c00a686e10: 0000000000000005 000000c00510ee58
Mar 16 19:54:41 netpm bgpd: 000000c00a686e20: 00000000005381f1  000000c000147270
Mar 16 19:54:41 netpm bgpd: 000000c00a686e30: 000000c0013ee090 0000000000d465c0
Mar 16 19:54:41 netpm bgpd: 000000c00a686e40: 0000000000ff5550 0000000000000000
Mar 16 19:54:41 netpm bgpd: 000000c00a686e50: 0000000000000000 000000c00510f3d8
Mar 16 19:54:41 netpm bgpd: 000000c00a686e60: 0000000000536466  000000c0013ee090
Mar 16 19:54:41 netpm bgpd: 000000c00a686e70: 000000c0013ee00a 0000000000000000
Mar 16 19:54:41 netpm bgpd: 000000c00a686e80: 0000000000000000 0000000000d465c0
Mar 16 19:54:41 netpm bgpd: 000000c00a686e90: 0000000000ff5550 0000000000000019
Mar 16 19:54:41 netpm bgpd: 000000c00a686ea0: 00000000004cafa5  000000c00052e1f4
Mar 16 19:54:41 netpm bgpd: 000000c00a686eb0: 0000000000000001 0000000000000000
Mar 16 19:54:41 netpm bgpd: 000000c00a686ec0: 00000000000000a8 0000000000000008
Mar 16 19:54:41 netpm bgpd: 000000c00a686ed0: 0000000000000000 0000000000000000
Mar 16 19:54:41 netpm bgpd: fatal error: unknown caller pc
Mar 16 19:54:41 netpm bgpd: runtime: pointer 0xc020a8a640 to unallocated span span.base()=0xc0082e4000 span.limit=0xc0082e5fe0 span.state=0
Mar 16 19:54:41 netpm bgpd: runtime: found in object at *(0xc00a67eeb0+0x8)
Mar 16 19:54:41 netpm bgpd: object=0xc00a67eeb0 s.base()=0xc00a67e000 s.limit=0xc00a67ffe0 s.spanclass=14 s.elemsize=80 s.state=mSpanInUse
Mar 16 19:54:41 netpm bgpd: *(object+0) = 0x0
Mar 16 19:54:41 netpm bgpd: *(object+8) = 0xc020a8a640 <==
Mar 16 19:54:41 netpm bgpd: *(object+16) = 0x0
Mar 16 19:54:41 netpm bgpd: *(object+24) = 0x0
Mar 16 19:54:41 netpm bgpd: *(object+32) = 0x0
Mar 16 19:54:41 netpm bgpd: *(object+40) = 0x0
Mar 16 19:54:41 netpm bgpd: *(object+48) = 0x0
Mar 16 19:54:41 netpm bgpd: *(object+56) = 0x0
Mar 16 19:54:41 netpm bgpd: *(object+64) = 0x10000005b5d79df
Mar 16 19:54:41 netpm bgpd: *(object+72) = 0x0
Mar 16 19:54:41 netpm bgpd: fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

What did you do?

We don't use many "unsafe" libraries and CGO_ENABLED=0

[heschi]

cc @golang/release

[dmitshur]

Thanks for the report. Is this an issue you're able to reproduce? Since Go 1.16.2 is no longer supported, it would help to know if this also happens with Go 1.18, 1.17.x or tip, and a sequence of steps to reproduce the problem.

Heschi, was this meant to be a CC to @​golang/runtime?

[heschi]

Ugh. @golang/runtime .

[dkropachev]

I got this issue on 1.18.7, which has been running on docker 1.18 image

No steps to reproduce, running into it under high load time to time.

panic.log

runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc2d2581138, fp:0xc2d2581158} stack=[0xc2d257a000,0xc2d2582000)
0x000000c2d2581038:  0x0000000000000000  0x0000000000000000
0x000000c2d2581048:  0x0000000000000000  0x0000000000000000
0x000000c2d2581058:  0x0000000000000000  0x0000000000000000
0x000000c2d2581068:  0x0000000000000000  0x0000000000000000
0x000000c2d2581078:  0x0000000000000000  0x0000000000000000
0x000000c2d2581088:  0x0000000000000000  0x0000000000000000
0x000000c2d2581098:  0x0000000000000000  0x0000000000000000
0x000000c2d25810a8:  0x0000000000000000  0x0000000000000000
0x000000c2d25810b8:  0x0000000000000000  0x0000000000000000
0x000000c2d25810c8:  0x0000000000000000  0x0000000000000000
0x000000c2d25810d8:  0x0000000000000000  0x0000000000000000
0x000000c2d25810e8:  0x0000000000000000  0x0000000000000000
0x000000c2d25810f8:  0x0000000000000000  0x0000000000000000
0x000000c2d2581108:  0x0000000000000000  0x0000000000000000
0x000000c2d2581118:  0x0000000000000000  0x0000000000000000
0x000000c2d2581128:  0x0000000000000000  0x0000000000000000
0x000000c2d2581138: <0x0000000000000000  0x0000000000000000
0x000000c2d2581148:  0x0000000000000000 !0x0000000000000000
0x000000c2d2581158: >0x0000000000000000  0x0000000000000000
0x000000c2d2581168:  0x0000000000000000  0x0000000000000000
0x000000c2d2581178:  0x0000000000000000  0x0000000000000000
0x000000c2d2581188:  0x0000000000000000  0x0000000000000000
0x000000c2d2581198:  0x0000000000000000  0x0000000000000000
0x000000c2d25811a8:  0x0000000000000000  0x0000000000000000
0x000000c2d25811b8:  0x0000000000000000  0x0000000000000000
0x000000c2d25811c8:  0x0000000000000000  0x0000000000000000
0x000000c2d25811d8:  0x0000000000000000  0x0000000000000000
0x000000c2d25811e8:  0x0000000000000000  0x0000000000000000
0x000000c2d25811f8:  0x0000000000000000  0x0000000000000000
0x000000c2d2581208:  0x0000000000000000  0x0000000000000000
0x000000c2d2581218:  0x0000000000000000  0x0000000000000000
0x000000c2d2581228:  0x0000000000000000  0x0000000000000000
0x000000c2d2581238:  0x0000000000000000  0x0000000000000000
0x000000c2d2581248:  0x0000000000000000  0x0000000000000000
fatal error: unknown caller pc

runtime stack:
runtime.throw({0x755d50?, 0x970940?})
        /usr/local/go/src/runtime/panic.go:992 +0x71
runtime.gentraceback(0xec8?, 0x7fd543593978?, 0x0?, 0x0?, 0x0, 0x0, 0x7fffffff, 0x781068, 0x6cdaca?, 0x0)
        /usr/local/go/src/runtime/traceback.go:269 +0x1c2a
runtime.copystack(0xc0005bf380, 0x7fd543593c70?)
        /usr/local/go/src/runtime/stack.go:930 +0x2f5
runtime.shrinkstack(0xc0005bf380)
        /usr/local/go/src/runtime/stack.go:1212 +0x126
runtime.scanstack(0xc0005bf380, 0xc000023238)
        /usr/local/go/src/runtime/mgcmark.go:754 +0xc7
runtime.markroot.func1()
        /usr/local/go/src/runtime/mgcmark.go:241 +0xc5
runtime.markroot(0xc000023238, 0x1814, 0x1)
        /usr/local/go/src/runtime/mgcmark.go:214 +0x1a5
runtime.gcDrain(0xc000023238, 0x2)
        /usr/local/go/src/runtime/mgcmark.go:1047 +0x39f
runtime.gcBgMarkWorker.func2()
        /usr/local/go/src/runtime/mgc.go:1291 +0x154
runtime.systemstack()
        /usr/local/go/src/runtime/asm_amd64.s:469 +0x49

goroutine 612 [GC worker (idle)]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:436 fp=0xc0003de758 sp=0xc0003de750 pc=0x4621e0
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1263 +0x1b1 fp=0xc0003de7e0 sp=0xc0003de758 pc=0x41c571
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0003de7e8 sp=0xc0003de7e0 pc=0x464401
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1131 +0x25

goroutine 1 [chan receive]:
github.com/scylladb/scylla-bench/pkg/results.(*TestResults).GetResultsFromThreadsAndMerge(0xc00038b8c0)
        /go/scylla-bench-0.1.15/pkg/results/thread_results.go:45 +0x105
github.com/scylladb/scylla-bench/pkg/results.(*TestResults).GetTotalResults(0xc00038b8c0)
        /go/scylla-bench-0.1.15/pkg/results/thread_results.go:82 +0xcc
main.main()
        /go/scylla-bench-0.1.15/main.go:631 +0x39bd