runtime: fatal error: inconsistent state in stackBarrier

[tlbtlbtlb]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8rc3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build054843203=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

This is running http://github.com/openai/go-vncdriver, which uses the CGO library to interface with Python. It happened 4 times in 9500 hours of runtime.

I also saw what looks like the same crash with go 1.6: #18718

What did you expect to see?

No crash

What did you see instead?

runtime: newstack sp=0xc4227c36c8 stack=[0xc4227bc000, 0xc4227c3fa0]
	morebuf={pc:0x7fe8de82f46f sp:0xc4227c36d0 lr:0x0}
	sched={pc:0x7fe8de817507 sp:0xc4227c36c8 lr:0x0 ctxt:0x0}
runtime: failed to unwind through stackBarrier at SP 0xc4227c36d0; [ @@@ ==>]
fatal error: inconsistent state in stackBarrier

runtime stack:
runtime.throw(0x7fe8de92dae5, 0x22)
	/usr/local/go/src/runtime/panic.go:566 +0x97
runtime.gentraceback(0x7fe8de82f46f, 0xc4227c36d0, 0x0, 0xc42010a000, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0, ...)
	/usr/local/go/src/runtime/traceback.go:217 +0x1a31
runtime.traceback1(0x7fe8de82f46f, 0xc4227c36d0, 0x0, 0xc42010a000, 0x0)
	/usr/local/go/src/runtime/traceback.go:690 +0x123
runtime.traceback(0x7fe8de82f46f, 0xc4227c36d0, 0x0, 0xc42010a000)
	/usr/local/go/src/runtime/traceback.go:651 +0x54
runtime.newstack()
	/usr/local/go/src/runtime/stack.go:950 +0xb5c
runtime.morestack()
	/usr/local/go/src/runtime/asm_amd64.s:366 +0x74

goroutine 19 [syscall, locked to thread]:
runtime: failed to unwind through stackBarrier at SP 0xc4227c36d0; [ @@@ ==>]
fatal error: inconsistent state in stackBarrier
panic during panic

runtime stack:
runtime.startpanic_m()
	/usr/local/go/src/runtime/panic.go:623 +0x100
runtime.systemstack(0x7fe8debbfcd8)
	/usr/local/go/src/runtime/asm_amd64.s:314 +0xa1
runtime.startpanic()
	/usr/local/go/src/runtime/panic.go:544 +0x1e
runtime.throw(0x7fe8de92dae5, 0x22)
	/usr/local/go/src/runtime/panic.go:565 +0x8a
runtime.gentraceback(0x7fe8de82f46f, 0xc4227c36d0, 0x0, 0xc42010a000, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0, ...)
	/usr/local/go/src/runtime/traceback.go:217 +0x1a31
runtime.traceback1(0x7fe8de82f46f, 0xc4227c36d0, 0x0, 0xc42010a000, 0x0)
	/usr/local/go/src/runtime/traceback.go:690 +0x123
runtime.traceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc42010a000)
	/usr/local/go/src/runtime/traceback.go:651 +0x54
runtime.tracebackothers(0xc420764ea0)
	/usr/local/go/src/runtime/traceback.go:796 +0x2f0
runtime.dopanic_m(0xc420764ea0, 0x7fe8de803007, 0x7fe714bc45e0)
	/usr/local/go/src/runtime/panic.go:669 +0x2a6
runtime.dopanic.func1()
	/usr/local/go/src/runtime/panic.go:553 +0x3e
runtime.systemstack(0x7fe714bc45b0)
	/usr/local/go/src/runtime/asm_amd64.s:314 +0xa1
runtime.dopanic(0x0)
	/usr/local/go/src/runtime/panic.go:554 +0x7b
runtime.throw(0x7fe8de92dae5, 0x22)
	/usr/local/go/src/runtime/panic.go:566 +0x97
runtime.gentraceback(0x7fe8de82f46f, 0xc4227c36d0, 0x0, 0xc42010a000, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0, ...)
	/usr/local/go/src/runtime/traceback.go:217 +0x1a31
runtime.traceback1(0x7fe8de82f46f, 0xc4227c36d0, 0x0, 0xc42010a000, 0x0)
	/usr/local/go/src/runtime/traceback.go:690 +0x123
runtime.traceback(0x7fe8de82f46f, 0xc4227c36d0, 0x0, 0xc42010a000)
	/usr/local/go/src/runtime/traceback.go:651 +0x54
runtime.newstack()
	/usr/local/go/src/runtime/stack.go:950 +0xb5c
runtime.morestack()
	/usr/local/go/src/runtime/asm_amd64.s:366 +0x74

[ianlancetaylor]

CC @aclements

[aclements]

@tlbtlbtlb, do you have any GODEBUG environment variable set? Go 1.8 doesn't use stack barrier by default unless enabled by GODEBUG, so I'm confused how this would even be possible.

[aclements]

@tlbtlbtlb, it appears your binary was actually compiled with Go 1.7.5, not Go 1.8rc3. That's the only version where the line numbers in your traceback match up with the code (specifically the combination of stack.go:950 calling traceback and traceback.go:651 calling traceback1).

[tlbtlbtlb]

@aclements You're right, I'm sorry, our docker build was getting one version on one system and another version on another system where I copied the go version from. I can't find this crash on a 1.8rc3 image, so I'll close this.

[rsc]

No bug, no milestone.

[tlbtlbtlb]

I haven't seen this since I upgraded (correctly) to 1.8rc3.

[rsc]

Thanks for the confirmation Trevor.