runtime: unexpected return pc for runtime.sigpanic on darwin in x/sys/unix

[yangwenmai]

What version of Go are you using (go version)?

$ go version
go version go1.17 darwin/amd64

Does this issue reproduce with the latest release?

only go1.17
go1.16.6 can't reproduce this issue.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN="/Users/maiyang/develop/goworkspace/bin"
GOCACHE="/Users/maiyang/Library/Caches/go-build"
GOENV="/Users/maiyang/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/maiyang/develop/goworkspace/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/maiyang/develop/goworkspace"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/maiyang/develop/go1.17"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/maiyang/develop/go1.17/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/hy/qzqhs53939lg6b93mr419lnh0000gn/T/go-build1352877857=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

$ go mod tidy && go mod vendor
$ go build -o ./build/server ./cmd/xxxserver

What did you expect to see?

no error

What did you see instead?

runtime stack:
runtime: unexpected return pc for runtime.sigpanic called from 0x7fff20574cbe
stack: frame={sp:0x7ffeefbff218, fp:0x7ffeefbff268} stack=[0x7ffeefb802b8,0x7ffeefbff320)
0x00007ffeefbff118:  0x01007ffeefbff138  0x0000000000000004
0x00007ffeefbff128:  0x000000000000001f  0x00007fff20574cbe
0x00007ffeefbff138:  0x0b01dfacedebac1e  0x0000000000000001
0x00007ffeefbff148:  0x000000000403a011 <runtime.throw+0x0000000000000071>  0x00007ffeefbff1e8
0x00007ffeefbff158:  0x00000000063522e2  0x00007ffeefbff1a0
0x00007ffeefbff168:  0x000000000403a2c8 <runtime.fatalthrow.func1+0x0000000000000048>  0x00000000086ce240
0x00007ffeefbff178:  0x0000000000000001  0x0000000000000001
0x00007ffeefbff188:  0x00007ffeefbff1e8  0x000000000403a011 <runtime.throw+0x0000000000000071>
0x00007ffeefbff198:  0x00000000086ce240  0x00007ffeefbff1d8
0x00007ffeefbff1a8:  0x000000000403a250 <runtime.fatalthrow+0x0000000000000050>  0x00007ffeefbff1b8
0x00007ffeefbff1b8:  0x000000000403a280 <runtime.fatalthrow.func1+0x0000000000000000>  0x00000000086ce240
0x00007ffeefbff1c8:  0x000000000403a011 <runtime.throw+0x0000000000000071>  0x00007ffeefbff1e8
0x00007ffeefbff1d8:  0x00007ffeefbff208  0x000000000403a011 <runtime.throw+0x0000000000000071>
0x00007ffeefbff1e8:  0x00007ffeefbff1f0  0x000000000403a040 <runtime.throw.func1+0x0000000000000000>
0x00007ffeefbff1f8:  0x0000000006383dc6  0x000000000000002a
0x00007ffeefbff208:  0x00007ffeefbff258  0x0000000004050636 <runtime.sigpanic+0x0000000000000396>
0x00007ffeefbff218: <0x0000000006383dc6  0x00000000086ce240
0x00007ffeefbff228:  0x00007ffeefbff298  0x000000000402b186 <runtime.(*mheap).allocSpan+0x0000000000000546>
0x00007ffeefbff238:  0x000000c003aec000  0x0000000000002000
0x00007ffeefbff248:  0x000000c000000008  0x0000000000000000
0x00007ffeefbff258:  0x00007ffeefbff2a0 !0x00007fff20574cbe
0x00007ffeefbff268: >0x00007ffeefbff2a0  0x00000000083cc000
0x00007ffeefbff278:  0x0000000000000d78  0x0000000004612f45 <golang.org/x/sys/unix.libc_ioctl_trampoline+0x0000000000000005>
0x00007ffeefbff288:  0x000000000406faff <runtime.syscall+0x000000000000001f>  0x000000c0033173a8
0x00007ffeefbff298:  0x000000000406f46c <runtime.nanotime_trampoline+0x000000000000000c>  0x000000c003317378
0x00007ffeefbff2a8:  0x000000000406d930 <runtime.asmcgocall+0x0000000000000070>  0x0000000000000001
0x00007ffeefbff2b8:  0x0000000004012600 <runtime.(*linearAlloc).alloc+0x0000000000000000>  0x2600000000000020
0x00007ffeefbff2c8:  0x0000000000000000  0x0000000008704f60
0x00007ffeefbff2d8:  0x0000000000000cb0  0x000000c0000001a0
0x00007ffeefbff2e8:  0x000000000406ba49 <runtime.systemstack+0x0000000000000049>  0x0000000000000004
0x00007ffeefbff2f8:  0x0000000006b574b0  0x00000000086ce240
0x00007ffeefbff308:  0x00007ffeefbff350  0x000000000406b945 <runtime.mstart+0x0000000000000005>
0x00007ffeefbff318:  0x000000000406b8fd <runtime.rt0_go+0x000000000000013d>
runtime.throw({0x6383dc6, 0x86ce240})
	/Users/maiyang/develop/go1.17/src/runtime/panic.go:1198 +0x71
runtime: unexpected return pc for runtime.sigpanic called from 0x7fff20574cbe
stack: frame={sp:0x7ffeefbff218, fp:0x7ffeefbff268} stack=[0x7ffeefb802b8,0x7ffeefbff320)
0x00007ffeefbff118:  0x01007ffeefbff138  0x0000000000000004
0x00007ffeefbff128:  0x000000000000001f  0x00007fff20574cbe
0x00007ffeefbff138:  0x0b01dfacedebac1e  0x0000000000000001
0x00007ffeefbff148:  0x000000000403a011 <runtime.throw+0x0000000000000071>  0x00007ffeefbff1e8
0x00007ffeefbff158:  0x00000000063522e2  0x00007ffeefbff1a0
0x00007ffeefbff168:  0x000000000403a2c8 <runtime.fatalthrow.func1+0x0000000000000048>  0x00000000086ce240
0x00007ffeefbff178:  0x0000000000000001  0x0000000000000001
0x00007ffeefbff188:  0x00007ffeefbff1e8  0x000000000403a011 <runtime.throw+0x0000000000000071>
0x00007ffeefbff198:  0x00000000086ce240  0x00007ffeefbff1d8
0x00007ffeefbff1a8:  0x000000000403a250 <runtime.fatalthrow+0x0000000000000050>  0x00007ffeefbff1b8
0x00007ffeefbff1b8:  0x000000000403a280 <runtime.fatalthrow.func1+0x0000000000000000>  0x00000000086ce240
0x00007ffeefbff1c8:  0x000000000403a011 <runtime.throw+0x0000000000000071>  0x00007ffeefbff1e8
0x00007ffeefbff1d8:  0x00007ffeefbff208  0x000000000403a011 <runtime.throw+0x0000000000000071>
0x00007ffeefbff1e8:  0x00007ffeefbff1f0  0x000000000403a040 <runtime.throw.func1+0x0000000000000000>
0x00007ffeefbff1f8:  0x0000000006383dc6  0x000000000000002a
0x00007ffeefbff208:  0x00007ffeefbff258  0x0000000004050636 <runtime.sigpanic+0x0000000000000396>
0x00007ffeefbff218: <0x0000000006383dc6  0x00000000086ce240
0x00007ffeefbff228:  0x00007ffeefbff298  0x000000000402b186 <runtime.(*mheap).allocSpan+0x0000000000000546>
0x00007ffeefbff238:  0x000000c003aec000  0x0000000000002000
0x00007ffeefbff248:  0x000000c000000008  0x0000000000000000
0x00007ffeefbff258:  0x00007ffeefbff2a0 !0x00007fff20574cbe
0x00007ffeefbff268: >0x00007ffeefbff2a0  0x00000000083cc000
0x00007ffeefbff278:  0x0000000000000d78  0x0000000004612f45 <golang.org/x/sys/unix.libc_ioctl_trampoline+0x0000000000000005>
0x00007ffeefbff288:  0x000000000406faff <runtime.syscall+0x000000000000001f>  0x000000c0033173a8
0x00007ffeefbff298:  0x000000000406f46c <runtime.nanotime_trampoline+0x000000000000000c>  0x000000c003317378
0x00007ffeefbff2a8:  0x000000000406d930 <runtime.asmcgocall+0x0000000000000070>  0x0000000000000001
0x00007ffeefbff2b8:  0x0000000004012600 <runtime.(*linearAlloc).alloc+0x0000000000000000>  0x2600000000000020
0x00007ffeefbff2c8:  0x0000000000000000  0x0000000008704f60
0x00007ffeefbff2d8:  0x0000000000000cb0  0x000000c0000001a0
0x00007ffeefbff2e8:  0x000000000406ba49 <runtime.systemstack+0x0000000000000049>  0x0000000000000004
0x00007ffeefbff2f8:  0x0000000006b574b0  0x00000000086ce240
0x00007ffeefbff308:  0x00007ffeefbff350  0x000000000406b945 <runtime.mstart+0x0000000000000005>
0x00007ffeefbff318:  0x000000000406b8fd <runtime.rt0_go+0x000000000000013d>
runtime.sigpanic()
	/Users/maiyang/develop/go1.17/src/runtime/signal_unix.go:719 +0x396

goroutine 1 [syscall]:
syscall.syscall(0x4612f40, 0x2, 0x40487413, 0xc003317438)
	/Users/maiyang/develop/go1.17/src/runtime/sys_darwin.go:22 +0x3b fp=0xc0033173a8 sp=0xc003317388 pc=0x406a4fb
syscall.syscall(0x8, 0x401b52f, 0x0, 0x8)
	<autogenerated>:1 +0x26 fp=0xc0033173f0 sp=0xc0033173a8 pc=0x4070306
golang.org/x/sys/unix.ioctl(0xc003a1fce3, 0x1, 0xc0031b72c0)
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/vendor/golang.org/x/sys/unix/zsyscall_darwin_amd64.go:691 +0x39 fp=0xc003317420 sp=0xc0033173f0 pc=0x4612839
golang.org/x/sys/unix.IoctlGetTermios(...)
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/vendor/golang.org/x/sys/unix/ioctl.go:72
github.com/sirupsen/logrus.isTerminal(0xc003317a00)
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/vendor/github.com/sirupsen/logrus/terminal_check_bsd.go:11 +0x50 fp=0xc003317490 sp=0xc003317420 pc=0x467a070
github.com/sirupsen/logrus.checkIfTerminal({0x6b89880, 0xc000010020})
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/vendor/github.com/sirupsen/logrus/terminal_check_notappengine.go:13 +0x75 fp=0xc0033174a8 sp=0xc003317490 pc=0x467a115
github.com/sirupsen/logrus.(*TextFormatter).init(0xc003a4c660, 0x11b70660)
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/vendor/github.com/sirupsen/logrus/text_formatter.go:106 +0x34 fp=0xc003317528 sp=0xc0033174a8 pc=0x467a1f4
github.com/sirupsen/logrus.(*TextFormatter).Format.func1()
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/vendor/github.com/sirupsen/logrus/text_formatter.go:196 +0x25 fp=0xc003317548 sp=0xc003317528 pc=0x467b8a5
sync.(*Once).doSlow(0x6bb8e40, 0xc000699de0)
	/Users/maiyang/develop/go1.17/src/sync/once.go:68 +0xd2 fp=0xc0033175b0 sp=0xc003317548 pc=0x4078252
sync.(*Once).Do(...)
	/Users/maiyang/develop/go1.17/src/sync/once.go:59
github.com/sirupsen/logrus.(*TextFormatter).Format(0xc003a4c660, 0xc003197650)
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/vendor/github.com/sirupsen/logrus/text_formatter.go:196 +0xebe fp=0xc003317b20 sp=0xc0033175b0 pc=0x467b25e
github.com/sirupsen/logrus.(*Entry).write(0xc003197650)
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/vendor/github.com/sirupsen/logrus/entry.go:279 +0x56 fp=0xc003317bb0 sp=0xc003317b20 pc=0x4674a36
github.com/sirupsen/logrus.(*Entry).log(0xc003317c78, 0x4, {0xc00056c720, 0x26})
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/vendor/github.com/sirupsen/logrus/entry.go:251 +0x219 fp=0xc003317c20 sp=0xc003317bb0 pc=0x4674639
github.com/sirupsen/logrus.(*Entry).Log(0xc003197570, 0x4, {0xc003317c78, 0x1, 0x1})
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/vendor/github.com/sirupsen/logrus/entry.go:293 +0x4f fp=0xc003317c50 sp=0xc003317c20 pc=0x4674c8f
github.com/sirupsen/logrus.(*Entry).Logf(0xc003197570, 0x4, {0x6340112, 0x4011ee7}, {0xc003ad81c0, 0x5dece00, 0x1})
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/vendor/github.com/sirupsen/logrus/entry.go:338 +0x85 fp=0xc003317c98 sp=0xc003317c50 pc=0x46751e5
github.com/sirupsen/logrus.(*Entry).Infof(0xc00153b000, {0x6340112, 0x9}, {0xc003ad81c0, 0x0, 0x0})
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/vendor/github.com/sirupsen/logrus/entry.go:351 +0x37 fp=0xc003317ce0 sp=0xc003317c98 pc=0x46753b7
github.com/xxx/xxx/log.Infof(...)
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/log/exported.go:79
github.com/xxx/xxx/tracer.Init()
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/tracer/tracer.go:33 +0x153 fp=0xc003317e70 sp=0xc003317ce0 pc=0x48c1db3
main.main()
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/cmd/xxxserver/main.go:54 +0x4a fp=0xc003317f80 sp=0xc003317e70 pc=0x5c3684a
runtime.main()
	/Users/maiyang/develop/go1.17/src/runtime/proc.go:255 +0x227 fp=0xc003317fe0 sp=0xc003317f80 pc=0x403c6a7
runtime.goexit()
	/Users/maiyang/develop/go1.17/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc003317fe8 sp=0xc003317fe0 pc=0x406dc21

goroutine 52 [chan receive]:
github.com/xxx/xxx/pkg/track.newLogTracker.func1()
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/pkg/track/log.go:34 +0x11f
created by github.com/xxx/xxx/pkg/track.newLogTracker
	/Users/maiyang/develop/goworkspace/src/github.com/xxx/xxx/pkg/track/log.go:33 +0xdd

[mknyszek]

@yangwenmai Thanks for the issue. This isn't all that much to go on though, and it seems like you're not at liberty to share details about the code. How consistently does it crash? Is it possible for you to create a small reproducer?

[mknyszek]

CC @aclements @prattmic I'll take a look at this but just wanted y'all to be aware.

[ncw]

I'm seeing this too with the rclone CI build (I saw it with 1.17-rc2 but go 1.17 came out before I had time to report an issue!)

See: https://github.com/rclone/rclone/runs/3363791573?check_suite_focus=true

Both the macOS/amd64 and the macOS/arm64 builds have this

I don't have a mac to reproduce locally.

To repro with rclone code, on macos/amd64, check out rclone master then make quicktest

I think this might be error handling going wrong but I'm not sure.

[mknyszek]

@ncw Ah, thanks. So far it seems like this is macOS specific. I can try to reproduce.

[mknyszek]

Nice, that reproduced quite readily.

A closer look at the PC suggests this is failing somewhere inside a macOS library, actually. That's also backed up by the fact that all the stack traces I've seen so far are end at syscall.syscall (or something like it).

[mknyszek]

Oh, another common thread here is that all the failures I see so far travel through x/sys/unix, either through Umask or Ioctl.

Though, I believe I can rule out any direct issues with x/sys/unix. rclone depends on x/sys at 09eb48e85fd7 which is from 4 months ago. This is also correlated in both examples with 1.17's release, not an upgrade to x/sys/unix so it must be something deeper (or some way x/sys/unix interacts with the new 1.17 runtime and compiler).

[mknyszek]

@prattmic pointed out to me that he saw something similar before and sure enough: #45702.

The TL;DR is to use Go 1.17 with x/sys/unix, you need to update x/sys/unix to at least golang/sys@a76c4d0. @ncw @yangwenmai

[ncw]

Thanks @mknyszek I'm trying that out in the rclone CI now.

[ncw]

It worked! Thanks @mknyszek for solving the problem.

[yangwenmai]

It worked! Thanks @mknyszek for solving the problem.