runtime: "fatal error: signal_recv: inconsistent state" crash on ios/arm64

[stonelgh]

What version of Go are you using (go version)?

$ go version
go version go1.16.4 darwin/amd64

Does this issue reproduce with the latest release?

yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="off"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/stone/Library/Caches/go-build"
GOENV="/Users/stone/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/stone/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/stone/go"
GOPRIVATE=""
GOPROXY="https://goproxy.cn,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.4"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/kk/t_y76cl95cz5y1dtl9spqxt00000gn/T/go-build233946771=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Make a library for a iOS App with the latest gomobile. The program crashes a lot of times with the following output:

fatal error: signal_recv: inconsistent state

goroutine 20 [running]:
runtime.throw(0x10686cce1, 0x1f)
        /usr/local/go/src/runtime/panic.go:1117 +0x54 fp=0x13004df90 sp=0x13004df60 pc=0x105c804b4
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:161 +0x1d4 fp=0x13004dfb0 sp=0x13004df90 pc=0x105cb0684
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x20 fp=0x13004dfd0 sp=0x13004dfb0 pc=0x10603ab50
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x13004dfd0 sp=0x13004dfd0 pc=0x105cb3a94
created by os/signal.Notify.func1.1
        /usr/local/go/src/os/signal/signal.go:151 +0x44

...

We haven't found a way to consistently reproduce the failure. It usually crashes when users try to resume the app.

What did you expect to see?

The app is resumed and runs normally.

What did you see instead?

The app crashed.

[stonelgh]

I modified runtime/os_darwin.go to print the return value of read(sigNoteRead, unsafe.Pointer(&b), 1) called in function sigNoteSleep.
It outputs -4 when it crashes. According to runtime/stubs2.go, -4 is a negative errno value. Errno 4 means EINTR(interrupted system call).
That indicates the read system call is interrupted and sigNoteSleep returns prematurely. As a result, sig.state = sigReceiving is detected by runtime/sigqueue.go: func signal_recv() as an inconsistent state and throws.

Shall we check the return value of read and retry when it is interrupted?

[ianlancetaylor]

Yes.

[ianlancetaylor]

At least, yes, try that, and see if it fixes the problem. Thanks.

[stonelgh]

I've made a code change in sigNoteSleep. Will let you know the result.

func sigNoteSleep(*note) {
	for {
		entersyscallblock()
		var b byte
		n := read(sigNoteRead, unsafe.Pointer(&b), 1)
		exitsyscall()
		if n != -_EINTR {
			break
		}
	}
}

Updated at 2021-06-10 to remove the check to sig.state.

[jim3ma]

Same issue in macos/amd64

[dr2chase]

You have a reproducer for macos that does not require gomobile?

[stonelgh]

Not yet. Maybe @jim3ma have one?

[dr2chase]

Any news on the goodness of the proposed fix?

[stonelgh]

Due to the fact that we haven't found a way to consistently reproduce the issue, we haven't observed that the new logic have been triggered by now. We'll continue testing until we get enough evidences. Will let you know ASAP.

[jim3ma]

Emm, observed many times, but can not consistently reproduce the issue.

[cherrymui]

@jim3ma It doesn't have to consistently reproduce. As long as it fails somewhat frequently, that would be very helpful. Thanks!

[stonelgh]

@dr2chase @cherrymui @ianlancetaylor Confirmed: the proposed patch (with an update to drop the check to sig.state) can fix the mentioned crash.

[kavin808]

Same issue on our iOS App which causing crashes with SIGPIPE exception.
We are now adding signal(SIGPIPE, SIG_IGN) to ignore this exception type as a temporary solution.Hope this issue can be fixed soon.

[ianlancetaylor]

@stonelgh Thanks for testing and reporting back.

[gopherbot]

Change https://golang.org/cl/326778 mentions this issue: runtime: loop on EINTR in macOS sigNoteSleep