SIGINT crash during graceful shutdown with memory storage on macOS Mojave

[willbuckner]

When running cockroach start --store=type=mem,size=30% ..., pressing Ctrl+C/sending SIGINT results in a crash during shutdown. The log below contains both stdout and stderr output.

cockroach.log

Please describe the issue you observed, and any steps we can take to reproduce it:

To Reproduce

^C
*@spacecraft~⦕ tail cockroach.log
r11    0x213
r12    0x0
r13    0x3
r14    0x33
r15    0x0
rip    0x405ed1f
rflags 0x213
cs     0x2b
fs     0x0
gs     0x0
*@spacecraft~⦕

What did you do? Describe in your own words.

I started cockroach 2.0.5 using in-memory storage, and it crashes on SIGINT.

Expected behavior
Not crashing--graceful shutdown.

Additional data / screenshots
cockroach.log

Note that it crashes regardless of any SQL being run.

Environment:

*@spacecraft~/go/src/my_project⦕ cockroach version
Build Tag:    v2.0.5
Build Time:   2018/08/13 20:55:02
Distribution: CCL
Platform:     darwin amd64 (x86_64-apple-darwin17.7.0)
Go Version:   go1.10.3
C Compiler:   4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)
Build SHA-1:  3f2f0f5eea9b9d552e471eba8c37504a0595342f
Build Type:   development

*@spacecraft~⦕ uname -v
Darwin Kernel Version 18.0.0: Wed Aug 22 20:13:40 PDT 2018; root:xnu-4903.201.2~1/RELEASE_X86_64

Installed from brew install cockroach on macOS Mojave.

Additional context

[tbg]

Thanks @willbuckner. I've been running into something similar today, though for me it immediately crashes the process when it starts.

@benesch look what this is:

I181015 17:34:35.167704 574 server/updates.go:233  [n1] A new version is available: 2.0.6, details: https://www.cockroachlabs.com/docs/releases/v2.0.6.html
SIGTRAP: trace trap
PC=0x405ed1f m=0 sigcode=1

goroutine 0 [idle]:
invalid spdelta runtime.sigtramp 0x405ece0 0x405ed1f 0xea402 -1
runtime: unexpected return pc for runtime.sigtramp called from 0x500
stack: frame={sp:0xc420009a88, fp:0xc420009a8f} stack=[0x7ffeefb7edc0,0x7ffeefbfe240)

runtime.sigtramp(0xc420009ee000, 0xc420009f4800, 0x1e00, 0xc420009f4800, 0xc420009ab000, 0x0, 0x0, 0x0, 0x0, 0x300, ...)
	?:0 +0x3f

goroutine 53 [syscall]:
runtime.notetsleepg(0x7375460, 0xae3d6a8, 0x0)
	/usr/local/Cellar/go/1.10.3/libexec/src/runtime/lock_sema.go:280 +0x4b fp=0xc420489f60 sp=0xc420489f20 pc=0x4013b1b
runtime.timerproc(0x7375440)
	/usr/local/Cellar/go/1.10.3/libexec/src/runtime/time.go:261 +0x2e7 fp=0xc420489fd8 sp=0xc420489f60 pc=0x404ddc7
runtime.goexit()
	/usr/local/Cellar/go/1.10.3/libexec/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420489fe0 sp=0xc420489fd8 pc=0x405d9a1
created by runtime.(*timersBucket).addtimerLocked
	/usr/local/Cellar/go/1.10.3/libexec/src/runtime/time.go:160 +0x107

[tbg]

Ok, I think I'm closing in on a small repro (not for this exact crash, but for the one I get locally via gopsutil which I hope amounts to the same thing at the end). I've trimmed down pkg/cmd/cockroach to the following. If I comment out the ccl import, the crash goes away. Looking there now.

package main

// NB: the C import doesn't make a difference, I just have it here so that
// I don't accidentally drop out of cgo mode.

// static int foo = 0;
import "C"

import (
	"context"

	_ "github.com/cockroachdb/cockroach/pkg/ccl" // ccl init hooks
	// _ "github.com/cockroachdb/cockroach/pkg/ccl/cliccl" // cliccl init hooks
	// _ "github.com/cockroachdb/cockroach/pkg/ui/distccl" // ccl web UI init hook
	gopsnet "github.com/shirou/gopsutil/net"
)

func main() {
	_, _ = gopsnet.IOCountersWithContext(context.TODO(), true)
}

[benesch]

Amazing! This comes to mind:

cockroach/pkg/ccl/storageccl/engineccl/rocksdb.go

Line 37 in 342312a

engine.SetRocksDBOpenHook(C.DBOpenHookCCL)

[tbg]

Yeah, that's what I tracked it down to! Currently trimming the fat.

[tbg]

No conclusion here yet but I doubt @benesch can resist taking this down tomorrow.

The last I know is that the crash only happens when CryptoPP:UsesAESNI is referenced (not called - referenced).

Essentially the repro boils down to

func main() {
	_ = C.UsesAESNI

	cmd := exec.Command("echo")
	cmd.Run()
}

Very 🍿 to learn what this is caused by.

[tbg]

Branch is here: https://github.com/tschottdorf/cockroach/tree/delme-repro

make build && ./cockroach repros (not on the latest commit, that's an attempt at cutting out more code but I couldn't get it to build).