Skip to content

gccgo: SIGSEGV during cgo call #23393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tamird opened this issue Jan 9, 2018 · 12 comments
Closed

gccgo: SIGSEGV during cgo call #23393

tamird opened this issue Jan 9, 2018 · 12 comments
Milestone

Comments

@tamird
Copy link
Contributor

tamird commented Jan 9, 2018

go version go1.9 gccgo (GCC) 8.0.0 20180108 (experimental) linux/amd64

Reproduced by making a ./cockroach quit RPC to a running CockroachDB server compiled with gccgo; the client exited normally, but the server crashed during shutdown.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=1 addr=8 pc=43850201]

goroutine 1290 [syscall, locked to thread]:
runtime.dopanic
        ../../../src/libgo/go/runtime/panic.go:892
runtime.throw
        ../../../src/libgo/go/runtime/panic.go:808
runtime.sigpanic
        ../../../src/libgo/go/runtime/signal_unix.go:311
runtime.sighandler
        ../../../src/libgo/go/runtime/signal_sighandler.go:59
runtime.sigtrampgo
        ../../../src/libgo/go/runtime/signal_unix.go:297
runtime.sigtramp
        ../../../src/libgo/runtime/go-signal.c:86

        :0

        :0

        :0
engine._Cfunc_DBUnlockFile
        /tmp/go-build/github.com/cockroachdb/cockroach/pkg/storage/engine/_obj/_cgo_gotypes.go:567
engine.$nested85
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:2355
engine.unlockFile
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:2355
engine.$nested86
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/temp_dir.go:58
github_com_cockroachdb_cockroach_pkg_util_stop.Close.N55_github_com_cockroachdb_cockroach_pkg_util_stop.CloserFn
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:93
github_com_cockroachdb_cockroach_pkg_util_stop.Stop.pN54_github_com_cockroachdb_cockroach_pkg_util_stop.Stopper
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:393
server.$nested3
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/server/admin.go:1206
runtime.kickoff
        ../../../src/libgo/go/runtime/proc.go:1122

        :0
created by github_com_cockroachdb_cockroach_pkg_server.Drain.pN55_github_com_cockroachdb_cockroach_pkg_server.adminServer
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/server/admin.go:1202 +1043

goroutine 1 [chan receive]:
cli.runStart
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:633
cli.$nested24
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/cli/error.go:37
cli.$nested27
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/cli/error.go:95
github_com_cockroachdb_cockroach_vendor_github_com_spf13_cobra.execute.pN70_github_com_cockroachdb_cockroach_vendor_github_com_spf13_cobra.Command
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:698
github_com_cockroachdb_cockroach_vendor_github_com_spf13_cobra.ExecuteC.pN70_github_com_cockroachdb_cockroach_vendor_github_com_spf13_cobra.Command
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:783
github_com_cockroachdb_cockroach_vendor_github_com_spf13_cobra.Execute.pN70_github_com_cockroachdb_cockroach_vendor_github_com_spf13_cobra.Command
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:736
github_com_cockroachdb_cockroach_pkg_cli.Run
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:153
github_com_cockroachdb_cockroach_pkg_cli.Main
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:51
main.main
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/main.go:27

goroutine 19 [syscall, 15 minutes]:
        goroutine in C code; stack unavailable
created by os_signal..import
        ../../../src/libgo/go/os/signal/signal_unix.go:26 +44

goroutine 3 [chan receive]:
github_com_cockroachdb_cockroach_pkg_util_log.flushDaemon.pN54_github_com_cockroachdb_cockroach_pkg_util_log.loggingT
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1043
created by github_com_cockroachdb_cockroach_pkg_util_log..import
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:570 +2519

goroutine 9 [select, 15 minutes, locked to thread]:

goroutine 15 [semacquire, 15 minutes]:
sync.runtime_notifyListWait
        ../../../src/libgo/go/runtime/sema.go:507
sync.Wait.pN9_sync.Cond
        ../../../src/libgo/go/sync/cond.go:56
github_com_cockroachdb_cockroach_pkg_storage_engine.syncLoop.pN59_github_com_cockroachdb_cockroach_pkg_storage_engine.RocksDB
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:625
created by github_com_cockroachdb_cockroach_pkg_storage_engine.open.pN59_github_com_cockroachdb_cockroach_pkg_storage_engine.RocksDB
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:612 +1923

goroutine 84 [semacquire]:
sync.runtime_notifyListWait
        ../../../src/libgo/go/runtime/sema.go:507
sync.Wait.pN9_sync.Cond
        ../../../src/libgo/go/sync/cond.go:56
github_com_cockroachdb_cockroach_pkg_storage_engine.syncLoop.pN59_github_com_cockroachdb_cockroach_pkg_storage_engine.RocksDB
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:625
created by github_com_cockroachdb_cockroach_pkg_storage_engine.open.pN59_github_com_cockroachdb_cockroach_pkg_storage_engine.RocksDB
        /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:612 +1923
@gopherbot gopherbot added this to the Gccgo milestone Jan 9, 2018
@ianlancetaylor
Copy link
Contributor

Can you show us a way to reproduce the problem locally?

@tamird
Copy link
Contributor Author

tamird commented Jan 26, 2018

@ianlancetaylor it's a bit tricky, I'm afraid, because CockroachDB requires gccgo 8+, and one of its dependencies (RocksDB) doesn't compile on GCC 8.

Here's one (annoying) way to do it:

go get -d github.com/cockroachdb/cockroach
cd $(go env GOPATH)/src/github.com/cockroachdb/cockroach
git remote add tamird github.com/tamird/cockroach && git fetch tamird && git checkout tamird/gccgo
# compile with a non-GCC8 toolchain first to build rocksdb
make build
# compile again with gccgo (rocksdb won't be recompiled)
make build GO=<path to gccgo's go>
# might need to set LD_LIBRARY_PATH
./cockroach start --insecure -s type=mem,size=1GiB
# wait about 10 seconds, some logs will scroll by
<ctrl-c>
<you will see a stack trace beginning with:
initiating graceful shutdown of server                                                                                                              
fatal error: unexpected signal during runtime execution                                                                                             
[signal SIGSEGV: segmentation violation code=1 addr=8 pc=55500911]>

@ianlancetaylor
Copy link
Contributor

Thanks for the detailed instructions. I've tried a couple of times, and I get this error from make build. Any suggestions? Thanks.

GOPATH set to /home/iant/gopath
Running make with -j8
build/node-run.sh -C ./pkg/ui yarn install
ERROR: [Errno 2] No such file or directory: 'install'
Makefile:318: recipe for target 'pkg/ui/yarn.installed' failed
make: *** [pkg/ui/yarn.installed] Error 1
make: *** Waiting for unfinished jobs....

@tamird
Copy link
Contributor Author

tamird commented Jan 30, 2018 via email

@ianlancetaylor
Copy link
Contributor

Hmmm, I installed a program named "yarn" using apt-get install cmdtest, but now it looks like it might be the wrong program. man yarn says "yarn - scenario testing of Unix command line tools". That doesn't sound like what you are describing. Is there a standard Ubuntu installation of the tool you are talking about?

@tamird
Copy link
Contributor Author

tamird commented Jan 30, 2018 via email

@thanm
Copy link
Contributor

thanm commented Jan 30, 2018

I am able to reproduce this on my work machine. I'll poke at it for a bit.

@thanm
Copy link
Contributor

thanm commented Jan 30, 2018

Call stack (partial) at crash from within GDB:

#0  0x000000000347d3bb in rocksdb::(anonymous namespace)::PosixEnv::UnlockFile(rocksdb::FileLock*) ()
#1  0x00000000032a76de in DBUnlockFile ()
#2  0x00000000022c2534 in _cgo_b1f425fc2dd9_Cfunc_DBUnlockFile (p0=<optimized out>) at cgo-gcc-prolog:509
#3  0x000000000228df99 in engine._Cfunc_DBUnlockFile (p0=0x0) at /tmp/go-build/b205/_cgo_gotypes.go:589
#4  0x00000000036044cd in __morestack () at ../../../gcc-trunk/libgcc/config/i386/morestack.S:546
#5  0x00000000022ab762 in engine.unlockFile..func1 (_cgo0=0x0) at /ssd2/go4.issue23393/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:2537
#6  0x00000000022ab6ab in engine.unlockFile (lock=0x0) at /ssd2/go4.issue23393/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:2537

At point of crash, we are here:

_ZN7rocksdb12_GLOBAL__N_18PosixEnv10UnlockFileEPNS_8FileLockE():
 347d3a0:	push   %rbp
 347d3a1:	mov    %rsp,%rbp
 347d3a4:	push   %r15
 347d3a6:	push   %r14
 347d3a8:	push   %r13
 347d3aa:	lea    0x10(%rdx),%r13
 347d3ae:	push   %r12
 347d3b0:	mov    %rdi,%r12
 347d3b3:	push   %rbx
 347d3b4:	mov    %rdx,%rbx
 347d3b7:	sub    $0x48,%rsp
 347d3bb:	mov    0x8(%rdx),%esi   <<== 'rdx' is zero
 347d3be:	xor    %edx,%edx
 347d3c0:	movl   $0x0,(%rdi)
 347d3c6:	movl   $0x0,0x4(%rdi)
 347d3cd:	movq   $0x0,0x8(%rdi)
 347d3d5:	mov    %r13,%rdi

Moving up the stack, it looks as though 'lock' is zero well before we enter the cgo code.

@tamird
Copy link
Contributor Author

tamird commented Jan 30, 2018

Reproduced as you did using gdb. Here's the a slightly larger stack:

#0  0x000000000341348f in rocksdb::(anonymous namespace)::PosixEnv::UnlockFile(rocksdb::FileLock*) ()
#1  0x00000000032529ae in DBUnlockFile ()
#2  0x00000000022aefa9 in engine._Cfunc_DBUnlockFile (p0=0x0) at /tmp/go-build/b220/_cgo_gotypes.go:589
#3  0x00000000035f4225 in __morestack () at ../../../src/libgcc/config/i386/morestack.S:546
#4  0x00000000022cc772 in engine.unlockFile..func1 (_cgo0=0x0)
    at /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:2537
#5  0x00000000022cc6bb in engine.unlockFile (lock=0x0)
    at /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:2537
#6  0x00000000022ccf0d in engine.CreateTempDir..func1 ()
    at /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/temp_dir.go:58
#7  0x0000000001f14c25 in github_com_cockroachdb_cockroach_pkg_util_stop.CloserFn.Close (pointer=0xc4210a8538)
    at /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:93
#8  0x0000000001f18bc8 in stop.r (s=0xc4200ca090, ctx=..., pr.5=false)
    at /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:424
#9  0x0000000001f178cb in github_com_cockroachdb_cockroach_pkg_util_stop.Stopper.Stop (rt.3=0xc4200ca090, pt.4=...)
    at /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:393
#10 0x0000000002a3620a in server.func1 () at /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/server/admin.go:1206
#11 0x0000000002a91b87 in server.github_com_cockroachdb_cockroach_pkg_server..thunk27 (__go_thunk_parameter=0xc429e64cd0)
    at /usr/local/home/tduberstein/go/src/github.com/cockroachdb/cockroach/pkg/server/admin.go:1202
#12 0x00007ffff6aeff89 in runtime.kickoff () at ../../../src/libgo/go/runtime/proc.go:1161

#7 is the interesting one, that is this code:
https://github.com/cockroachdb/cockroach/blob/e4c03f9750e53f56d0516d0785f38a706a28eea9/pkg/storage/engine/temp_dir.go#L57-L61

note that we're storing a function literal as a callback to be caller during shutdown - I think gccgo is miscompiling here, such that the closed-over flock is zero when the function is called.

@ianlancetaylor
Copy link
Contributor

The value of flock is set by this function:

func lockFile(filename string) (C.DBFileLock, error) {
	var lock C.DBFileLock
	return lock, statusToError(C.DBLockFile(goToCSlice([]byte(filename)), &lock))
}

This looks to me like an order of evaluation problem, similar to what is being discussed right now in #23188. This code is assuming that the call to C.DBLockFile sets &lock before the value of lock is evaluated in the return statement. The language spec makes no such guarantee. For correctness this code must be rewritten as something like

func lockFile(filename string) (C.DBFileLock, error) {
	var lock C.DBFileLock
	err := statusToError(C.DBLockFile(goToCSlice([]byte(filename)), &lock))
	return lock, err
}

@tamird
Copy link
Contributor Author

tamird commented Jan 31, 2018

Ah, I think that's correct. I found another issue like this one in a different area of the code.

@tamird
Copy link
Contributor Author

tamird commented Jan 31, 2018

Confirmed: that change fixes the problem. Closing, and I'll follow the discussion in #23188.

@tamird tamird closed this as completed Jan 31, 2018
@golang golang locked and limited conversation to collaborators Jan 31, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants