runtime: crash with signal 0xb near runtime.sweepone

[rhysh]

$ go version
go version devel +8cd191b Sat May 30 12:21:56 2015 +0000 linux/amd64
$ uname -a | awk '$2="host"'
Linux host 3.13.0-52-generic #86~precise1-Ubuntu SMP Tue May 5 18:08:21 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I have a process that receives data over a few hundred concurrent TCP connections and writes them to files. It's been crashing on recent versions of tip (it was stable on 1.4.1).

fatal error: unexpected signal during runtime execution
[signal 0xb code=0x80 addr=0x0 pc=0x4246fd]

runtime stack:
runtime.throw(0x8ef170, 0x2a)
    /usr/local/go/src/runtime/panic.go:527 +0x96
runtime.sigpanic()
    /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5d
runtime.mSpan_Sweep(0x7f0550633990, 0x8ec450008ec00, 0xc200002401)
    /usr/local/go/src/runtime/mgcsweep.go:182 +0x1bd
runtime.sweepone(0x1)
    /usr/local/go/src/runtime/mgcsweep.go:97 +0x161
runtime.gosweepone.func1()
    /usr/local/go/src/runtime/mgcsweep.go:109 +0x28
runtime.systemstack(0xc2094bdf08)
    /usr/local/go/src/runtime/asm_amd64.s:278 +0xb1
runtime.gosweepone(0xa8dd18)
    /usr/local/go/src/runtime/mgcsweep.go:110 +0x44
runtime.mCentral_CacheSpan(0xa8f020, 0x7f0550acb570)
    /usr/local/go/src/runtime/mcentral.go:43 +0xab
runtime.mCache_Refill(0x7f0550ad31c0, 0xe, 0x7f0550acb570)
    /usr/local/go/src/runtime/mcache.go:118 +0xd5
runtime.mallocgc.func2()
    /usr/local/go/src/runtime/malloc.go:608 +0x32
runtime.systemstack(0xc208016000)
    /usr/local/go/src/runtime/asm_amd64.s:262 +0x7c
runtime.mstart()
    /usr/local/go/src/runtime/proc1.go:656

goroutine 78 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:216 fp=0xc2086dbca8 sp=0xc2086dbca0
runtime.mallocgc(0xd0, 0x815da0, 0x0, 0xc2086dbda8)
    /usr/local/go/src/runtime/malloc.go:609 +0x7b9 fp=0xc2086dbd78 sp=0xc2086dbca8
runtime.newobject(0x815da0, 0x7f055091bfb0)
    /usr/local/go/src/runtime/malloc.go:731 +0x49 fp=0xc2086dbda0 sp=0xc2086dbd78
redacted(0xc210331080, 0x7e, 0x80, 0x7f055091bfb0, 0xc2080bec90, 0x0, 0x0)
    /redacted.go:314 +0xf6 fp=0xc2086dbe18 sp=0xc2086dbda0
redacted(0xc210331080, 0x7e, 0x80, 0x7f055091bfb0, 0xc2080bec90, 0x0, 0x0)
    /redacted.go:300 +0x70 fp=0xc2086dbe58 sp=0xc2086dbe18
redacted(0xc20800f5f0, 0xc2080c0180, 0xc208010420, 0xc2080fc050)
    /redacted.go:147 +0x128 fp=0xc2086dbf90 sp=0xc2086dbe58
redacted(0xc2086a2000, 0xc20800f5f0, 0xc2080c0180, 0xc208010420, 0xc2080fc050)
    /redacted.go:111 +0x6d fp=0xc2086dbfb8 sp=0xc2086dbf90
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1670 +0x1 fp=0xc2086dbfc0 sp=0xc2086dbfb8
created by redacted
    /redacted.go:112 +0x37f

goroutine 1 [chan receive, 102 minutes]:
main.main()
    /redacted.go:230 +0xe7b

goroutine 5 [chan send]:
redacted(0xc20809e000)
    /redacted.go:199 +0x1f0
created by redacted.init.1
    /redacted.go:184 +0x6e

[snip]

[ianlancetaylor]

CC @RLH @aclements

It would be very helpful to have a reproducible test case.

[aclements]

If it's reliable (or fairly reliable), but you can't reduce it to a test case you can share, it would be helpful if you could bisect it to a bad commit.

Another potentially useful thing to do would be to run with GODEBUG=efence=1, though that perturbs the memory layout enough that the crash conditions may not occur.

[rhysh]

I've only seen this particular crash once in the past few days of running (#11023 is much more common), so I don't think I'll be able to bisect effectively.

I tried running with GODEBUG=efence=1, but my program crashes with "fatal error: out of memory (stackalloc)" every minute or so and I haven't seen it crash with any other errors of interest (it crashed once with #11023). Is the memory exhaustion expected even on amd64?

I'll see what I can do on a reduced test case.

[aclements]

I tried running with GODEBUG=efence=1, but my program crashes with "fatal error: out of memory (stackalloc)" every minute or so and I haven't seen it crash with any other errors of interest (it crashed once with #11023). Is the memory exhaustion expected even on amd64?

That's a little surprising, but efence does change memory allocation patterns fairly substantially.

I'll see what I can do on a reduced test case.

Thanks!

[aclements]

For reference, we've seen this twice on the build dashboard:

2015-05-07T21:08:29-9626561/nacl-amd64p32
2015-05-14T15:55:42-94934f8/linux-amd64-sid

I've also seen it once on a trybot:

https://storage.googleapis.com/go-build-log/98c65eab/freebsd-amd64-gce101_4e0f3a2f.log (https://go-review.googlesource.com/#/c/10481)

[gopherbot]

CL https://golang.org/cl/10714 mentions this issue.

[gopherbot]

CL https://golang.org/cl/10713 mentions this issue.

[gopherbot]

CL https://golang.org/cl/10791 mentions this issue.

[aclements]

Hi @rhysh. We've fixed several memory corruption and lost write barrier issues in the runtime over the past few weeks. As for #11023, please try to reproduce the problem with current master and reopen this issue if it's still happening. Thanks!