runtime: crash with "fatal error: deferproc: d.panic != nil after newdefer"

[rhysh]

$ go version
go version devel +8cd191b Sat May 30 12:21:56 2015 +0000 linux/amd64
$ uname -a | awk '$2="host"'
Linux host 3.13.0-52-generic #86~precise1-Ubuntu SMP Tue May 5 18:08:21 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I have a process that receives data over a few hundred concurrent TCP connections and writes them to files. It's been crashing on recent versions of tip (it was stable on 1.4.1).

fatal error: deferproc: d.panic != nil after newdefer

runtime stack:
runtime.throw(0x8e46f0, 0x28)
    /usr/local/go/src/runtime/panic.go:527 +0x96
runtime.deferproc.func1()
    /usr/local/go/src/runtime/panic.go:74 +0x7f
runtime.systemstack(0xc208016000)
    /usr/local/go/src/runtime/asm_amd64.s:262 +0x7c
runtime.mstart()
    /usr/local/go/src/runtime/proc1.go:656

goroutine 1368 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:216 fp=0xc208b97328 sp=0xc208b97320
runtime.deferproc(0xc200000008, 0x929568)
    /usr/local/go/src/runtime/panic.go:80 +0xbb fp=0xc208b97380 sp=0xc208b97328
net.(*netFD).Read(0xc20bb53f10, 0xc208c3505a, 0xfa6, 0xfa6, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:224 +0xde fp=0xc208b97450 sp=0xc208b97380
net.(*conn).Read(0xc208932b10, 0xc208c3505a, 0xfa6, 0xfa6, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:124 +0xe7 fp=0xc208b974b8 sp=0xc208b97450
bufio.(*Scanner).Scan(0xc20b525a00, 0xc208010420)
    /usr/local/go/src/bufio/scan.go:180 +0x87a fp=0xc208b97678 sp=0xc208b974b8
redacted(0xc20800c5f0, 0x7f1b5fc07fe8, 0xc208932b10, 0xc208010420)
    /redacted.go:99 +0x93 fp=0xc208b977c0 sp=0xc208b97678
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1670 +0x1 fp=0xc208b977c8 sp=0xc208b977c0
created by redacted
    /redacted.go:81 +0x21e

goroutine 1 [chan receive, 14 minutes]:
main.main()
    /redacted.go:230 +0xe7b

goroutine 5 [chan send]:
redacted(0xc20809e000)
    /redacted.go:199 +0x1f0
created by redacted.init.1
    /redacted.go:184 +0x6e

[snip]

[bradfitz]

How quickly does it crash? Any chance you could bisect it down to a specific revision which caused it to start failing?

[rhysh]

I've only seen this particular crash once. I'd been running on 4e4c1f9 from last Wednesday May 27 for a few days, and only saw this crash after an upgrade to 8cd191b (from Sunday May 30). It might be recent, but my process doesn't crash quickly or reliably enough to bisect (I'm also seeing #10941 a few times a day).

[rhysh]

I haven't observed this one for quite a while. I think it's resolved. Thanks!