runtime: panic fatal error: malloc deadlock

[dgryski]

What version of Go are you using (go version)?

I've bisected the issue to 2edc4d4

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

darwin/amd64

What did you do?

During development of a large internal service, we were running our test suite. The tests triggered a nil pointer dereference. On 1.9.4 the output was the usual

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1300854]

But on 1.10 we instead got

panic: fatal error: malloc deadlock
panic during panic

runtime stack:
runtime.startpanic_m()
	/Users/dgryski/work/src/go/src/runtime/panic.go:693 +0x179
runtime.systemstack(0x0)
	/Users/dgryski/work/src/go/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
	/Users/dgryski/work/src/go/src/runtime/proc.go:1170

goroutine 207 [running]:
runtime.systemstack_switch()
	/Users/dgryski/work/src/go/src/runtime/asm_amd64.s:363 fp=0xc420373a88 sp=0xc420373a80 pc=0x1057fe0
runtime.startpanic()
	/Users/dgryski/work/src/go/src/runtime/panic.go:592 +0x1e fp=0xc420373aa0 sp=0xc420373a88 pc=0x102c8be
runtime.throw(0x1367aa4, 0xf)
	/Users/dgryski/work/src/go/src/runtime/panic.go:618 +0x74 fp=0xc420373ac0 sp=0xc420373aa0 pc=0x102c9e4
runtime.mallocgc(0x2010, 0x0, 0x1303401, 0x12ae3a8)
	/Users/dgryski/work/src/go/src/runtime/malloc.go:621 +0x99d fp=0xc420373b60 sp=0xc420373ac0 pc=0x10126cd
runtime.itabAdd(0x2541840)
	/Users/dgryski/work/src/go/src/runtime/iface.go:122 +0x83 fp=0xc420373bb0 sp=0xc420373b60 pc=0x100f403
runtime.getitab(0x1303480, 0x12e2680, 0xc4202e4301, 0xc4202e43b0)
	/Users/dgryski/work/src/go/src/runtime/iface.go:70 +0x404 fp=0xc420373c28 sp=0xc420373bb0 pc=0x100f284
runtime.assertE2I2(0x1303480, 0x12e2680, 0xc42022ad30, 0x7, 0xc400000007, 0xc420373cb8)
	/Users/dgryski/work/src/go/src/runtime/iface.go:592 +0x43 fp=0xc420373c58 sp=0xc420373c28 pc=0x10107e3
runtime.printany(0x12e2680, 0xc42022ad30)
	/Users/dgryski/work/src/go/src/runtime/error.go:77 +0x69 fp=0xc420373d18 sp=0xc420373c58 pc=0x1007709
runtime.printpanics(0xc420373da0)
	/Users/dgryski/work/src/go/src/runtime/panic.go:420 +0x71 fp=0xc420373d38 sp=0xc420373d18 pc=0x102c101
panic(0x12fd360, 0x1528330)
	/Users/dgryski/work/src/go/src/runtime/panic.go:553 +0x3b4 fp=0xc420373dd8 sp=0xc420373d38 pc=0x102c544
runtime.panicmem()
	/Users/dgryski/work/src/go/src/runtime/panic.go:63 +0x5e fp=0xc420373df8 sp=0xc420373dd8 pc=0x102b24e
runtime.sigpanic()
	/Users/dgryski/work/src/go/src/runtime/signal_unix.go:388 +0x17a fp=0xc420373e48 sp=0xc420373df8 pc=0x1041fca
github.com/fastly/service1/service2.(*GetStats).Add(...)
	/Users/dgryski/go/src/github.com/fastly/service1/service2/client.go:222
github.com/fastly/service1/service2.(*Client).GetAll.func2()
	/Users/dgryski/go/src/github.com/fastly/service1/service2/client.go:276 +0x27d fp=0xc420373fe0 sp=0xc420373e48 pc=0x1292c8d
runtime.goexit()
	/Users/dgryski/work/src/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420373fe8 sp=0xc420373fe0 pc=0x105ab31
created by github.com/fastly/service1/service2.(*Client).GetAll
	/Users/dgryski/go/src/github.com/fastly/service1/service2/client.go:285 +0x232

[ output trimmed of all the other goroutine stack traces]

Some notes from trying to minimize this into a useful test case:

• There are lots of goroutines running, and the nil dereference happens in one
• When running with -test.v, the normal panic output is produced
• I need to run multiple tests from this package in addition to the failing test. Running just the one test that fails produced the normal panic output.
• Adding -race doesn't make a difference

[ianlancetaylor]

We crash in the unlikely case that we have to grow the itab table while adding an itab entry for converting string to stringer. I will send a CL.

[gopherbot]

Change https://golang.org/cl/96755 mentions this issue: runtime: in prepanic, ensure printany doesn't allocate printing a string

[dgryski]

I can confirm this CL fixes my issue.

[andybons]

Reopening for cherry-pick

[andybons]

CL 96755 OK for Go 1.10.1

[gopherbot]

Change https://golang.org/cl/102781 mentions this issue: [release-branch.go1.10] runtime: don't check for String/Error methods in printany

[SuperQ]

We're still seeing issues that appear to be related on go1.10.1, should we file a separate issue or reopen?

[ianlancetaylor]

@SuperQ Please file a new issue. We're pretty sure this problem is fixed. I would guess that you are seeing a different problem with similar symptoms.

[denismakogon]

faced with this type of error while was doing something like this:

a function that generates a slice of int64 items:

func makeInt64Range(min, max int64) []int64 {
	a := make([]int64, max-min+1)
	for i := range a {
		a[i] = min + int64(i)
	}
	return a
}

accidentally (was distracted by random convo) it was called with mismatched boundaries:

	int64Range := makeInt64Range(1025, 1000)

and got this:

panic: fatal error: malloc deadlock
panic during panic

runtime stack:
runtime.startpanic_m()
	/usr/local/go/src/runtime/panic.go:693 +0x179
runtime.systemstack(0x0)
	/usr/local/go/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1170

Golang comes from: golang:1.10-alpine3.7 docker image.

Tried that on playground: https://play.golang.org/p/D5j5EzDgNiW
Error is: panic: runtime error: makeslice: len out of range which appear to be different from what i have.

[aclements]

@denismakogon, please open a new issue, thanks!

[denismakogon]

@aclements thanks, it appears that fixed was merged somewhere on the road to 1.10.2 since i don't get malloc error on the playground (which is 1.10.2). Will test on golang:1.10.2-alpine3.7 (when that will be available) and if it's still there I will definitely open an issue.