Skip to content

build: test timeouts / flakes (TestCgoPprofPIE, TestCgoExternalThreadSIGPROF) #18038

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
bradfitz opened this issue Nov 24, 2016 · 4 comments
Closed

build: test timeouts / flakes (TestCgoPprofPIE, TestCgoExternalThreadSIGPROF) #18038

bradfitz opened this issue Nov 24, 2016 · 4 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone
Go1.8

Comments

@bradfitz
Copy link
Contributor

After upgrading the Kubernetes builder and by necessity making a change to the buildlet process to catch SIGINT and SIGTERM, we saw four or five of these failures right away:

https://storage.googleapis.com/go-build-log/7b35b6e1/linux-amd64_64bfee29.log
https://storage.googleapis.com/go-build-log/ad40ae83/linux-amd64_050c050f.log
etc

One of them:

ok  	regexp/syntax	0.417s
panic: test timed out after 3m0s

goroutine 104441 [running]:
panic(0x5eb900, 0xc4204e4460)
	/tmp/workdir/go/src/runtime/panic.go:531 +0x1cf fp=0xc42005df78 sp=0xc42005dee0
testing.startAlarm.func1()
	/tmp/workdir/go/src/testing/testing.go:1041 +0xf9 fp=0xc42005dfe0 sp=0xc42005df78
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42005dfe8 sp=0xc42005dfe0
created by time.goFunc
	/tmp/workdir/go/src/time/sleep.go:170 +0x44

goroutine 1 [chan receive, 2 minutes, locked to thread]:
runtime.gopark(0x6509b8, 0xc420328598, 0x641be9, 0xc, 0xc42002a617, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42018cb30 sp=0xc42018cb00
runtime.goparkunlock(0xc420328598, 0x641be9, 0xc, 0x42ea17, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc42018cb70 sp=0xc42018cb30
runtime.chanrecv(0x5e9b00, 0xc420328540, 0x0, 0x58361a01, 0xc420010101)
	/tmp/workdir/go/src/runtime/chan.go:513 +0x371 fp=0xc42018cc10 sp=0xc42018cb70
runtime.chanrecv1(0x5e9b00, 0xc420328540, 0x0)
	/tmp/workdir/go/src/runtime/chan.go:395 +0x35 fp=0xc42018cc48 sp=0xc42018cc10
testing.tRunner.func1(0xc4200c21c0)
	/tmp/workdir/go/src/testing/testing.go:655 +0x17e fp=0xc42018ccc8 sp=0xc42018cc48
testing.tRunner(0xc4200c21c0, 0xc42006dd90)
	/tmp/workdir/go/src/testing/testing.go:681 +0x149 fp=0xc42018cd08 sp=0xc42018ccc8
testing.runTests(0xc42000cbe0, 0x76eb00, 0xce, 0xce, 0xc42006de01)
	/tmp/workdir/go/src/testing/testing.go:906 +0x29d fp=0xc42018cdc0 sp=0xc42018cd08
testing.(*M).Run(0xc42018cf20, 0xc42006def0)
	/tmp/workdir/go/src/testing/testing.go:841 +0xfc fp=0xc42018ceb0 sp=0xc42018cdc0
runtime_test.TestMain(0xc42006df20)
	/tmp/workdir/go/src/runtime/crash_test.go:28 +0x2f fp=0xc42018cf00 sp=0xc42018ceb0
main.main()
	runtime/_test/_testmain.go:844 +0xf7 fp=0xc42018cf88 sp=0xc42018cf00
runtime.main()
	/tmp/workdir/go/src/runtime/proc.go:185 +0x20a fp=0xc42018cfe0 sp=0xc42018cf88
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42018cfe8 sp=0xc42018cfe0

goroutine 17 [syscall, 2 minutes, locked to thread]:
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42005cfe8 sp=0xc42005cfe0

goroutine 2 [force gc (idle), 2 minutes]:
runtime.gopark(0x6509b8, 0x770930, 0x642ec6, 0xf, 0x100000000650814, 0x1)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42004a768 sp=0xc42004a738
runtime.goparkunlock(0x770930, 0x642ec6, 0xf, 0xc420000114, 0x1)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc42004a7a8 sp=0xc42004a768
runtime.forcegchelper()
	/tmp/workdir/go/src/runtime/proc.go:226 +0x9e fp=0xc42004a7e0 sp=0xc42004a7a8
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42004a7e8 sp=0xc42004a7e0
created by runtime.init.4
	/tmp/workdir/go/src/runtime/proc.go:215 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0x6509b8, 0x770ca0, 0x641ff9, 0xd, 0x458014, 0x1)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42004af58 sp=0xc42004af28
runtime.goparkunlock(0x770ca0, 0x641ff9, 0xd, 0x14, 0x1)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc42004af98 sp=0xc42004af58
runtime.bgsweep(0xc420078000)
	/tmp/workdir/go/src/runtime/mgcsweep.go:72 +0x12e fp=0xc42004afd8 sp=0xc42004af98
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42004afe0 sp=0xc42004afd8
created by runtime.gcenable
	/tmp/workdir/go/src/runtime/mgc.go:209 +0x61

goroutine 4 [finalizer wait]:
runtime.gopark(0x6509b8, 0x78c848, 0x642884, 0xe, 0x14, 0x1)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4203b2f18 sp=0xc4203b2ee8
runtime.goparkunlock(0x78c848, 0x642884, 0xe, 0x2000000014, 0x1)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc4203b2f58 sp=0xc4203b2f18
runtime.runfinq()
	/tmp/workdir/go/src/runtime/mfinal.go:161 +0xb2 fp=0xc4203b2fe0 sp=0xc4203b2f58
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4203b2fe8 sp=0xc4203b2fe0
created by runtime.createfing
	/tmp/workdir/go/src/runtime/mfinal.go:142 +0x62

goroutine 5 [timer goroutine (idle)]:
runtime.gopark(0x6509b8, 0x770de0, 0x64672f, 0x16, 0x651d14, 0x1)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42004bf20 sp=0xc42004bef0
runtime.goparkunlock(0x770de0, 0x64672f, 0x16, 0x14, 0x1)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc42004bf60 sp=0xc42004bf20
runtime.timerproc()
	/tmp/workdir/go/src/runtime/time.go:202 +0x2cf fp=0xc42004bfe0 sp=0xc42004bf60
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42004bfe8 sp=0xc42004bfe0
created by runtime.addtimerLocked
	/tmp/workdir/go/src/runtime/time.go:116 +0xed

goroutine 22920 [chan send, 2 minutes]:
runtime.gopark(0x6509b8, 0xc4203281d8, 0x640987, 0x9, 0x16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420223de0 sp=0xc420223db0
runtime.goparkunlock(0xc4203281d8, 0x640987, 0x9, 0xc420029916, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc420223e20 sp=0xc420223de0
runtime.chansend(0x5e9b00, 0xc420328180, 0xc420223f4f, 0xc4204ae001, 0x4d2b18, 0xc4204ae0e0)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc420223ed0 sp=0xc420223e20
runtime.chansend1(0x5e9b00, 0xc420328180, 0xc420223f4f)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc420223f10 sp=0xc420223ed0
testing.tRunner.func1(0xc4204ae0e0)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc420223f90 sp=0xc420223f10
testing.tRunner(0xc4204ae0e0, 0x651438)
	/tmp/workdir/go/src/testing/testing.go:681 +0x149 fp=0xc420223fd0 sp=0xc420223f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420223fd8 sp=0xc420223fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 23248 [chan send, 2 minutes]:
runtime.gopark(0x6509b8, 0xc420221df8, 0x640987, 0x9, 0x16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420225de0 sp=0xc420225db0
runtime.goparkunlock(0xc420221df8, 0x640987, 0x9, 0xc420020016, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc420225e20 sp=0xc420225de0
runtime.chansend(0x5e9b00, 0xc420221da0, 0xc420225f4f, 0xc42061b301, 0x4d2b18, 0xc42061b340)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc420225ed0 sp=0xc420225e20
runtime.chansend1(0x5e9b00, 0xc420221da0, 0xc420225f4f)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc420225f10 sp=0xc420225ed0
testing.tRunner.func1(0xc42061b340)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc420225f90 sp=0xc420225f10
testing.tRunner(0xc42061b340, 0x651a70)
	/tmp/workdir/go/src/testing/testing.go:681 +0x149 fp=0xc420225fd0 sp=0xc420225f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420225fd8 sp=0xc420225fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 23207 [chan send, 2 minutes]:
runtime.gopark(0x6509b8, 0xc420220238, 0x640987, 0x9, 0x16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420d9f8b8 sp=0xc420d9f888
runtime.goparkunlock(0xc420220238, 0x640987, 0x9, 0xc42001c616, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc420d9f8f8 sp=0xc420d9f8b8
runtime.chansend(0x5e9b00, 0xc4202201e0, 0xc420d9fa27, 0xc42061a001, 0x4d2b18, 0xc42061a000)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc420d9f9a8 sp=0xc420d9f8f8
runtime.chansend1(0x5e9b00, 0xc4202201e0, 0xc420d9fa27)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc420d9f9e8 sp=0xc420d9f9a8
testing.tRunner.func1(0xc42061a000)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc420d9fa68 sp=0xc420d9f9e8
runtime.call32(0x0, 0x651d90, 0xc420254070, 0x800000008)
	/tmp/workdir/go/src/runtime/asm_amd64.s:501 +0x48 fp=0xc420d9fa98 sp=0xc420d9fa68
runtime.Goexit()
	/tmp/workdir/go/src/runtime/panic.go:366 +0x153 fp=0xc420d9fad8 sp=0xc420d9fa98
testing.(*common).SkipNow(0xc42061a000)
	/tmp/workdir/go/src/testing/testing.go:574 +0x39 fp=0xc420d9faf0 sp=0xc420d9fad8
testing.(*common).Skipf(0xc42061a000, 0x64b493, 0x23, 0xc420d9fc40, 0x2, 0x2)
	/tmp/workdir/go/src/testing/testing.go:561 +0x83 fp=0xc420d9fb38 sp=0xc420d9faf0
runtime_test.checkGdbVersion(0xc42061a000)
	/tmp/workdir/go/src/runtime/runtime-gdb_test.go:52 +0x439 fp=0xc420d9fc90 sp=0xc420d9fb38
runtime_test.TestGdbAutotmpTypes(0xc42061a000)
	/tmp/workdir/go/src/runtime/runtime-gdb_test.go:299 +0x65 fp=0xc420d9ff90 sp=0xc420d9fc90
testing.tRunner(0xc42061a000, 0x651588)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc420d9ffd0 sp=0xc420d9ff90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420d9ffd8 sp=0xc420d9ffd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 179 [GC worker (idle)]:
runtime.gopark(0x6507f8, 0xc420017980, 0x6432b6, 0x10, 0xc42045c014, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420312750 sp=0xc420312720
runtime.gcBgMarkWorker(0xc42001a000)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4203127d8 sp=0xc420312750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4203127e0 sp=0xc4203127d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 24320 [chan receive, 2 minutes]:
runtime.gopark(0x6509b8, 0xc4200149b8, 0x641be9, 0xc, 0xc4202d6317, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42012fe98 sp=0xc42012fe68
runtime.goparkunlock(0xc4200149b8, 0x641be9, 0xc, 0xc420249d17, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc42012fed8 sp=0xc42012fe98
runtime.chanrecv(0x5e9b00, 0xc420014960, 0x0, 0xc42061a601, 0x9aae314)
	/tmp/workdir/go/src/runtime/chan.go:513 +0x371 fp=0xc42012ff78 sp=0xc42012fed8
runtime.chanrecv1(0x5e9b00, 0xc420014960, 0x0)
	/tmp/workdir/go/src/runtime/chan.go:395 +0x35 fp=0xc42012ffb0 sp=0xc42012ff78
testing.runTests.func1.1(0xc4200c21c0)
	/tmp/workdir/go/src/testing/testing.go:905 +0x47 fp=0xc42012ffd8 sp=0xc42012ffb0
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42012ffe0 sp=0xc42012ffd8
created by testing.runTests.func1
	/tmp/workdir/go/src/testing/testing.go:905 +0xaf

goroutine 22625 [semacquire, 2 minutes]:
runtime.gopark(0x6509b8, 0x77a5a0, 0x641151, 0xa, 0xc4203b4a19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4203b4aa8 sp=0xc4203b4a78
runtime.goparkunlock(0x77a5a0, 0x641151, 0xa, 0xc4203b4a19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc4203b4ae8 sp=0xc4203b4aa8
runtime.semacquire(0x770cc4, 0x3)
	/tmp/workdir/go/src/runtime/sema.go:130 +0x24a fp=0xc4203b4b58 sp=0xc4203b4ae8
sync.runtime_SemacquireMutex(0x770cc4)
	/tmp/workdir/go/src/runtime/sema.go:62 +0x34 fp=0xc4203b4b78 sp=0xc4203b4b58
sync.(*Mutex).Lock(0x770cc0)
	/tmp/workdir/go/src/sync/mutex.go:87 +0x9d fp=0xc4203b4bb0 sp=0xc4203b4b78
runtime_test.buildTestProg(0xc420242460, 0x6417d1, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:116 +0x83 fp=0xc4203b4d90 sp=0xc4203b4bb0
runtime_test.runTestProg(0xc420242460, 0x6417d1, 0xb, 0x6461db, 0x16, 0xc4201063d8, 0xc4201caf60)
	/tmp/workdir/go/src/runtime/crash_test.go:68 +0x89 fp=0xc4203b4ef8 sp=0xc4203b4d90
runtime_test.TestCgoExternalThreadPanic(0xc420242460)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:77 +0x7b fp=0xc4203b4f90 sp=0xc4203b4ef8
testing.tRunner(0xc420242460, 0x651348)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc4203b4fd0 sp=0xc4203b4f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4203b4fd8 sp=0xc4203b4fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 23229 [chan send, 2 minutes]:
runtime.gopark(0x6509b8, 0xc42039eb38, 0x640987, 0x9, 0x16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420da3878 sp=0xc420da3848
runtime.goparkunlock(0xc42039eb38, 0x640987, 0x9, 0xc42001d916, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc420da38b8 sp=0xc420da3878
runtime.chansend(0x5e9b00, 0xc42039eae0, 0xc420da39e7, 0xc4200c2701, 0x4d2b18, 0xc4200c27e0)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc420da3968 sp=0xc420da38b8
runtime.chansend1(0x5e9b00, 0xc42039eae0, 0xc420da39e7)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc420da39a8 sp=0xc420da3968
testing.tRunner.func1(0xc4200c27e0)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc420da3a28 sp=0xc420da39a8
runtime.call32(0x0, 0x651d90, 0xc4200441b0, 0x800000008)
	/tmp/workdir/go/src/runtime/asm_amd64.s:501 +0x48 fp=0xc420da3a58 sp=0xc420da3a28
runtime.Goexit()
	/tmp/workdir/go/src/runtime/panic.go:366 +0x153 fp=0xc420da3a98 sp=0xc420da3a58
testing.(*common).SkipNow(0xc4200c27e0)
	/tmp/workdir/go/src/testing/testing.go:574 +0x39 fp=0xc420da3ab0 sp=0xc420da3a98
testing.(*common).Skipf(0xc4200c27e0, 0x64b493, 0x23, 0xc420da3c00, 0x2, 0x2)
	/tmp/workdir/go/src/testing/testing.go:561 +0x83 fp=0xc420da3af8 sp=0xc420da3ab0
runtime_test.checkGdbVersion(0xc4200c27e0)
	/tmp/workdir/go/src/runtime/runtime-gdb_test.go:52 +0x439 fp=0xc420da3c50 sp=0xc420da3af8
runtime_test.TestGdbBacktrace(0xc4200c27e0)
	/tmp/workdir/go/src/runtime/runtime-gdb_test.go:225 +0x65 fp=0xc420da3f90 sp=0xc420da3c50
testing.tRunner(0xc4200c27e0, 0x651590)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc420da3fd0 sp=0xc420da3f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420da3fd8 sp=0xc420da3fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 23247 [chan send, 2 minutes]:
runtime.gopark(0x6509b8, 0xc420221cd8, 0x640987, 0x9, 0x16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420227de0 sp=0xc420227db0
runtime.goparkunlock(0xc420221cd8, 0x640987, 0x9, 0xc42001d916, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc420227e20 sp=0xc420227de0
runtime.chansend(0x5e9b00, 0xc420221c80, 0xc420227f4f, 0xc42061b201, 0x4d2b18, 0xc42061b260)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc420227ed0 sp=0xc420227e20
runtime.chansend1(0x5e9b00, 0xc420221c80, 0xc420227f4f)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc420227f10 sp=0xc420227ed0
testing.tRunner.func1(0xc42061b260)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc420227f90 sp=0xc420227f10
testing.tRunner(0xc42061b260, 0x651a78)
	/tmp/workdir/go/src/testing/testing.go:681 +0x149 fp=0xc420227fd0 sp=0xc420227f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420227fd8 sp=0xc420227fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 24930 [GC worker (idle)]:
runtime.gopark(0x6507f8, 0xc42013c010, 0x6432b6, 0x10, 0xc4201cd714, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4201cd750 sp=0xc4201cd720
runtime.gcBgMarkWorker(0xc42001c600)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4201cd7d8 sp=0xc4201cd750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4201cd7e0 sp=0xc4201cd7d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 21669 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc4200e4010, 0x6432b6, 0x10, 0x5fd214, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4201db750 sp=0xc4201db720
runtime.gcBgMarkWorker(0xc420029900)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4201db7d8 sp=0xc4201db750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4201db7e0 sp=0xc4201db7d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 25158 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc42013c3e0, 0x6432b6, 0x10, 0x5fd214, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420333f50 sp=0xc420333f20
runtime.gcBgMarkWorker(0xc42003e000)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc420333fd8 sp=0xc420333f50
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420333fe0 sp=0xc420333fd8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 21670 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc4200e4480, 0x6432b6, 0x10, 0x5fd214, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42019e750 sp=0xc42019e720
runtime.gcBgMarkWorker(0xc420033300)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc42019e7d8 sp=0xc42019e750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42019e7e0 sp=0xc42019e7d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 29072 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc4204e5110, 0x6432b6, 0x10, 0x5fd214, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420eb4750 sp=0xc420eb4720
runtime.gcBgMarkWorker(0xc42003cc00)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc420eb47d8 sp=0xc420eb4750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420eb47e0 sp=0xc420eb47d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 23228 [chan send, 2 minutes]:
runtime.gopark(0x6509b8, 0xc42039e9b8, 0x640987, 0x9, 0x16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42006b458 sp=0xc42006b428
runtime.goparkunlock(0xc42039e9b8, 0x640987, 0x9, 0xc42001a016, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc42006b498 sp=0xc42006b458
runtime.chansend(0x5e9b00, 0xc42039e960, 0xc42006b5c7, 0xc4200c2201, 0x4d2b18, 0xc4200c22a0)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc42006b548 sp=0xc42006b498
runtime.chansend1(0x5e9b00, 0xc42039e960, 0xc42006b5c7)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc42006b588 sp=0xc42006b548
testing.tRunner.func1(0xc4200c22a0)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc42006b608 sp=0xc42006b588
runtime.call32(0x0, 0x651d90, 0xc4201064f0, 0x800000008)
	/tmp/workdir/go/src/runtime/asm_amd64.s:501 +0x48 fp=0xc42006b638 sp=0xc42006b608
runtime.Goexit()
	/tmp/workdir/go/src/runtime/panic.go:366 +0x153 fp=0xc42006b678 sp=0xc42006b638
testing.(*common).SkipNow(0xc4200c22a0)
	/tmp/workdir/go/src/testing/testing.go:574 +0x39 fp=0xc42006b690 sp=0xc42006b678
testing.(*common).Skipf(0xc4200c22a0, 0x64b493, 0x23, 0xc42006b7e0, 0x2, 0x2)
	/tmp/workdir/go/src/testing/testing.go:561 +0x83 fp=0xc42006b6d8 sp=0xc42006b690
runtime_test.checkGdbVersion(0xc4200c22a0)
	/tmp/workdir/go/src/runtime/runtime-gdb_test.go:52 +0x439 fp=0xc42006b830 sp=0xc42006b6d8
runtime_test.TestGdbPython(0xc4200c22a0)
	/tmp/workdir/go/src/runtime/runtime-gdb_test.go:90 +0x70 fp=0xc42006bf90 sp=0xc42006b830
testing.tRunner(0xc4200c22a0, 0x651598)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc42006bfd0 sp=0xc42006bf90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42006bfd8 sp=0xc42006bfd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22973 [chan send, 2 minutes]:
runtime.gopark(0x6509b8, 0xc42039e538, 0x640987, 0x9, 0x16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420226de0 sp=0xc420226db0
runtime.goparkunlock(0xc42039e538, 0x640987, 0x9, 0xc42001b316, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc420226e20 sp=0xc420226de0
runtime.chansend(0x5e9b00, 0xc42039e4e0, 0xc420226f4f, 0xc4200c2501, 0x4d2b18, 0xc4200c2540)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc420226ed0 sp=0xc420226e20
runtime.chansend1(0x5e9b00, 0xc42039e4e0, 0xc420226f4f)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc420226f10 sp=0xc420226ed0
testing.tRunner.func1(0xc4200c2540)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc420226f90 sp=0xc420226f10
testing.tRunner(0xc4200c2540, 0x6516e8)
	/tmp/workdir/go/src/testing/testing.go:681 +0x149 fp=0xc420226fd0 sp=0xc420226f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420226fd8 sp=0xc420226fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22974 [chan send, 2 minutes]:
runtime.gopark(0x6509b8, 0xc42039e658, 0x640987, 0x9, 0x16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42019bde0 sp=0xc42019bdb0
runtime.goparkunlock(0xc42039e658, 0x640987, 0x9, 0xc42001b316, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc42019be20 sp=0xc42019bde0
runtime.chansend(0x5e9b00, 0xc42039e600, 0xc42019bf4f, 0xc4200c2601, 0x4d2b18, 0xc4200c2620)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc42019bed0 sp=0xc42019be20
runtime.chansend1(0x5e9b00, 0xc42039e600, 0xc42019bf4f)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc42019bf10 sp=0xc42019bed0
testing.tRunner.func1(0xc4200c2620)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc42019bf90 sp=0xc42019bf10
testing.tRunner(0xc4200c2620, 0x6516f0)
	/tmp/workdir/go/src/testing/testing.go:681 +0x149 fp=0xc42019bfd0 sp=0xc42019bf90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42019bfd8 sp=0xc42019bfd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22623 [semacquire, 2 minutes]:
runtime.gopark(0x6509b8, 0x77a5a0, 0x641151, 0xa, 0xc421120e19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc421120aa8 sp=0xc421120a78
runtime.goparkunlock(0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc421120ae8 sp=0xc421120aa8
runtime.semacquire(0x770cc4, 0x3)
	/tmp/workdir/go/src/runtime/sema.go:130 +0x24a fp=0xc421120b58 sp=0xc421120ae8
sync.runtime_SemacquireMutex(0x770cc4)
	/tmp/workdir/go/src/runtime/sema.go:62 +0x34 fp=0xc421120b78 sp=0xc421120b58
sync.(*Mutex).Lock(0x770cc0)
	/tmp/workdir/go/src/sync/mutex.go:87 +0x9d fp=0xc421120bb0 sp=0xc421120b78
runtime_test.buildTestProg(0xc4202422a0, 0x6417d1, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:116 +0x83 fp=0xc421120d90 sp=0xc421120bb0
runtime_test.runTestProg(0xc4202422a0, 0x6417d1, 0xb, 0x641979, 0xc, 0xc4200440d8, 0xc4201e1760)
	/tmp/workdir/go/src/runtime/crash_test.go:68 +0x89 fp=0xc421120ef8 sp=0xc421120d90
runtime_test.TestCgoTraceback(0xc4202422a0)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:40 +0x7b fp=0xc421120f90 sp=0xc421120ef8
testing.tRunner(0xc4202422a0, 0x651398)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc421120fd0 sp=0xc421120f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc421120fd8 sp=0xc421120fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 25059 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc420d94050, 0x6432b6, 0x10, 0x5fd214, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42025c750 sp=0xc42025c720
runtime.gcBgMarkWorker(0xc42002c000)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc42025c7d8 sp=0xc42025c750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42025c7e0 sp=0xc42025c7d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 22622 [semacquire, 2 minutes]:
runtime.gopark(0x6509b8, 0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc421121aa8 sp=0xc421121a78
runtime.goparkunlock(0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc421121ae8 sp=0xc421121aa8
runtime.semacquire(0x770cc4, 0x3)
	/tmp/workdir/go/src/runtime/sema.go:130 +0x24a fp=0xc421121b58 sp=0xc421121ae8
sync.runtime_SemacquireMutex(0x770cc4)
	/tmp/workdir/go/src/runtime/sema.go:62 +0x34 fp=0xc421121b78 sp=0xc421121b58
sync.(*Mutex).Lock(0x770cc0)
	/tmp/workdir/go/src/sync/mutex.go:87 +0x9d fp=0xc421121bb0 sp=0xc421121b78
runtime_test.buildTestProg(0xc4202421c0, 0x6417d1, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:116 +0x83 fp=0xc421121d90 sp=0xc421121bb0
runtime_test.runTestProg(0xc4202421c0, 0x6417d1, 0xb, 0x643999, 0x11, 0xc420106218, 0xc42012af60)
	/tmp/workdir/go/src/runtime/crash_test.go:68 +0x89 fp=0xc421121ef8 sp=0xc421121d90
runtime_test.TestCgoSignalDeadlock(0xc4202421c0)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:31 +0x84 fp=0xc421121f90 sp=0xc421121ef8
testing.tRunner(0xc4202421c0, 0x651388)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc421121fd0 sp=0xc421121f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc421121fd8 sp=0xc421121fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 23198 [GC worker (idle)]:
runtime.gopark(0x6507f8, 0xc4200165b0, 0x6432b6, 0x10, 0xc420eb8014, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4201cb750 sp=0xc4201cb720
runtime.gcBgMarkWorker(0xc42001d900)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4201cb7d8 sp=0xc4201cb750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4201cb7e0 sp=0xc4201cb7d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 22972 [chan send, 2 minutes]:
runtime.gopark(0x6509b8, 0xc42039e418, 0x640987, 0x9, 0x16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42019ade0 sp=0xc42019adb0
runtime.goparkunlock(0xc42039e418, 0x640987, 0x9, 0xc420021316, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc42019ae20 sp=0xc42019ade0
runtime.chansend(0x5e9b00, 0xc42039e3c0, 0xc42019af4f, 0xc4200c2401, 0x4d2b18, 0xc4200c2460)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc42019aed0 sp=0xc42019ae20
runtime.chansend1(0x5e9b00, 0xc42039e3c0, 0xc42019af4f)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc42019af10 sp=0xc42019aed0
testing.tRunner.func1(0xc4200c2460)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc42019af90 sp=0xc42019af10
testing.tRunner(0xc4200c2460, 0x651708)
	/tmp/workdir/go/src/testing/testing.go:681 +0x149 fp=0xc42019afd0 sp=0xc42019af90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42019afd8 sp=0xc42019afd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22975 [chan send, 2 minutes]:
runtime.gopark(0x6509b8, 0xc42039e778, 0x640987, 0x9, 0x16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4203b0de0 sp=0xc4203b0db0
runtime.goparkunlock(0xc42039e778, 0x640987, 0x9, 0xc42001c616, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc4203b0e20 sp=0xc4203b0de0
runtime.chansend(0x5e9b00, 0xc42039e720, 0xc4203b0f4f, 0xc4200c2701, 0x4d2b18, 0xc4200c2700)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc4203b0ed0 sp=0xc4203b0e20
runtime.chansend1(0x5e9b00, 0xc42039e720, 0xc4203b0f4f)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc4203b0f10 sp=0xc4203b0ed0
testing.tRunner.func1(0xc4200c2700)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc4203b0f90 sp=0xc4203b0f10
testing.tRunner(0xc4200c2700, 0x651700)
	/tmp/workdir/go/src/testing/testing.go:681 +0x149 fp=0xc4203b0fd0 sp=0xc4203b0f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4203b0fd8 sp=0xc4203b0fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22624 [semacquire, 2 minutes]:
runtime.gopark(0x6509b8, 0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4203b5aa8 sp=0xc4203b5a78
runtime.goparkunlock(0x77a5a0, 0x641151, 0xa, 0xc4203b5a19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc4203b5ae8 sp=0xc4203b5aa8
runtime.semacquire(0x770cc4, 0x3)
	/tmp/workdir/go/src/runtime/sema.go:130 +0x24a fp=0xc4203b5b58 sp=0xc4203b5ae8
sync.runtime_SemacquireMutex(0x770cc4)
	/tmp/workdir/go/src/runtime/sema.go:62 +0x34 fp=0xc4203b5b78 sp=0xc4203b5b58
sync.(*Mutex).Lock(0x770cc0)
	/tmp/workdir/go/src/sync/mutex.go:87 +0x9d fp=0xc4203b5bb0 sp=0xc4203b5b78
runtime_test.buildTestProg(0xc420242380, 0x6417d1, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:116 +0x83 fp=0xc4203b5d90 sp=0xc4203b5bb0
runtime_test.runTestProg(0xc420242380, 0x6417d1, 0xb, 0x641f91, 0xd, 0xc420106358, 0xc42012b760)
	/tmp/workdir/go/src/runtime/crash_test.go:68 +0x89 fp=0xc4203b5ef8 sp=0xc4203b5d90
runtime_test.TestCgoCallbackGC(0xc420242380)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:65 +0x84 fp=0xc4203b5f90 sp=0xc4203b5ef8
testing.tRunner(0xc420242380, 0x651318)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc4203b5fd0 sp=0xc4203b5f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4203b5fd8 sp=0xc4203b5fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 21754 [GC worker (idle)]:
runtime.gopark(0x6507f8, 0xc42110a000, 0x6432b6, 0x10, 0x5af614, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420317750 sp=0xc420317720
runtime.gcBgMarkWorker(0xc420024c00)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4203177d8 sp=0xc420317750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4203177e0 sp=0xc4203177d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 23886 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc420236070, 0x6432b6, 0x10, 0x14, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4204fd750 sp=0xc4204fd720
runtime.gcBgMarkWorker(0xc420030c00)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4204fd7d8 sp=0xc4204fd750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4204fd7e0 sp=0xc4204fd7d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 21862 [GC worker (idle)]:
runtime.gopark(0x6507f8, 0xc42112a010, 0x6432b6, 0x10, 0xc4205f7f14, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4205f7f50 sp=0xc4205f7f20
runtime.gcBgMarkWorker(0xc420020000)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4205f7fd8 sp=0xc4205f7f50
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4205f7fe0 sp=0xc4205f7fd8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 22971 [chan send, 2 minutes]:
runtime.gopark(0x6509b8, 0xc42039e2f8, 0x640987, 0x9, 0x16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420058de0 sp=0xc420058db0
runtime.goparkunlock(0xc42039e2f8, 0x640987, 0x9, 0xc42001a016, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc420058e20 sp=0xc420058de0
runtime.chansend(0x5e9b00, 0xc42039e2a0, 0xc420058f4f, 0xc4200c2301, 0x4d2b18, 0xc4200c2380)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc420058ed0 sp=0xc420058e20
runtime.chansend1(0x5e9b00, 0xc42039e2a0, 0xc420058f4f)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc420058f10 sp=0xc420058ed0
testing.tRunner.func1(0xc4200c2380)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc420058f90 sp=0xc420058f10
testing.tRunner(0xc4200c2380, 0x6516f8)
	/tmp/workdir/go/src/testing/testing.go:681 +0x149 fp=0xc420058fd0 sp=0xc420058f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420058fd8 sp=0xc420058fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22626 [semacquire, 2 minutes]:
runtime.gopark(0x6509b8, 0x77a5a0, 0x641151, 0xa, 0xc42025ac19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc421127b78 sp=0xc421127b48
runtime.goparkunlock(0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc421127bb8 sp=0xc421127b78
runtime.semacquire(0x770cc4, 0x3)
	/tmp/workdir/go/src/runtime/sema.go:130 +0x24a fp=0xc421127c28 sp=0xc421127bb8
sync.runtime_SemacquireMutex(0x770cc4)
	/tmp/workdir/go/src/runtime/sema.go:62 +0x34 fp=0xc421127c48 sp=0xc421127c28
sync.(*Mutex).Lock(0x770cc0)
	/tmp/workdir/go/src/sync/mutex.go:87 +0x9d fp=0xc421127c80 sp=0xc421127c48
runtime_test.buildTestProg(0xc420242540, 0x6417d1, 0xb, 0xc4200165a0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:116 +0x83 fp=0xc421127e60 sp=0xc421127c80
runtime_test.TestCgoExternalThreadSIGPROF(0xc420242540)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:110 +0xbf fp=0xc421127f90 sp=0xc421127e60
testing.tRunner(0xc420242540, 0x651350)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc421127fd0 sp=0xc421127f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc421127fd8 sp=0xc421127fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22627 [semacquire, 2 minutes]:
runtime.gopark(0x6509b8, 0x77a5a0, 0x641151, 0xa, 0xc42025d419, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc421124b58 sp=0xc421124b28
runtime.goparkunlock(0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc421124b98 sp=0xc421124b58
runtime.semacquire(0x770cc4, 0x3)
	/tmp/workdir/go/src/runtime/sema.go:130 +0x24a fp=0xc421124c08 sp=0xc421124b98
sync.runtime_SemacquireMutex(0x770cc4)
	/tmp/workdir/go/src/runtime/sema.go:62 +0x34 fp=0xc421124c28 sp=0xc421124c08
sync.(*Mutex).Lock(0x770cc0)
	/tmp/workdir/go/src/sync/mutex.go:87 +0x9d fp=0xc421124c60 sp=0xc421124c28
runtime_test.buildTestProg(0xc420242620, 0x6417d1, 0xb, 0xc4200160e0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:116 +0x83 fp=0xc421124e40 sp=0xc421124c60
runtime_test.TestCgoExternalThreadSignal(0xc420242620)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:133 +0xc2 fp=0xc421124f90 sp=0xc421124e40
testing.tRunner(0xc420242620, 0x651358)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc421124fd0 sp=0xc421124f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc421124fd8 sp=0xc421124fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22629 [semacquire, 2 minutes]:
runtime.gopark(0x6509b8, 0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc421123aa8 sp=0xc421123a78
runtime.goparkunlock(0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc421123ae8 sp=0xc421123aa8
runtime.semacquire(0x770cc4, 0x3)
	/tmp/workdir/go/src/runtime/sema.go:130 +0x24a fp=0xc421123b58 sp=0xc421123ae8
sync.runtime_SemacquireMutex(0x770cc4)
	/tmp/workdir/go/src/runtime/sema.go:62 +0x34 fp=0xc421123b78 sp=0xc421123b58
sync.(*Mutex).Lock(0x770cc0)
	/tmp/workdir/go/src/sync/mutex.go:87 +0x9d fp=0xc421123bb0 sp=0xc421123b78
runtime_test.buildTestProg(0xc4202427e0, 0x6417d1, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:116 +0x83 fp=0xc421123d90 sp=0xc421123bb0
runtime_test.runTestProg(0xc4202427e0, 0x6417d1, 0xb, 0x643988, 0x11, 0xc420106598, 0xc42025df60)
	/tmp/workdir/go/src/runtime/crash_test.go:68 +0x89 fp=0xc421123ef8 sp=0xc421123d90
runtime_test.TestCgoExecSignalMask(0xc4202427e0)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:168 +0x7b fp=0xc421123f90 sp=0xc421123ef8
testing.tRunner(0xc4202427e0, 0x651340)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc421123fd0 sp=0xc421123f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc421123fd8 sp=0xc421123fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22630 [semacquire, 2 minutes]:
runtime.gopark(0x6509b8, 0x77a5a0, 0x641151, 0xa, 0xc421122e19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc421122aa8 sp=0xc421122a78
runtime.goparkunlock(0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc421122ae8 sp=0xc421122aa8
runtime.semacquire(0x770cc4, 0x3)
	/tmp/workdir/go/src/runtime/sema.go:130 +0x24a fp=0xc421122b58 sp=0xc421122ae8
sync.runtime_SemacquireMutex(0x770cc4)
	/tmp/workdir/go/src/runtime/sema.go:62 +0x34 fp=0xc421122b78 sp=0xc421122b58
sync.(*Mutex).Lock(0x770cc0)
	/tmp/workdir/go/src/sync/mutex.go:87 +0x9d fp=0xc421122bb0 sp=0xc421122b78
runtime_test.buildTestProg(0xc4202428c0, 0x6417d1, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:116 +0x83 fp=0xc421122d90 sp=0xc421122bb0
runtime_test.runTestProg(0xc4202428c0, 0x6417d1, 0xb, 0x6413a6, 0xb, 0xc420106618, 0xc420256f60)
	/tmp/workdir/go/src/runtime/crash_test.go:68 +0x89 fp=0xc421122ef8 sp=0xc421122d90
runtime_test.TestEnsureDropM(0xc4202428c0)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:182 +0x7b fp=0xc421122f90 sp=0xc421122ef8
testing.tRunner(0xc4202428c0, 0x651498)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc421122fd0 sp=0xc421122f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc421122fd8 sp=0xc421122fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22631 [chan send, 2 minutes]:
runtime.gopark(0x6509b8, 0xc420328fb8, 0x640987, 0x9, 0x16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42005fde0 sp=0xc42005fdb0
runtime.goparkunlock(0xc420328fb8, 0x640987, 0x9, 0xc42001c616, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc42005fe20 sp=0xc42005fde0
runtime.chansend(0x5e9b00, 0xc420328f60, 0xc42005ff4f, 0xc420242901, 0x4d2b18, 0xc4202429a0)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc42005fed0 sp=0xc42005fe20
runtime.chansend1(0x5e9b00, 0xc420328f60, 0xc42005ff4f)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc42005ff10 sp=0xc42005fed0
testing.tRunner.func1(0xc4202429a0)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc42005ff90 sp=0xc42005ff10
testing.tRunner(0xc4202429a0, 0x651320)
	/tmp/workdir/go/src/testing/testing.go:681 +0x149 fp=0xc42005ffd0 sp=0xc42005ff90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42005ffd8 sp=0xc42005ffd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22632 [semacquire, 2 minutes]:
runtime.gopark(0x6509b8, 0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420190aa8 sp=0xc420190a78
runtime.goparkunlock(0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc420190ae8 sp=0xc420190aa8
runtime.semacquire(0x770cc4, 0x3)
	/tmp/workdir/go/src/runtime/sema.go:130 +0x24a fp=0xc420190b58 sp=0xc420190ae8
sync.runtime_SemacquireMutex(0x770cc4)
	/tmp/workdir/go/src/runtime/sema.go:62 +0x34 fp=0xc420190b78 sp=0xc420190b58
sync.(*Mutex).Lock(0x770cc0)
	/tmp/workdir/go/src/sync/mutex.go:87 +0x9d fp=0xc420190bb0 sp=0xc420190b78
runtime_test.buildTestProg(0xc420242a80, 0x6417d1, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:116 +0x83 fp=0xc420190d90 sp=0xc420190bb0
runtime_test.runTestProg(0xc420242a80, 0x6417d1, 0xb, 0x643256, 0x10, 0xc4201067d8, 0xc420257760)
	/tmp/workdir/go/src/runtime/crash_test.go:68 +0x89 fp=0xc420190ef8 sp=0xc420190d90
runtime_test.TestCgoPanicDeadlock(0xc420242a80)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:235 +0x7b fp=0xc420190f90 sp=0xc420190ef8
testing.tRunner(0xc420242a80, 0x651360)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc420190fd0 sp=0xc420190f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420190fd8 sp=0xc420190fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22633 [semacquire, 2 minutes]:
runtime.gopark(0x6509b8, 0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42018faa8 sp=0xc42018fa78
runtime.goparkunlock(0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc42018fae8 sp=0xc42018faa8
runtime.semacquire(0x770cc4, 0x3)
	/tmp/workdir/go/src/runtime/sema.go:130 +0x24a fp=0xc42018fb58 sp=0xc42018fae8
sync.runtime_SemacquireMutex(0x770cc4)
	/tmp/workdir/go/src/runtime/sema.go:62 +0x34 fp=0xc42018fb78 sp=0xc42018fb58
sync.(*Mutex).Lock(0x770cc0)
	/tmp/workdir/go/src/sync/mutex.go:87 +0x9d fp=0xc42018fbb0 sp=0xc42018fb78
runtime_test.buildTestProg(0xc420242b60, 0x6417d1, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:116 +0x83 fp=0xc42018fd90 sp=0xc42018fbb0
runtime_test.runTestProg(0xc420242b60, 0x6417d1, 0xb, 0x642c7d, 0xf, 0xc420106858, 0xc420257f60)
	/tmp/workdir/go/src/runtime/crash_test.go:68 +0x89 fp=0xc42018fef8 sp=0xc42018fd90
runtime_test.TestCgoCCodeSIGPROF(0xc420242b60)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:244 +0x7b fp=0xc42018ff90 sp=0xc42018fef8
testing.tRunner(0xc420242b60, 0x651310)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc42018ffd0 sp=0xc42018ff90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42018ffd8 sp=0xc42018ffd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22634 [semacquire, 2 minutes]:
runtime.gopark(0x6509b8, 0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42018eab0 sp=0xc42018ea80
runtime.goparkunlock(0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc42018eaf0 sp=0xc42018eab0
runtime.semacquire(0x770cc4, 0x3)
	/tmp/workdir/go/src/runtime/sema.go:130 +0x24a fp=0xc42018eb60 sp=0xc42018eaf0
sync.runtime_SemacquireMutex(0x770cc4)
	/tmp/workdir/go/src/runtime/sema.go:62 +0x34 fp=0xc42018eb80 sp=0xc42018eb60
sync.(*Mutex).Lock(0x770cc0)
	/tmp/workdir/go/src/sync/mutex.go:87 +0x9d fp=0xc42018ebb8 sp=0xc42018eb80
runtime_test.buildTestProg(0xc420242c40, 0x6417d1, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:116 +0x83 fp=0xc42018ed98 sp=0xc42018ebb8
runtime_test.runTestProg(0xc420242c40, 0x6417d1, 0xb, 0x6424b0, 0xe, 0xc420258760, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:68 +0x89 fp=0xc42018ef00 sp=0xc42018ed98
runtime_test.TestCgoCrashTraceback(0xc420242c40)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:256 +0x7b fp=0xc42018ef90 sp=0xc42018ef00
testing.tRunner(0xc420242c40, 0x651330)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc42018efd0 sp=0xc42018ef90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42018efd8 sp=0xc42018efd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22635 [semacquire, 2 minutes]:
runtime.gopark(0x6509b8, 0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42018daa8 sp=0xc42018da78
runtime.goparkunlock(0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc42018dae8 sp=0xc42018daa8
runtime.semacquire(0x770cc4, 0x3)
	/tmp/workdir/go/src/runtime/sema.go:130 +0x24a fp=0xc42018db58 sp=0xc42018dae8
sync.runtime_SemacquireMutex(0x770cc4)
	/tmp/workdir/go/src/runtime/sema.go:62 +0x34 fp=0xc42018db78 sp=0xc42018db58
sync.(*Mutex).Lock(0x770cc0)
	/tmp/workdir/go/src/sync/mutex.go:87 +0x9d fp=0xc42018dbb0 sp=0xc42018db78
runtime_test.buildTestProg(0xc420242d20, 0x6417d1, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:116 +0x83 fp=0xc42018dd90 sp=0xc42018dbb0
runtime_test.runTestProg(0xc420242d20, 0x6417d1, 0xb, 0x6434d6, 0x10, 0xc420106958, 0xc420259760)
	/tmp/workdir/go/src/runtime/crash_test.go:68 +0x89 fp=0xc42018def8 sp=0xc42018dd90
runtime_test.TestCgoTracebackContext(0xc420242d20)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:266 +0x7b fp=0xc42018df90 sp=0xc42018def8
testing.tRunner(0xc420242d20, 0x651390)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc42018dfd0 sp=0xc42018df90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42018dfd8 sp=0xc42018dfd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22636 [semacquire, 2 minutes]:
runtime.gopark(0x6509b8, 0x77a5a0, 0x641151, 0xa, 0xc42005eb19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42005ea68 sp=0xc42005ea38
runtime.goparkunlock(0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc42005eaa8 sp=0xc42005ea68
runtime.semacquire(0x770cc4, 0x3)
	/tmp/workdir/go/src/runtime/sema.go:130 +0x24a fp=0xc42005eb18 sp=0xc42005eaa8
sync.runtime_SemacquireMutex(0x770cc4)
	/tmp/workdir/go/src/runtime/sema.go:62 +0x34 fp=0xc42005eb38 sp=0xc42005eb18
sync.(*Mutex).Lock(0x770cc0)
	/tmp/workdir/go/src/sync/mutex.go:87 +0x9d fp=0xc42005eb70 sp=0xc42005eb38
runtime_test.buildTestProg(0xc420242e00, 0x6417d1, 0xb, 0xc420d94030, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:116 +0x83 fp=0xc42005ed50 sp=0xc42005eb70
runtime_test.testCgoPprof(0xc420242e00, 0x0, 0x0, 0x640106, 0x8)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:280 +0xd5 fp=0xc42005ef58 sp=0xc42005ed50
runtime_test.TestCgoPprof(0xc420242e00)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:318 +0x52 fp=0xc42005ef90 sp=0xc42005ef58
testing.tRunner(0xc420242e00, 0x651380)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc42005efd0 sp=0xc42005ef90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42005efd8 sp=0xc42005efd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22637 [syscall]:
syscall.Syscall6(0xf7, 0x1, 0x1905, 0xc420059918, 0x1000004, 0x0, 0x0, 0x7f62e6cf9960, 0x0, 0xc42000c300)
	/tmp/workdir/go/src/syscall/asm_linux_amd64.s:44 +0x5 fp=0xc4200598c0 sp=0xc4200598b8
os.(*Process).blockUntilWaitable(0xc4200d63c0, 0xc42050a000, 0xc42000c360, 0x5)
	/tmp/workdir/go/src/os/wait_waitid.go:28 +0xa5 fp=0xc4200599c8 sp=0xc4200598c0
os.(*Process).wait(0xc4200d63c0, 0x411230, 0xc42026e028, 0xc4202c6140)
	/tmp/workdir/go/src/os/exec_unix.go:22 +0x4d fp=0xc420059a58 sp=0xc4200599c8
os.(*Process).Wait(0xc4200d63c0, 0x0, 0x0, 0x650518)
	/tmp/workdir/go/src/os/exec.go:115 +0x2b fp=0xc420059a88 sp=0xc420059a58
os/exec.(*Cmd).Wait(0xc42050a000, 0x0, 0x0)
	/tmp/workdir/go/src/os/exec/exec.go:435 +0x62 fp=0xc420059b10 sp=0xc420059a88
os/exec.(*Cmd).Run(0xc42050a000, 0xc4200ce000, 0xc4204e2240)
	/tmp/workdir/go/src/os/exec/exec.go:280 +0x5c fp=0xc420059b38 sp=0xc420059b10
os/exec.(*Cmd).CombinedOutput(0xc42050a000, 0xc42050a000, 0x9, 0x6417d1, 0xb, 0xc42000c180)
	/tmp/workdir/go/src/os/exec/exec.go:495 +0x127 fp=0xc420059b70 sp=0xc420059b38
runtime_test.buildTestProg(0xc420242ee0, 0x6417d1, 0xb, 0xc420d94020, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:142 +0x40a fp=0xc420059d50 sp=0xc420059b70
runtime_test.testCgoPprof(0xc420242ee0, 0x64777f, 0x19, 0x640106, 0x8)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:280 +0xd5 fp=0xc420059f58 sp=0xc420059d50
runtime_test.TestCgoPprofPIE(0xc420242ee0)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:322 +0x55 fp=0xc420059f90 sp=0xc420059f58
testing.tRunner(0xc420242ee0, 0x651368)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc420059fd0 sp=0xc420059f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420059fd8 sp=0xc420059fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22638 [chan send]:
runtime.gopark(0x6509b8, 0xc420329798, 0x640987, 0x9, 0xc42007ab16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42005ade0 sp=0xc42005adb0
runtime.goparkunlock(0xc420329798, 0x640987, 0x9, 0xc42001c616, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc42005ae20 sp=0xc42005ade0
runtime.chansend(0x5e9b00, 0xc420329740, 0xc42005af4f, 0xc420242f01, 0x4d2b18, 0xc420242fc0)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc42005aed0 sp=0xc42005ae20
runtime.chansend1(0x5e9b00, 0xc420329740, 0xc42005af4f)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc42005af10 sp=0xc42005aed0
testing.tRunner.func1(0xc420242fc0)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc42005af90 sp=0xc42005af10
testing.tRunner(0xc420242fc0, 0x651378)
	/tmp/workdir/go/src/testing/testing.go:681 +0x149 fp=0xc42005afd0 sp=0xc42005af90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42005afd8 sp=0xc42005afd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22639 [chan send]:
runtime.gopark(0x6509b8, 0xc4203298b8, 0x640987, 0x9, 0xc4202d6116, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420229de0 sp=0xc420229db0
runtime.goparkunlock(0xc4203298b8, 0x640987, 0x9, 0xc42001b316, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc420229e20 sp=0xc420229de0
runtime.chansend(0x5e9b00, 0xc420329860, 0xc420229f4f, 0xc420243001, 0x4d2b18, 0xc4202430a0)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc420229ed0 sp=0xc420229e20
runtime.chansend1(0x5e9b00, 0xc420329860, 0xc420229f4f)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc420229f10 sp=0xc420229ed0
testing.tRunner.func1(0xc4202430a0)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc420229f90 sp=0xc420229f10
testing.tRunner(0xc4202430a0, 0x651370)
	/tmp/workdir/go/src/testing/testing.go:681 +0x149 fp=0xc420229fd0 sp=0xc420229f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420229fd8 sp=0xc420229fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22641 [chan send, 2 minutes]:
runtime.gopark(0x6509b8, 0xc420329af8, 0x640987, 0x9, 0x16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420193bc0 sp=0xc420193b90
runtime.goparkunlock(0xc420329af8, 0x640987, 0x9, 0xc420020016, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc420193c00 sp=0xc420193bc0
runtime.chansend(0x5e9b00, 0xc420329aa0, 0xc420d9dd2f, 0xc420243201, 0x4d2b18, 0xc420243260)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc420193cb0 sp=0xc420193c00
runtime.chansend1(0x5e9b00, 0xc420329aa0, 0xc420d9dd2f)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc420193cf0 sp=0xc420193cb0
testing.tRunner.func1(0xc420243260)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc420193d70 sp=0xc420193cf0
runtime.call32(0x0, 0x651d90, 0xc420106bf0, 0x800000008)
	/tmp/workdir/go/src/runtime/asm_amd64.s:501 +0x48 fp=0xc420193da0 sp=0xc420193d70
runtime.Goexit()
	/tmp/workdir/go/src/runtime/panic.go:366 +0x153 fp=0xc420193de0 sp=0xc420193da0
testing.(*common).SkipNow(0xc420243260)
	/tmp/workdir/go/src/testing/testing.go:574 +0x39 fp=0xc420193df8 sp=0xc420193de0
testing.(*common).Skip(0xc420243260, 0xc42005bf20, 0x1, 0x1)
	/tmp/workdir/go/src/testing/testing.go:555 +0x6f fp=0xc420193e30 sp=0xc420193df8
runtime_test.TestRaceSignal(0xc420243260)
	/tmp/workdir/go/src/runtime/crash_cgo_test.go:372 +0x570 fp=0xc420193f90 sp=0xc420193e30
testing.tRunner(0xc420243260, 0x651888)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc420193fd0 sp=0xc420193f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420193fd8 sp=0xc420193fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22716 [semacquire, 2 minutes]:
runtime.gopark(0x6509b8, 0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc421125aa8 sp=0xc421125a78
runtime.goparkunlock(0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc421125ae8 sp=0xc421125aa8
runtime.semacquire(0x770cc4, 0x3)
	/tmp/workdir/go/src/runtime/sema.go:130 +0x24a fp=0xc421125b58 sp=0xc421125ae8
sync.runtime_SemacquireMutex(0x770cc4)
	/tmp/workdir/go/src/runtime/sema.go:62 +0x34 fp=0xc421125b78 sp=0xc421125b58
sync.(*Mutex).Lock(0x770cc0)
	/tmp/workdir/go/src/sync/mutex.go:87 +0x9d fp=0xc421125bb0 sp=0xc421125b78
runtime_test.buildTestProg(0xc4204aeee0, 0x6417dc, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:116 +0x83 fp=0xc421125d90 sp=0xc421125bb0
runtime_test.runTestProg(0xc4204aeee0, 0x6417dc, 0xb, 0x642cf5, 0xf, 0xc420106058, 0xc4205f5760)
	/tmp/workdir/go/src/runtime/crash_test.go:68 +0x89 fp=0xc421125ef8 sp=0xc421125d90
runtime_test.TestNetpollDeadlock(0xc4204aeee0)
	/tmp/workdir/go/src/runtime/crash_test.go:405 +0x7b fp=0xc421125f90 sp=0xc421125ef8
testing.tRunner(0xc4204aeee0, 0x651748)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc421125fd0 sp=0xc421125f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc421125fd8 sp=0xc421125fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22717 [semacquire, 2 minutes]:
runtime.gopark(0x6509b8, 0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4211269b8 sp=0xc421126988
runtime.goparkunlock(0x77a5a0, 0x641151, 0xa, 0x19, 0x4)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc4211269f8 sp=0xc4211269b8
runtime.semacquire(0x770cc4, 0x3)
	/tmp/workdir/go/src/runtime/sema.go:130 +0x24a fp=0xc421126a68 sp=0xc4211269f8
sync.runtime_SemacquireMutex(0x770cc4)
	/tmp/workdir/go/src/runtime/sema.go:62 +0x34 fp=0xc421126a88 sp=0xc421126a68
sync.(*Mutex).Lock(0x770cc0)
	/tmp/workdir/go/src/sync/mutex.go:87 +0x9d fp=0xc421126ac0 sp=0xc421126a88
runtime_test.buildTestProg(0xc4204aefc0, 0x6404ce, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/crash_test.go:116 +0x83 fp=0xc421126ca0 sp=0xc421126ac0
runtime_test.runTestProg(0xc4204aefc0, 0x6404ce, 0x8, 0x6425ba, 0xe, 0xc42050c400, 0x7f62e6cfb580)
	/tmp/workdir/go/src/runtime/crash_test.go:68 +0x89 fp=0xc421126e08 sp=0xc421126ca0
runtime_test.TestPanicTraceback(0xc4204aefc0)
	/tmp/workdir/go/src/runtime/crash_test.go:414 +0x7e fp=0xc421126f90 sp=0xc421126e08
testing.tRunner(0xc4204aefc0, 0x6517f8)
	/tmp/workdir/go/src/testing/testing.go:679 +0x13f fp=0xc421126fd0 sp=0xc421126f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc421126fd8 sp=0xc421126fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22739 [chan send, 2 minutes]:
runtime.gopark(0x6509b8, 0xc420362538, 0x640987, 0x9, 0x16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420222de0 sp=0xc420222db0
runtime.goparkunlock(0xc420362538, 0x640987, 0x9, 0xc42001c616, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc420222e20 sp=0xc420222de0
runtime.chansend(0x5e9b00, 0xc4203624e0, 0xc420222f4f, 0xc42061a201, 0x4d2b18, 0xc42061a2a0)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc420222ed0 sp=0xc420222e20
runtime.chansend1(0x5e9b00, 0xc4203624e0, 0xc420222f4f)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc420222f10 sp=0xc420222ed0
testing.tRunner.func1(0xc42061a2a0)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc420222f90 sp=0xc420222f10
testing.tRunner(0xc42061a2a0, 0x651448)
	/tmp/workdir/go/src/testing/testing.go:681 +0x149 fp=0xc420222fd0 sp=0xc420222f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420222fd8 sp=0xc420222fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 22919 [chan send, 2 minutes]:
runtime.gopark(0x6509b8, 0xc4203280b8, 0x640987, 0x9, 0x16, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420224de0 sp=0xc420224db0
runtime.goparkunlock(0xc4203280b8, 0x640987, 0x9, 0xc420020016, 0x3)
	/tmp/workdir/go/src/runtime/proc.go:267 +0x5e fp=0xc420224e20 sp=0xc420224de0
runtime.chansend(0x5e9b00, 0xc420328060, 0xc420224f4f, 0xc4204ae001, 0x4d2b18, 0xc4204ae000)
	/tmp/workdir/go/src/runtime/chan.go:229 +0x2dd fp=0xc420224ed0 sp=0xc420224e20
runtime.chansend1(0x5e9b00, 0xc420328060, 0xc420224f4f)
	/tmp/workdir/go/src/runtime/chan.go:113 +0x4d fp=0xc420224f10 sp=0xc420224ed0
testing.tRunner.func1(0xc4204ae000)
	/tmp/workdir/go/src/testing/testing.go:674 +0x208 fp=0xc420224f90 sp=0xc420224f10
testing.tRunner(0xc4204ae000, 0x651440)
	/tmp/workdir/go/src/testing/testing.go:681 +0x149 fp=0xc420224fd0 sp=0xc420224f90
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420224fd8 sp=0xc420224fd0
created by testing.(*T).Run
	/tmp/workdir/go/src/testing/testing.go:716 +0x2c7

goroutine 28856 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc4204e0a10, 0x6432b6, 0x10, 0x5fd214, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4204ff750 sp=0xc4204ff720
runtime.gcBgMarkWorker(0xc420038000)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4204ff7d8 sp=0xc4204ff750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4204ff7e0 sp=0xc4204ff7d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 23689 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc4200e8000, 0x6432b6, 0x10, 0x14, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420611f50 sp=0xc420611f20
runtime.gcBgMarkWorker(0xc42002ac00)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc420611fd8 sp=0xc420611f50
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420611fe0 sp=0xc420611fd8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 28148 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc4201467e0, 0x6432b6, 0x10, 0x14, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420324750 sp=0xc420324720
runtime.gcBgMarkWorker(0xc420035900)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4203247d8 sp=0xc420324750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4203247e0 sp=0xc4203247d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 28010 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc4206a85b0, 0x6432b6, 0x10, 0x14, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4206bdf50 sp=0xc4206bdf20
runtime.gcBgMarkWorker(0xc42003a600)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4206bdfd8 sp=0xc4206bdf50
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4206bdfe0 sp=0xc4206bdfd8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 24747 [GC worker (idle)]:
runtime.gopark(0x6507f8, 0xc420c32000, 0x6432b6, 0x10, 0x5fd214, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4201e5f50 sp=0xc4201e5f20
runtime.gcBgMarkWorker(0xc420021300)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4201e5fd8 sp=0xc4201e5f50
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4201e5fe0 sp=0xc4201e5fd8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 28228 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc42041d270, 0x6432b6, 0x10, 0x14, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420258750 sp=0xc420258720
runtime.gcBgMarkWorker(0xc420036c00)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4202587d8 sp=0xc420258750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4202587e0 sp=0xc4202587d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 26703 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc420236570, 0x6432b6, 0x10, 0x14, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420046750 sp=0xc420046720
runtime.gcBgMarkWorker(0xc42003f300)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4200467d8 sp=0xc420046750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4200467e0 sp=0xc4200467d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 21668 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc4200e4000, 0x6432b6, 0x10, 0x14, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420681f50 sp=0xc420681f20
runtime.gcBgMarkWorker(0xc420028600)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc420681fd8 sp=0xc420681f50
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420681fe0 sp=0xc420681fd8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 22140 [GC worker (idle)]:
runtime.gopark(0x6507f8, 0xc4205c4020, 0x6432b6, 0x10, 0xc4200c2614, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420130750 sp=0xc420130720
runtime.gcBgMarkWorker(0xc42001b300)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4201307d8 sp=0xc420130750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4201307e0 sp=0xc4201307d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 22740 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc4200165e0, 0x6432b6, 0x10, 0x14, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4210ff750 sp=0xc4210ff720
runtime.gcBgMarkWorker(0xc42002d300)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4210ff7d8 sp=0xc4210ff750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4210ff7e0 sp=0xc4210ff7d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 22360 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc421116000, 0x6432b6, 0x10, 0x14, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42034a750 sp=0xc42034a720
runtime.gcBgMarkWorker(0xc420026000)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc42034a7d8 sp=0xc42034a750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42034a7e0 sp=0xc42034a7d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 28011 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc4206a85c0, 0x6432b6, 0x10, 0x5fd214, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42013f750 sp=0xc42013f720
runtime.gcBgMarkWorker(0xc42003b900)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc42013f7d8 sp=0xc42013f750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42013f7e0 sp=0xc42013f7d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 28071 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc42112aab0, 0x6432b6, 0x10, 0x14, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4201cff50 sp=0xc4201cff20
runtime.gcBgMarkWorker(0xc420032000)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4201cffd8 sp=0xc4201cff50
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4201cffe0 sp=0xc4201cffd8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 25058 [GC worker (idle)]:
runtime.gopark(0x6507f8, 0xc420d94040, 0x6432b6, 0x10, 0x5fd214, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42012f750 sp=0xc42012f720
runtime.gcBgMarkWorker(0xc420022600)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc42012f7d8 sp=0xc42012f750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42012f7e0 sp=0xc42012f7d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 25157 [GC worker (idle)]:
runtime.gopark(0x6507f8, 0xc42013c080, 0x6432b6, 0x10, 0x5fd214, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420ebaf50 sp=0xc420ebaf20
runtime.gcBgMarkWorker(0xc420027300)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc420ebafd8 sp=0xc420ebaf50
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420ebafe0 sp=0xc420ebafd8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 24931 [GC worker (idle)]:
runtime.gopark(0x6507f8, 0xc42013c040, 0x6432b6, 0x10, 0x5fd214, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4210fd750 sp=0xc4210fd720
runtime.gcBgMarkWorker(0xc42001ec00)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4210fd7d8 sp=0xc4210fd750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4210fd7e0 sp=0xc4210fd7d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 29071 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc4204e5100, 0x6432b6, 0x10, 0x5fd214, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4210fc750 sp=0xc4210fc720
runtime.gcBgMarkWorker(0xc420039300)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4210fc7d8 sp=0xc4210fc750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4210fc7e0 sp=0xc4210fc7d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 25028 [GC worker (idle)]:
runtime.gopark(0x6507f8, 0xc42013c060, 0x6432b6, 0x10, 0x5fd214, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc42034b750 sp=0xc42034b720
runtime.gcBgMarkWorker(0xc420023900)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc42034b7d8 sp=0xc42034b750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42034b7e0 sp=0xc42034b7d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 21768 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc4200f2000, 0x6432b6, 0x10, 0xc4201b1f14, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc4201b1f50 sp=0xc4201b1f20
runtime.gcBgMarkWorker(0xc42002e600)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4201b1fd8 sp=0xc4201b1f50
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4201b1fe0 sp=0xc4201b1fd8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 105257 [syscall]:
syscall.Syscall(0x0, 0xc, 0xc420376000, 0x200, 0x7f62e6cfa770, 0x19, 0xc420376000)
	/tmp/workdir/go/src/syscall/asm_linux_amd64.s:18 +0x5 fp=0xc420398cb0 sp=0xc420398ca8
syscall.read(0xc, 0xc420376000, 0x200, 0x200, 0xc420599400, 0x7f62e6cfa770, 0x0)
	/tmp/workdir/go/src/syscall/zsyscall_linux_amd64.go:783 +0x55 fp=0xc420398d08 sp=0xc420398cb0
syscall.Read(0xc, 0xc420376000, 0x200, 0x200, 0x42bdff, 0x6500e0, 0xc420398dc0)
	/tmp/workdir/go/src/syscall/syscall_unix.go:162 +0x49 fp=0xc420398d50 sp=0xc420398d08
os.(*File).read(0xc42026e010, 0xc420376000, 0x200, 0x200, 0x4c0ecb, 0x5ebac0, 0x200)
	/tmp/workdir/go/src/os/file_unix.go:165 +0x4f fp=0xc420398d98 sp=0xc420398d50
os.(*File).Read(0xc42026e010, 0xc420376000, 0x200, 0x200, 0xc420086d00, 0x5, 0xc420398e60)
	/tmp/workdir/go/src/os/file.go:101 +0x76 fp=0xc420398e00 sp=0xc420398d98
bytes.(*Buffer).ReadFrom(0xc4200ce000, 0x752cc0, 0xc42026e010, 0x7f62d723d8c8, 0xc4200ce000, 0xc420032001)
	/tmp/workdir/go/src/bytes/buffer.go:179 +0x160 fp=0xc420398e70 sp=0xc420398e00
io.copyBuffer(0x752680, 0xc4200ce000, 0x752cc0, 0xc42026e010, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/io/io.go:384 +0x2cb fp=0xc420398ed8 sp=0xc420398e70
io.Copy(0x752680, 0xc4200ce000, 0x752cc0, 0xc42026e010, 0xc420398f60, 0xc42032ad00, 0x470c54)
	/tmp/workdir/go/src/io/io.go:360 +0x68 fp=0xc420398f38 sp=0xc420398ed8
os/exec.(*Cmd).writerDescriptor.func1(0xc4200d6090, 0x1f2)
	/tmp/workdir/go/src/os/exec/exec.go:254 +0x4d fp=0xc420398f98 sp=0xc420398f38
os/exec.(*Cmd).Start.func1(0xc42050a000, 0xc42000c360)
	/tmp/workdir/go/src/os/exec/exec.go:371 +0x27 fp=0xc420398fd0 sp=0xc420398f98
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420398fd8 sp=0xc420398fd0
created by os/exec.(*Cmd).Start
	/tmp/workdir/go/src/os/exec/exec.go:372 +0x4e4

goroutine 28009 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc420419f70, 0x6432b6, 0x10, 0x14, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420398750 sp=0xc420398720
runtime.gcBgMarkWorker(0xc420034600)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc4203987d8 sp=0xc420398750
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc4203987e0 sp=0xc4203987d8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98

goroutine 26108 [GC worker (idle), 2 minutes]:
runtime.gopark(0x6507f8, 0xc420c322d0, 0x6432b6, 0x10, 0x14, 0x0)
	/tmp/workdir/go/src/runtime/proc.go:261 +0x13a fp=0xc420366f50 sp=0xc420366f20
runtime.gcBgMarkWorker(0xc42002f900)
	/tmp/workdir/go/src/runtime/mgc.go:1471 +0x138 fp=0xc420366fd8 sp=0xc420366f50
runtime.goexit()
	/tmp/workdir/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420366fe0 sp=0xc420366fd8
created by runtime.gcBgMarkStartWorkers
	/tmp/workdir/go/src/runtime/mgc.go:1392 +0x98
FAIL	runtime	180.055s
2016/11/23 22:41:49 Failed: exit status 1

Not sure if this is the runtime, the buildlet (signals), Kubernetes, GKE, etc.

/cc @ianlancetaylor
@bradfitz bradfitz added Builders x/build issues (builders, bots, dashboards) Testing An issue that has been verified to require only test changes, not just a test failure. labels Nov 24, 2016
@bradfitz bradfitz added this to the Go1.8 milestone Nov 24, 2016
@ianlancetaylor
Copy link
Contributor

I think the system is simply overloaded.

The test program is running the parallel tests. Many of the tests have completed and are sleeping in a channel send (at testing.go:674) to indicate that they have completed. The channel receive is at testing.go:655, but it waits for the tests in turn. A bunch of other tests are sleeping in the mutex lock at runtime/crash_test.go:116, which is waiting for a chance to build the test program, or, more likely, to confirm that the test program has already been built.

The only test that is actively running is TestCgoPprofPIE, but it isn't waiting for the test proper; it is waiting for the test program to be built. TestCgoPprofPIE builds the test program with a different option (-ldflags=-extldflags=-pie) and that test currently holds the lock for building the test program. It is waiting for the command go build -o testprogcgo_-ldflags=-extldflags=-pie -ldflags=-extldflags=-pie to complete. What that command completes, it will run the test program. Judging by the fact that many of the goroutines have been sleeping for 2 minutes, I would guess that the go build command has been running for two minutes.

So, basically, everything is working fine. There is no indication of failure anywhere. There is no actual hanging test. It's just taking too long to build and run the test programs.

@quentinmit quentinmit added the NeedsFix The path to resolution is known, but the work has not been done. label Nov 24, 2016
@ianlancetaylor
Copy link
Contributor

I don't think there is anything to fix here. An overloaded system will time out in the tests. So it goes.

Anybody have a different opinion?

@bradfitz
Copy link
Contributor Author

There is something to fix here: I need to fight GKE more so Kubernetes doesn't over-schedule.

Apparently GKE doesn't yet reserve CPU and RAM for itself on each node, so if you're near 100% CPU utilization, you can be over-subscribed and get OOM killed or fight for CPU, even if you set all your limits correctly.

There are some workarounds we can do in the meantime, before GKE fixes the scheduling bug properly.

@aclements aclements changed the title runtime: test timeouts / flakes (TestCgoPprofPIE, TestCgoExternalThreadSIGPROF) build: test timeouts / flakes (TestCgoPprofPIE, TestCgoExternalThreadSIGPROF) Nov 29, 2016
@bradfitz
Copy link
Contributor Author

GKE is fixing this properly. In the meantime I've unnecessarily increased the CPU & RAM requirements of our container builders to cause our utilization to go down, but with the side effect of giving GKE space for itself. Good enough for now.

@golang golang locked and limited conversation to collaborators Dec 12, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
None yet
Milestone
Go1.8
Development

No branches or pull requests
4 participants
@bradfitz @quentinmit @ianlancetaylor @gopherbot