Skip to content

runtime: infinite loop under FreeBSD #4979

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
andybalholm opened this issue Mar 4, 2013 · 100 comments
Closed

runtime: infinite loop under FreeBSD #4979

andybalholm opened this issue Mar 4, 2013 · 100 comments
Labels
FrozenDueToAge
Milestone
Go1.1

Comments

@andybalholm
Copy link
Contributor 

Recently the proxy server that I am working on in Go started mysteriously freezing up
and using 100% or 200% CPU at irregular intervals. I have seen the problem only under
FreeBSD; I haven't gotten any complaints from people who are testing it on Linux. I saw
that it was consuming more system time than user time, so I could tell that it was doing
some syscalls; I decided to try to trace it with ktrace. The result is an endless
alternation of sched_yield and _umtx_op calls:

 21045 redwood  CALL  sched_yield
 21045 redwood  RET   sched_yield 0
 21045 redwood  CALL  _umtx_op(0xc210c8cc20,0x2,0x2,0,0)
 21045 redwood  RET   _umtx_op 0
 21045 redwood  CALL  sched_yield
 21045 redwood  RET   sched_yield 0
 21045 redwood  CALL  _umtx_op(0xc210c8cc20,0x2,0x2,0,0)
 21045 redwood  RET   _umtx_op 0

# uname -a
FreeBSD balholm.com 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #2 r244360M: Fri Dec 28
20:12:01 PST 2012     [email protected]:/usr/obj/usr/src/sys/GENERIC  amd64

$ go version
go version devel +677c9ac552ac Sun Mar 03 08:56:08 2013 +1100 darwin/amd64

(Yes, I am cross-compiling on a Mac.)

This appears to be a regression; I didn't have this problem till about a week ago.
@rsc
Copy link
Contributor

rsc commented Mar 5, 2013

Comment 1:

Please export GOTRACEBACK=2 in the environment before running your server. When it gets
into this mode, please try kill -ABRT. Perhaps that will be useful, perhaps not. 
In the ktrace, when it stops looping, does it start sleeping with nanosleep(2)?
This could be caused by the new scheduler but I don't quite know how.

Labels changed: added priority-later, removed priority-triage.

Status changed to Accepted.

@andybalholm
Copy link
Contributor Author

Comment 2:

OK. I'm setting GOTRACEBACK. I'll post a traceback when I get a chance.
I can't comment on what happens when the process stops looping, since it doesn't stop
(till it's killed). The ktrace dump from a few seconds of this loop is 5.5 million lines
exactly like the ones I posted.
My wild guess as to the cause would be some subtle difference in behavior between
FreeBSD's umtx and Linux's futex that could be hit by the new scheduler.

@rsc
Copy link
Contributor

rsc commented Mar 5, 2013

Comment 3:

Another thing you could do is attach gdb to the thread calling yield and
see where it is calling yield from.
There are only a few choices: proc.c, lock_futex.c, parfor.c, mgc0.c. Each
of those is a very different context, so that information would help narrow
down what might be going wrong.
Thanks.
Russ

@andybalholm
Copy link
Contributor Author

Comment 4:

Here's a stack trace (of a _umtx_op call) from GDB:
#0  runtime.sys_umtx_op ()
    at /usr/local/go/src/pkg/runtime/sys_freebsd_amd64.s:19
#1  0x000000000043ca62 in runtime.futexsleep (addr=void, val=void, ns=void)
    at /usr/local/go/src/pkg/runtime/thread_freebsd.c:56
#2  0x000000000042c089 in runtime.notesleep (n=void)
    at /usr/local/go/src/pkg/runtime/lock_futex.c:125
#3  0x0000000000435d7f in stopm () at /usr/local/go/src/pkg/runtime/proc.c:780
#4  0x0000000000436775 in findrunnable ()
    at /usr/local/go/src/pkg/runtime/proc.c:1050
#5  0x000000000043685d in schedule ()
    at /usr/local/go/src/pkg/runtime/proc.c:1072
#6  0x0000000000436a5c in park0 (gp=void)
    at /usr/local/go/src/pkg/runtime/proc.c:1120
#7  0x0000000000441f09 in runtime.mcall (fn=void)
    at /usr/local/go/src/pkg/runtime/asm_amd64.s:182
#8  0x000000c203a106e0 in ?? ()
#9  0x0000000000000000 in ?? ()

@andybalholm
Copy link
Contributor Author

Comment 5:

And here's a full stack trace from SIGABRT (attached).

Attachments:

  1. errors.log (50064 bytes)

@dvyukov
Copy link
Member

dvyukov commented Mar 6, 2013

Comment 6:

please do 'thread apply all bt' to get backtraces of all threads, it can be any of the
threads.

@andybalholm
Copy link
Contributor Author

Comment 7:

My latest crash was a little different, but clearly related.
This time it's stalling in the umtx call (0% CPU). GDB gives the following:
(gdb) thread apply all bt
Thread 1 (process 55918):
#0  runtime.sys_umtx_op ()
    at /usr/local/go/src/pkg/runtime/sys_freebsd_amd64.s:19
#1  0x000000000043ca62 in runtime.futexsleep (addr=void, val=void, ns=void)
    at /usr/local/go/src/pkg/runtime/thread_freebsd.c:56
#2  0x000000000042c089 in runtime.notesleep (n=void)
    at /usr/local/go/src/pkg/runtime/lock_futex.c:125
#3  0x00000000004355d0 in runtime.stoptheworld ()
    at /usr/local/go/src/pkg/runtime/proc.c:372
#4  0x00000000004307da in gc (args=void)
    at /usr/local/go/src/pkg/runtime/mgc0.c:1765
#5  0x0000000000441ff0 in ?? ()
    at /usr/local/go/src/pkg/runtime/asm_amd64.s:265
#6  0x0000000811162f8c in ?? ()
#7  0x0000000811162100 in ?? ()
#8  0x0000000811162fb8 in ?? ()
#9  0x0000000811162f60 in ?? ()
#10 0x000000000043072e in runtime.gc (force=void)
    at /usr/local/go/src/pkg/runtime/mgc0.c:1736
#11 0x0000000000000000 in ?? ()
ktrace gives a much shorter trace (this time I'm posting the complete trace):
 55918 redwood  CALL  _umtx_op(0xa6c3b8,0x2,0,0,0)
 55918 redwood  RET   _umtx_op -1 errno 4 Interrupted system call
 55918 redwood  CALL  _umtx_op(0xa6c3b8,0x2,0,0,0)
 55918 redwood  RET   _umtx_op -1 errno 4 Interrupted system call
 55918 redwood  CALL  _umtx_op(0xa6c3b8,0x2,0,0,0)
 55918 redwood  RET   _umtx_op -1 errno 4 Interrupted system call
 55918 redwood  PSIG  SIGTERM caught handler=0x4425a0 mask=0x0 code=0x10001
 55918 redwood  CALL  exit(0x2)
 55918 redwood  RET   _umtx_op -1 errno 4 Interrupted system call
 55918 redwood  RET   _umtx_op -1 errno 4 Interrupted system call
 55918 redwood  RET   _umtx_op -1 errno 4 Interrupted system call
 55918 redwood  RET   _umtx_op -1 errno 4 Interrupted system call
 55918 redwood  RET   _umtx_op -1 errno 4 Interrupted system call
 55918 redwood  RET   _umtx_op -1 errno 4 Interrupted system call
 55918 redwood  RET   _umtx_op -1 errno 4 Interrupted system call
 55918 redwood  RET   _umtx_op -1 errno 4 Interrupted system call
 55918 redwood  RET   _umtx_op -1 errno 4 Interrupted system call
 55918 redwood  RET   _umtx_op -1 errno 4 Interrupted system call
The interrupted system calls before SIGTERM are times I pressed control-C in gdb. Then I
sent the process SIGTERM to kill it. Do the lines after exit mean that 10 threads were
in calls to _umtx_op when I killed the process? But if so, why did thread apply all bt
only trace one thread?

@andybalholm
Copy link
Contributor Author

Comment 8:

For some reason gdb can only see one thread in the process. If I type 'info threads', it
only lists one. But this time I caught it in runtime.osyield:
(gdb) thread apply all bt
Thread 1 (process 56284):
#0  runtime.osyield () at /usr/local/go/src/pkg/runtime/sys_freebsd_amd64.s:250
#1  0x000000000042be9f in runtime.lock (l=void)
    at /usr/local/go/src/pkg/runtime/lock_futex.c:77
#2  0x0000000000426183 in sellock (sel=void) at /usr/local/go/src/pkg/runtime/chan.c:804
#3  0x0000000000426459 in selectgo (selp=void) at
/usr/local/go/src/pkg/runtime/chan.c:918
#4  0x0000000000426272 in runtime.selectgo (sel=void)
    at /usr/local/go/src/pkg/runtime/chan.c:841
#5  0x000000000047ab27 in net/http.(*persistConn).roundTrip (pc=0xc20d6bbb00, 
    req=0xc20e938690, resp=0xc20d6bbb00, err=...)
    at /usr/local/go/src/pkg/net/http/transport.go:784
#6  0x00000000004770bc in net/http.(*Transport).RoundTrip (t=0xa34520
<main.transport>, 
    req=0xc210214b60, resp=0x735708, err=...)
    at /usr/local/go/src/pkg/net/http/transport.go:173
#7  0x000000000040bb99 in main.(*retryTransport).RoundTrip (t=0xa34520
<main.transport>, 
    req=0xc210214b60, resp=0xb, err=...)
    at /Users/andy/Programs/Go/src/code.google.com/p/redwood-filter/proxy.go:241
#8  0x000000000040ac47 in main.proxyHandler.ServeHTTP (h=..., w=..., r=0xc210214b60)
    at /Users/andy/Programs/Go/src/code.google.com/p/redwood-filter/proxy.go:121
#9  0x000000000041fdf4 in main.(*proxyHandler).ServeHTTP (this=0xc2099bbba0, w=..., 
    r=0xc210214b60)
#10 0x00000000004716dc in net/http.serverHandler.ServeHTTP (sh=..., rw=...,
req=0xc210214b60)
    at /usr/local/go/src/pkg/net/http/server.go:1291
#11 0x000000000046f9a5 in net/http.(*conn).serve (c=0xc20e910700)
    at /usr/local/go/src/pkg/net/http/server.go:873
#12 0x0000000000436b40 in ?? () at /usr/local/go/src/pkg/runtime/proc.c:1144
#13 0x000000c20e910700 in ?? ()
#14 0x0000000000000000 in ?? ()

@andybalholm
Copy link
Contributor Author

Comment 9:

About a second before it went into its loop this time, I got the following line in my
log file:
2013-03-06 14:35:41,10.1.10.203,ignore,%v(PANIC=runtime error: invalid memory address or
nil pointer dereference),GET,200,application/pdf,0,"","","",""
The item that was replaced by a panic message was a URL (which I had just used to
download a PDF). Is the garbage collector freeing objects that it shouldn't?

@rsc
Copy link
Contributor

rsc commented Mar 8, 2013

Comment 10:

It is certainly possible that the garbage collector is doing bad things.
The gdb traces don't have the amount of information I was hoping for.
Perhaps the next thing to do is to look for every call to osyield or usleep
in the runtime and insert a different print statement before each one
(saying the name of the function being called, the length of time for usleep,
and something about where the call is).
Then the infinite loop becomes an infinite loop that prints a lot, and from the prints
we might understand what is looping.
One common reason for a loop like this is that usleep is not actually sleeping. However,
I don't see anything wrong with the implementation. The prints should tell us more.

@andybalholm
Copy link
Contributor Author

Comment 11:

OK. I've added the printfs. Now we just have to wait for a crash.
Right now I'm getting many, many copies of this line in the log:
calling usleep(10000) from sysmon
I've got this going to my 1TB drive, so hopefully we'll find something out before I run
out of disk space.

@rsc
Copy link
Contributor

rsc commented Mar 8, 2013

Comment 12:

Feel free to change the sysmon loop to have a larger max, like 1000*1000
instead of 10*1000.
That will slow those down (and if it doesn't, that's data!).

@dvyukov
Copy link
Member

dvyukov commented Mar 8, 2013

Comment 13:

#7 looks like another reincarnation of the bug fixed by
https://golang.org/cl/7424054

@dvyukov
Copy link
Member

dvyukov commented Mar 8, 2013

Comment 14:

#8 looks legal
but if completely hangs there, it suggests memory corruption -- chan mutex state is
overwritten and so nobody can lock it
there was some GC-related memory corruption fixed some time ago
do you run with GOMAXPROCS>1? if so, there can be some data races that lead to memory
corruption. Try running the server with under the race detector.

@andybalholm
Copy link
Contributor Author

Comment 15:

Memory corruption sounds like a real possibility. This morning I had a crash (program
exited with a panic); I'm attaching the tail of the log (including the stack traces) in
case it's helpful.
So I've been having all three major styles of crashes: freezes at 0% CPU, freezes at
100% or more CPU, and panics.
I'll try running it with the race detector today.

Attachments:

  1. errors.log (138794 bytes)

@dvyukov
Copy link
Member

dvyukov commented Mar 8, 2013

Comment 16:

[signal 0xa code=0x3 addr=0x44252f pc=0x44252f]
It's SIGBUS. I am curious why addr==pc. Is it a bug in freebsd signal handling code? Why
it does not print actual failing address?
Russ, can it be some issue with new closures? Perhaps it jumps on invalid pc...

@dvyukov
Copy link
Member

dvyukov commented Mar 8, 2013

Comment 17:

Both crashes are in memmove.  Can it allocate more than 2GB? because there is an open
bug for it:
https://golang.org/cl/7474047/

@rsc
Copy link
Contributor

rsc commented Mar 12, 2013

Comment 18:

[The time for maybe has passed.]

Labels changed: removed go1.1maybe.

@andybalholm
Copy link
Contributor Author

Comment 19:

This appears to be a FreeBSD kernel bug (now fixed). I tested Go versions dating back to
November, and still got the crashes. But when I updated my FreeBSD base system to the
tip of the 9-STABLE branch, the crashes seem to have gone away. I've been crash-free for
two days now.
From when the crashes started, I suspect the problem was in the ZFS module; they started
shortly after I added a ZFS disk to my system. Why a ZFS problem would crash a process
that wasn't even accessing a ZFS disk, I don't know; but it looks pretty suspicious.

@andybalholm
Copy link
Contributor Author

Comment 20:

Nope, it's not fixed (although crashes are rarer). I'll have to keep watching it for a
while and see if I can find a pattern.

@andybalholm
Copy link
Contributor Author

Comment 21:

The crashes only occur when the system is under some other load besides my Go program.
The worst kind of load seems to be accessing a ZFS disk. (ZFS uses a lot of kernel
memory.) When I changed my backup disk from ZFS to UFS, the crashes became rarer, but
they still happen occasionally.

@andybalholm
Copy link
Contributor Author

Comment 22:

Most of my crashes now seem to be nil pointers, send on closed channel, segfault, etc.
It seems that each crash is usually preceded by a panic that gets caught and logged.

@dvyukov
Copy link
Member

dvyukov commented Mar 29, 2013

Comment 23:

Have you tried to run it on other OSes?
What race detector says?

@dvyukov
Copy link
Member

dvyukov commented Mar 29, 2013

Comment 24:

A race of map would have the same symptoms.

@alberts
Copy link
Contributor

alberts commented Mar 29, 2013

Comment 25:

Just hit this on linux/amd64. Sounds like the same symptoms:
=== RUN TestInterp-118
Input: coverage.go
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x1 pc=0x406d48]
goroutine 6 [running]:
runtime.panic(0x63c600, 0xc200e89e90)
    /build/go.tip/go/src/pkg/runtime/panic.c:247 +0xb1
runtime.panicstring(0x80d0c8)
    /build/go.tip/go/src/pkg/runtime/panic.c:487 +0xa3
runtime.sigpanic()
    /build/go.tip/go/src/pkg/runtime/os_linux.c:236 +0x168
evacuate(0x5d8b00, 0xc2000f7b40, 0x55f)
    /build/go.tip/go/src/pkg/runtime/hashmap.c:-124 +0x168
grow_work(0x5d8b00, 0xc2000f7b40, 0x939)
    /build/go.tip/go/src/pkg/runtime/hashmap.c:-9 +0x83
hash_lookup(0x5d8b00, 0xc2000f7b40, 0x7fbef33c9878)
    /build/go.tip/go/src/pkg/runtime/hashmap.c:52 +0xc0
runtime.mapaccess1()
    /build/go.tip/go/src/pkg/runtime/hashmap.c:1119 +0x60
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2000dec40, 0xc2000e6e40,
0xc200cdfac0, 0x1, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:746 +0x5a
code.google.com/p/go.exp/ssa.(*Builder).cond(0xc2000e7380, 0xc2000dec40, 0xc2000e6e40,
0xc200cdfac0, 0xc200da35a0, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:317 +0x115
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).cond(0xc2000e7380, 0xc2000dec40, 0xc200100c00,
0xc200cdcdc0, 0xc200da3480, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:312 +0xe3
code.google.com/p/go.exp/ssa.(*Builder).cond(0xc2000e7380, 0xc2000dec40, 0xc2000e6c00,
0xc200cde780, 0xc200da3480, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:304 +0x3b7
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2000dec40, 0xc2000e6dc0,
0xc200cdfbc0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2381 +0x232c
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2000dec40,
0xc200ce7300, 0x4, 0x4, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2000dec40, 0xc2001650c0,
0xc200ceac30)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildFunction(0xc2000e7380, 0xc2000dec40)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2430 +0x251
code.google.com/p/go.exp/ssa.(*Builder).buildDecl(0xc2000e7380, 0xc2000f20e0,
0xc2000e6d80, 0xc200ceac90)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2719 +0x58a
code.google.com/p/go.exp/ssa.(*Builder).BuildPackage(0xc2000e7380, 0xc2000f20e0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2810 +0x7a5
code.google.com/p/go.exp/ssa.func·004(0xc2000f20e0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2739 +0x3a
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 1 [chan receive]:
runtime.park(0x40b300, 0xc2000e43b0, 0x80ddea)
    /build/go.tip/go/src/pkg/runtime/proc.c:1167 +0x64
runtime.chanrecv(0x5d80e0, 0xc2000e4360, 0x7fbef379ace0, 0x0, 0x0, ...)
    /build/go.tip/go/src/pkg/runtime/chan.c:366 +0x566
runtime.chanrecv1()
    /build/go.tip/go/src/pkg/runtime/chan.c:458 +0x38
testing.RunTests(0x6d8240, 0x8071d0, 0x1, 0x1, 0xc2000f2901, ...)
    /build/go.tip/go/src/pkg/testing/testing.go:427 +0x88e
testing.Main(0x6d8240, 0x8071d0, 0x1, 0x1, 0x810960, ...)
    /build/go.tip/go/src/pkg/testing/testing.go:358 +0x8a
main.main()
    code.google.com/p/go.exp/ssa/interp/_test/_testmain.go:43 +0x9a
runtime.main()
    /build/go.tip/go/src/pkg/runtime/proc.c:182 +0x92
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
goroutine 2 [syscall]:
runtime.entersyscallblock()
    /build/go.tip/go/src/pkg/runtime/proc.c:1324 +0x16e
runtime.MHeap_Scavenger()
    /build/go.tip/go/src/pkg/runtime/mheap.c:435 +0xee
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by runtime.main
    /build/go.tip/go/src/pkg/runtime/proc.c:165
goroutine 3 [semacquire]:
runtime.park(0x40b300, 0x81eb80, 0x806df0)
    /build/go.tip/go/src/pkg/runtime/proc.c:1167 +0x64
semacquireimpl(0xc200b3c368, 0x1)
    /build/go.tip/go/src/pkg/runtime/zsema_linux_amd64.c:113 +0x116
sync.runtime_Semacquire(0xc200b3c368)
    /build/go.tip/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*WaitGroup).Wait(0xc200d9e1c0)
    /build/go.tip/go/src/pkg/sync/waitgroup.go:107 +0xf2
code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages(0xc2000e7380)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2744 +0xf8
code.google.com/p/go.exp/ssa/interp_test.run(0xc2000f5090, 0xc2000bf550, 0x46, 0x6906b0,
0xb, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/interp/interp_test.go:168 +0x7eb
code.google.com/p/go.exp/ssa/interp_test.TestInterp(0xc2000f5090)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/interp/interp_test.go:188 +0x1a7
testing.tRunner(0xc2000f5090, 0x8071d0)
    /build/go.tip/go/src/pkg/testing/testing.go:346 +0x8a
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by testing.RunTests
    /build/go.tip/go/src/pkg/testing/testing.go:426 +0x86b
goroutine 4 [running]:
code.google.com/p/go.exp/ssa.func·004(0xc2000f2620)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 5 [running]:
code.google.com/p/go.exp/ssa.func·004(0xc2000e8850)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 7 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc2000e8e70)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 8 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc2000f2070)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 10 [running]:
code.google.com/p/go.exp/ssa.func·004(0xc200e5a8c0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 11 [running]:
code.google.com/p/go.exp/ssa.func·004(0xc2000e8000)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 12 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc2000f2690)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 13 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc2000f2ee0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 14 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc2000e87e0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 15 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc200e5a3f0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 16 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc200e5a460)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
goroutine 17 [runnable]:
code.google.com/p/go.exp/ssa.func·004(0xc200e5a850)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2738
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x1 pc=0x406d48]
goroutine 11 [running]:
runtime.panic(0x63c600, 0xc2015246e0)
    /build/go.tip/go/src/pkg/runtime/panic.c:247 +0xb1
runtime.panicstring(0x80d0c8)
    /build/go.tip/go/src/pkg/runtime/panic.c:487 +0xa3
runtime.sigpanic()
    /build/go.tip/go/src/pkg/runtime/os_linux.c:236 +0x168
----- stack segment boundary -----
evacuate(0x5d8b00, 0xc2000f7b40, 0x566)
    /build/go.tip/go/src/pkg/runtime/hashmap.c:-124 +0x168
grow_work(0x5d8b00, 0xc2000f7b40, 0x219)
    /build/go.tip/go/src/pkg/runtime/hashmap.c:-9 +0x83
hash_lookup(0x5d8b00, 0xc2000f7b40, 0x7fbef33b9220)
    /build/go.tip/go/src/pkg/runtime/hashmap.c:52 +0xc0
runtime.mapaccess1()
    /build/go.tip/go/src/pkg/runtime/hashmap.c:1119 +0x60
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2011ece00, 0xc2000e6e40,
0xc2000e6e80, 0xc2011d9100, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:746 +0x5a
code.google.com/p/go.exp/ssa.(*Builder).exprInPlace(0xc2000e7380, 0xc2011ece00,
0xc200c67b80, 0xc201524620, 0xc2000e6e40, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:739 +0x283
code.google.com/p/go.exp/ssa.(*Builder).assignStmt(0xc2000e7380, 0xc2011ece00,
0xc2000bb860, 0x1, 0x1, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1434 +0x43a
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2011ece00, 0xc2000e6f00,
0xc2000e6ec0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2257 +0x1e0a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2011ece00, 0xc2000e6dc0,
0xc2000e6fc0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2373 +0x2227
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2011ece00,
0xc2000bb8f0, 0x1, 0x1, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2011ece00, 0xc2001650c0,
0xc2000f7f60)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildDecl(0xc2000e7380, 0xc2000e8000,
0xc2000e6d80, 0xc2000f7f90)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2709 +0x3c3
code.google.com/p/go.exp/ssa.(*Builder).BuildPackage(0xc2000e7380, 0xc2000e8000)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2810 +0x7a5
code.google.com/p/go.exp/ssa.func·004(0xc2000e8000)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2739 +0x3a
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x1 pc=0x406d48]
goroutine 4 [running]:
runtime.panic(0x63c600, 0xc201524b70)
    /build/go.tip/go/src/pkg/runtime/panic.c:247 +0xb1
runtime.panicstring(0x80d0c8)
    /build/go.tip/go/src/pkg/runtime/panic.c:487 +0xa3
runtime.sigpanic()
    /build/go.tip/go/src/pkg/runtime/os_linux.c:236 +0x168
evacuate(0x5d8b00, 0xc2000f7b40, 0x56b)
    /build/go.tip/go/src/pkg/runtime/hashmap.c:-124 +0x168
grow_work(0x5d8b00, 0xc2000f7b40, 0x661)
    /build/go.tip/go/src/pkg/runtime/hashmap.c:-9 +0x83
hash_lookup(0x5d8b00, 0xc2000f7b40, 0x7fbef35378f0)
    /build/go.tip/go/src/pkg/runtime/hashmap.c:52 +0xc0
runtime.mapaccess1()
    /build/go.tip/go/src/pkg/runtime/hashmap.c:1119 +0x60
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc200420ee0, 0xc2000e6ac0,
0xc2005438a0, 0xc20033ff01, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:746 +0x5a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).addr(0xc2000e7380, 0xc200420ee0, 0xc200107080,
0xc20056fe70, 0xc20033ff00, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:682 +0x5e1
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc200420ee0, 0xc200107080,
0xc20056fe70, 0xc2011d7100, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:936 +0x185a
code.google.com/p/go.exp/ssa.(*Builder).exprInPlace(0xc2000e7380, 0xc200420ee0,
0xc200c67b80, 0xc201524b30, 0xc200107080, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:739 +0x283
code.google.com/p/go.exp/ssa.(*Builder).assignStmt(0xc2000e7380, 0xc200420ee0,
0xc2004aac80, 0x1, 0x1, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1434 +0x43a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc200420ee0, 0xc2000e6f00,
0xc200432c40)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2257 +0x1e0a
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc200420ee0,
0xc2003a3c00, 0x12, 0x20, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc200420ee0, 0xc2001650c0,
0xc20056c6c0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildFunction(0xc2000e7380, 0xc200420ee0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2430 +0x251
code.google.com/p/go.exp/ssa.(*Builder).buildDecl(0xc2000e7380, 0xc2000f2620,
0xc2000e6d80, 0xc20056c6f0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2719 +0x58a
code.google.com/p/go.exp/ssa.(*Builder).BuildPackage(0xc2000e7380, 0xc2000f2620)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2810 +0x7a5
code.google.com/p/go.exp/ssa.func·004(0xc2000f2620)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2739 +0x3a
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x1 pc=0x406d48]
goroutine 10 [running]:
runtime.panic(0x63c600, 0xc200b46470)
    /build/go.tip/go/src/pkg/runtime/panic.c:247 +0xb1
runtime.panicstring(0x80d0c8)
    /build/go.tip/go/src/pkg/runtime/panic.c:487 +0xa3
runtime.sigpanic()
    /build/go.tip/go/src/pkg/runtime/os_linux.c:236 +0x168
----- stack segment boundary -----
evacuate(0x5d8b00, 0xc2000f7b40, 0x5d5)
    /build/go.tip/go/src/pkg/runtime/hashmap.c:-124 +0x168
grow_work(0x5d8b00, 0xc2000f7b40, 0xf28)
    /build/go.tip/go/src/pkg/runtime/hashmap.c:-9 +0x83
hash_lookup(0x5d8b00, 0xc2000f7b40, 0x7fbef3379268)
    /build/go.tip/go/src/pkg/runtime/hashmap.c:52 +0xc0
runtime.mapaccess1()
    /build/go.tip/go/src/pkg/runtime/hashmap.c:1119 +0x60
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014cae00, 0xc2000e6ac0,
0xc20059e8a0, 0x1, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:746 +0x5a
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014cae00, 0xc2000e6c00,
0xc20054e360, 0xc2009352d0, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:837 +0xa3f
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014cae00, 0xc200165380,
0xc20080b500, 0xc2011e0500, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:861 +0x316e
code.google.com/p/go.exp/ssa.(*Builder).exprInPlace(0xc2000e7380, 0xc2014cae00,
0xc200c67b80, 0xc200b46420, 0xc200165380, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:739 +0x283
code.google.com/p/go.exp/ssa.(*Builder).assignStmt(0xc2000e7380, 0xc2014cae00,
0xc20094e360, 0x1, 0x1, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1434 +0x43a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cae00, 0xc2000e6f00,
0xc20080b540)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2257 +0x1e0a
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2014cae00,
0xc20059e900, 0x2, 0x2, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cae00, 0xc2001650c0,
0xc20054e390)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cae00, 0xc2000e6dc0,
0xc20080b580)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2383 +0x23c3
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2014cae00,
0xc20094e3d0, 0x1, 0x1, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cae00, 0xc2001650c0,
0xc20054e3c0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).forStmt(0xc2000e7380, 0xc2014cae00,
0xc20080b5c0, 0x0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1942 +0x243
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cae00, 0xc2001009c0,
0xc20080b5c0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2404 +0x48a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2014cae00,
0xc20080b600, 0x4, 0x4, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cae00, 0xc2001650c0,
0xc20054e3f0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildFunction(0xc2000e7380, 0xc2014cae00)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2430 +0x251
code.google.com/p/go.exp/ssa.(*Builder).lookup(0xc2000e7380, 0xc200e5a8c0, 0xc2000bf3c0,
0xc2003f2db0, 0xc2002a0000, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:239 +0x118
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014cba80, 0xc2000e6ac0,
0xc200976760, 0x0, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:874 +0x1d7
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).setCallFunc(0xc2000e7380, 0xc2014cba80,
0xc200806280, 0xc2011e16f8)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:969 +0xb0
code.google.com/p/go.exp/ssa.(*Builder).setCall(0xc2000e7380, 0xc2014cba80,
0xc200806280, 0xc2011e16f8)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1059 +0x5a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014cba80, 0xc2000e6e40,
0xc200806280, 0xc2011e0500, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:809 +0x12b5
code.google.com/p/go.exp/ssa.(*Builder).exprInPlace(0xc2000e7380, 0xc2014cba80,
0xc200c67b80, 0xc20026f1e0, 0xc2000e6e40, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:739 +0x283
code.google.com/p/go.exp/ssa.(*Builder).compLit(0xc2000e7380, 0xc2014cba80,
0xc200b40050, 0xc2011e0550, 0xc200806380, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1489 +0x1573
code.google.com/p/go.exp/ssa.(*Builder).addr(0xc2000e7380, 0xc2014cba80, 0xc2000e6cc0,
0xc200806380, 0x1, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:653 +0x347
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014cba80, 0xc200100c00,
0xc200976940, 0xc2011e0500, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:816 +0x778
code.google.com/p/go.exp/ssa.(*Builder).exprInPlace(0xc2000e7380, 0xc2014cba80,
0xc200c67b80, 0xc20151bf70, 0xc200100c00, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:739 +0x283
code.google.com/p/go.exp/ssa.(*Builder).assignStmt(0xc2000e7380, 0xc2014cba80,
0xc20047cfa0, 0x1, 0x1, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1434 +0x43a
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cba80, 0xc2000e6f00,
0xc2008063c0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2257 +0x1e0a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2014cba80,
0xc2009d4580, 0x7, 0x8, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014cba80, 0xc2001650c0,
0xc2009ff6f0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildFunction(0xc2000e7380, 0xc2014cba80)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2430 +0x251
code.google.com/p/go.exp/ssa.(*Builder).lookup(0xc2000e7380, 0xc200e5a8c0, 0xc2000bf3c0,
0xc200a17330, 0xc2002a0000, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:239 +0x118
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014ca540, 0xc2000e6ac0,
0xc200478fe0, 0xc20151b800, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:874 +0x1d7
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).setCallFunc(0xc2000e7380, 0xc2014ca540,
0xc200431580, 0xc2011e15d8)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:969 +0xb0
code.google.com/p/go.exp/ssa.(*Builder).setCall(0xc2000e7380, 0xc2014ca540,
0xc200431580, 0xc2011e15d8)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1059 +0x5a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014ca540, 0xc2000e6e40,
0xc200431580, 0xc2011e1301, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:809 +0x12b5
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014ca540, 0xc200100480,
0xc2005a60c0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2307 +0x1b26
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2014ca540,
0xc200431540, 0x4, 0x4, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014ca540, 0xc2001650c0,
0xc2005549f0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildFunction(0xc2000e7380, 0xc2014ca540)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2430 +0x251
code.google.com/p/go.exp/ssa.(*Builder).lookup(0xc2000e7380, 0xc200e5a8c0, 0xc2000bf3c0,
0xc2003f2c00, 0xc2002a0000, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:239 +0x118
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc2014c29a0, 0xc2000e6ac0,
0xc20056b140, 0xc200079200, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:874 +0x1d7
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).setCallFunc(0xc2000e7380, 0xc2014c29a0,
0xc200580000, 0xc2011e1278)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:969 +0xb0
code.google.com/p/go.exp/ssa.(*Builder).setCall(0xc2000e7380, 0xc2014c29a0,
0xc200580000, 0xc2011e1278)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1059 +0x5a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).exprN(0xc2000e7380, 0xc2014c29a0, 0xc2000e6e40,
0xc200580000, 0x1, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:402 +0x4b2
code.google.com/p/go.exp/ssa.(*Builder).assignStmt(0xc2000e7380, 0xc2014c29a0,
0xc200578fe0, 0x2, 0x2, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1450 +0x702
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014c29a0, 0xc2000e6f00,
0xc2005800c0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2257 +0x1e0a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014c29a0, 0xc2000e6dc0,
0xc200580340)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2373 +0x2227
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2014c29a0,
0xc20040a680, 0x1, 0x1, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014c29a0, 0xc2001650c0,
0xc200584c30)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014c29a0, 0xc2000e6dc0,
0xc200580380)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2383 +0x23c3
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc2014c29a0,
0xc200a13d80, 0x7, 0x8, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc2014c29a0, 0xc2001650c0,
0xc200576180)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildFunction(0xc2000e7380, 0xc2014c29a0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2430 +0x251
code.google.com/p/go.exp/ssa.(*Builder).lookup(0xc2000e7380, 0xc200e5a8c0, 0xc2000bf3c0,
0xc2003f2300, 0xc2002a0000, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:239 +0x118
code.google.com/p/go.exp/ssa.(*Builder).expr(0xc2000e7380, 0xc20143b8c0, 0xc2000e6ac0,
0xc200a10f60, 0x40ad00, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:874 +0x1d7
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).setCallFunc(0xc2000e7380, 0xc20143b8c0,
0xc200a0fc40, 0xc200da5ed8)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:969 +0xb0
code.google.com/p/go.exp/ssa.(*Builder).setCall(0xc2000e7380, 0xc20143b8c0,
0xc200a0fc40, 0xc200da5ed8)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:1059 +0x5a
----- stack segment boundary -----
code.google.com/p/go.exp/ssa.(*Builder).exprN(0xc2000e7380, 0xc20143b8c0, 0xc2000e6e40,
0xc200a0fc40, 0x1, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:402 +0x4b2
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc20143b8c0, 0xc200100480,
0xc200a14000)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2300 +0xfc7
code.google.com/p/go.exp/ssa.(*Builder).stmtList(0xc2000e7380, 0xc20143b8c0,
0xc200a059d0, 0x1, 0x1, ...)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:953 +0x72
code.google.com/p/go.exp/ssa.(*Builder).stmt(0xc2000e7380, 0xc20143b8c0, 0xc2001650c0,
0xc200a12660)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2369 +0x2123
code.google.com/p/go.exp/ssa.(*Builder).buildFunction(0xc2000e7380, 0xc20143b8c0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2430 +0x251
code.google.com/p/go.exp/ssa.(*Builder).buildDecl(0xc2000e7380, 0xc200e5a8c0,
0xc2000e6d80, 0xc200a126c0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2719 +0x58a
code.google.com/p/go.exp/ssa.(*Builder).BuildPackage(0xc2000e7380, 0xc200e5a8c0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2810 +0x7a5
code.google.com/p/go.exp/ssa.func·004(0xc200e5a8c0)
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2739 +0x3a
runtime.goexit()
    /build/go.tip/go/src/pkg/runtime/proc.c:1214
created by code.google.com/p/go.exp/ssa.(*Builder).BuildAllPackages
    /build/go.tip/gopath/src/code.google.com/p/go.exp/ssa/builder.go:2741 +0x157
FAIL    code.google.com/p/go.exp/ssa/interp 0.504s

@andybalholm
Copy link
Contributor Author

Comment 26:

Yes, it seems to run fine under Linux.
I'm running it with the race detector on MacOS now. No races found yet.

@andybalholm
Copy link
Contributor Author

Comment 27:

I ran it on my Mac with the race detector for several hours (6 minutes of CPU time), and
no races were detected.

@andybalholm
Copy link
Contributor Author

Comment 28:

It's been a long time since I've had an infinite loop (like in the issue description).
The crashes are basically all panics now.
I'm attaching a stack trace from one I had this morning. It's a nil pointer dereference
where there clearly can't be one if everything is working properly. The crash is
occurring in the line z[0] = x of the following function in math/big/nat.go:
func (z nat) setWord(x Word) nat {
    if x == 0 {
        return z.make(0)
    }
    z = z.make(1)
    z[0] = x
    return z
}
The previous line ensures that z has at least one element.

Attachments:

  1. crash Mar 30 0943.txt (22938 bytes)

@andybalholm
Copy link
Contributor Author

Comment 29:

Here's another stack trace.

Attachments:

  1. crash Mar 30 1850.txt (54970 bytes)

@andybalholm
Copy link
Contributor Author

Comment 30:

and another

Attachments:

  1. Crash Mar 30 2050.txt (27821 bytes)

@DanielMorsing
Copy link
Contributor

Comment 80:

Does the freebsd runtime have any of the mistakes that Dmitry fixed for the big GC bug?
There are a couple of places where a byte* is used where a void* would be more
appropriate, mostly in the code generated by defs_freebsd.go. I don't know if they can
cause a failure like this.
Andy, could try going into $GOROOT/src/pkg/runtime/mgc0.c and change the Debug constant
to 2, then do a run with the environment variable GOGC=1 ? It will be very slow but
should show us something.

@andybalholm
Copy link
Contributor Author

Comment 81:

Daniel,
This morning I had a crash in one of the zfreebsd files. I don't know if that would have
anything to do with the question you have about the runtime, but maybe.
I'll try running it with the settings you described.
Remy and Mikioh,
My code uses unsafe only under Linux. BSD has a simpler way to get the real destination
on redirected connections.

Attachments:

  1. crash Apr 11 0716.txt (48425 bytes)

@ianlancetaylor
Copy link
Contributor

Comment 82:

Dmitriy, in comment #69 you said you had this program running on GNU/Linux.  Have you
been able to replicate the problems there, or do they only arise on FreeBSD?

@andybalholm
Copy link
Contributor Author

Comment 83:

Linux doesn't seem like the best OS to try to replicate the problem. I have the program
running on 3 machines: linux/amd64, linux/386, and freebsd/amd64. The linux/386 one
isn't crashing, the linux/amd64 one crashes maybe once a week, and the freebsd/amd64 one
crashes about twice a day. The linux/amd64 one handles quite a bit more traffic than the
FreeBSD one.

@ianlancetaylor
Copy link
Contributor

Comment 84:

Andy, another thing you could try that might help is to edit the file
src/pkg/runtime/mgc0.c and set Debug = 1 and near the top of the file.  Then rebuild the
Go distribution, and then use that rebuilt distribution to rebuild your program.  Then
see what the next crash report looks like.

@andybalholm
Copy link
Contributor Author

Comment 85:

Daniel already has me running under Debug=2 and GOGC=1. It takes 12 minutes to load its
configuration files, my connection feels like dialup, and HTTPS doesn't work because it
times out, but we're getting lots of data. (It looks like it will be something like 3GB
per hour.)

@DanielMorsing
Copy link
Contributor

Comment 86:

Only now do I see that Debug=2 will spew a ton to stdout. Don't know if it's worth it to
keep it going.

@ianlancetaylor
Copy link
Contributor

Comment 87:

I managed to build and set up the program on a FreeBSD amd64 system.  I configured a
browser on my laptop to use that system as a proxy.  I do see that some web sites are
denied.  I ran several instances of the Go testsuite (cd test; go run run.go) in the
background to push up the load.
I've been poking at this for a while, but I have not yet managed to see a crash of the
redwood-filter program.  Oddly, I did see a crash in the Go testsuite.
How long do you usually have to wait until you see a crash?
Go testsuite crash:
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x4d0fb7]
goroutine 1633 [running]:
regexp/syntax.(*Regexp).MaxCap(0xc2011ba230, 0x1)
    /home/gopher/iant/go/src/pkg/regexp/syntax/regexp.go:297 +0x57
regexp/syntax.(*Regexp).MaxCap(0xc2011ba310, 0x1a)
    /home/gopher/iant/go/src/pkg/regexp/syntax/regexp.go:298 +0x74
regexp.compile(0xc2011b90e0, 0x1a, 0xd4, 0xc200078380, 0xc2011b90e0, ...)
    /home/gopher/iant/go/src/pkg/regexp/regexp.go:148 +0x84
regexp.Compile(0xc2011b90e0, 0x1a, 0xc200257680, 0xc20063da50, 0x5800fc, ...)
    /home/gopher/iant/go/src/pkg/regexp/regexp.go:109 +0x3b
regexp.MustCompile(0xc2011b90e0, 0x1a, 0x8107c4768)
    /home/gopher/iant/go/src/pkg/regexp/regexp.go:205 +0x38
main.(*test).wantedErrors(0xc2000ece00, 0xc200c9cce0, 0x17, 0x57aef0, 0x8, ...)
    /home/gopher/iant/go/test/run.go:785 +0x6c1
main.(*test).errorCheck(0xc2000ece00, 0xc2009ab000, 0x21d8c, 0x81093ab28, 0x2, ...)
    /home/gopher/iant/go/test/run.go:676 +0x617
main.(*test).run(0xc2000ece00)
    /home/gopher/iant/go/test/run.go:618 +0x2b4d
main.func·001()
    /home/gopher/iant/go/test/run.go:243 +0x27
created by main.runTests
    /home/gopher/iant/go/test/run.go:245 +0xa2
goroutine 1 [runnable]:
main.main()
    /home/gopher/iant/go/test/run.go:116 +0x38d
goroutine 3 [runnable]:
main.runTests()
    /home/gopher/iant/go/test/run.go:240 +0x36
created by main.startTest
    /home/gopher/iant/go/test/run.go:226 +0xc9
goroutine 4 [runnable]:
exit status 2

@andybalholm
Copy link
Contributor Author

Comment 88:

I see crashes once or twice a day.

@ianlancetaylor
Copy link
Contributor

Comment 89:

I see.  I didn't understand that the crashes were so rare.  This means that we don't
have a way to crash under controlled circumstances.  That is unfortunate.  I'm not sure
we are going to be able to solve this problem unless we can develop another test case.

@gopherbot
Copy link
Contributor

Comment 90 by dvyukov:

I was running 4 "GOGC=500 ./test/run" on freebsd/amd64 and get:
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x4d1017]
goroutine 1637 [running]:
runtime.panic(0x53f100, 0xc201563210)
/home/gopher/dvyukov/go/src/pkg/runtime/panic.c:247 +0xb1
runtime.panicstring(0x63e08b)
/home/gopher/dvyukov/go/src/pkg/runtime/panic.c:487 +0xa3
runtime.sigpanic()
/home/gopher/dvyukov/go/src/pkg/runtime/os_freebsd.c:188 +0x162
regexp/syntax.(*Regexp).MaxCap(0xc201569310, 0x0)
/home/gopher/dvyukov/go/src/pkg/regexp/syntax/regexp.go:297 +0x57
regexp/syntax.(*Regexp).MaxCap(0xc201569380, 0x0)
/home/gopher/dvyukov/go/src/pkg/regexp/syntax/regexp.go:298 +0x74
regexp/syntax.(*Regexp).MaxCap(0xc2015691c0, 0x1)
/home/gopher/dvyukov/go/src/pkg/regexp/syntax/regexp.go:298 +0x74
regexp/syntax.(*Regexp).MaxCap(0xc2015693f0, 0x0)
/home/gopher/dvyukov/go/src/pkg/regexp/syntax/regexp.go:298 +0x74
regexp/syntax.(*Regexp).MaxCap(0xc201569540, 0x1a)
/home/gopher/dvyukov/go/src/pkg/regexp/syntax/regexp.go:298 +0x74
regexp.compile(0xc201560460, 0x1a, 0xd4, 0xc200078380, 0xc201560460, ...)
/home/gopher/dvyukov/go/src/pkg/regexp/regexp.go:148 +0x84
regexp.Compile(0xc201560460, 0x1a, 0xc200abd340, 0xc2012d6c90, 0x58015c, ...)
/home/gopher/dvyukov/go/src/pkg/regexp/regexp.go:109 +0x3b
regexp.MustCompile(0xc201560460, 0x1a, 0x8106a4768)
/home/gopher/dvyukov/go/src/pkg/regexp/regexp.go:205 +0x38
main.(*test).wantedErrors(0xc20010bd20, 0xc201704cc0, 0x17, 0x57af50, 0x8, ...)
/home/gopher/dvyukov/go/test/run.go:785 +0x6c1
main.(*test).errorCheck(0xc20010bd20, 0xc200eaa000, 0x21d8c,
0x8106b5b28, 0x2, ...)
/home/gopher/dvyukov/go/test/run.go:676 +0x617
----- stack segment boundary -----
main.(*test).run(0xc20010bd20)
/home/gopher/dvyukov/go/test/run.go:618 +0x2b4d
main.func·001()
/home/gopher/dvyukov/go/test/run.go:243 +0x27
runtime.goexit()
/home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1223
created by main.runTests
/home/gopher/dvyukov/go/test/run.go:245 +0xa2
goroutine 1 [runnable]:
runtime.park(0x415cf0, 0xc20010cad0, 0x63f351)
/home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1175 +0x64
runtime.chanrecv(0x5034c0, 0xc20010ca80, 0x8106a9b80, 0x0, 0x0, ...)
/home/gopher/dvyukov/go/src/pkg/runtime/chan.c:397 +0x234
runtime.chanrecv1()
/home/gopher/dvyukov/go/src/pkg/runtime/chan.c:458 +0x38
main.main()
/home/gopher/dvyukov/go/test/run.go:116 +0x38d
runtime.main()
/home/gopher/dvyukov/go/src/pkg/runtime/proc.c:182 +0x92
runtime.goexit()
/home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1223
goroutine 2 [syscall]:
runtime.entersyscallblock()
/home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1333 +0x16e
runtime.MHeap_Scavenger()
/home/gopher/dvyukov/go/src/pkg/runtime/mheap.c:435 +0xee
runtime.goexit()
/home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1223
created by runtime.main
/home/gopher/dvyukov/go/src/pkg/runtime/proc.c:165
goroutine 3 [chan send]:
runtime.park(0x415cf0, 0xc200087410, 0x63f2de)
/home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1175 +0x64
runtime.chansend(0x5034c0, 0xc2000873c0, 0x8106b7fa0, 0x0, 0x402b36, ...)
/home/gopher/dvyukov/go/src/pkg/runtime/chan.c:261 +0x1d7
runtime.chansend1()
/home/gopher/dvyukov/go/src/pkg/runtime/chan.c:450 +0x4c
main.runTests()
/home/gopher/dvyukov/go/test/run.go:240 +0x36
runtime.goexit()
/home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1223
created by main.startTest
/home/gopher/dvyukov/go/test/run.go:226 +0xc9
goroutine 1649 [runnable]:
runtime.exitsyscall()
/home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1388 +0x119
syscall.Syscall()
/home/gopher/dvyukov/go/src/pkg/syscall/asm_freebsd_amd64.s:34 +0x5b
syscall.read(0x5, 0xc2000b8c00, 0x200, 0x200, 0x8, ...)
/home/gopher/dvyukov/go/src/pkg/syscall/zsyscall_freebsd_amd64.go:895 +0x70
syscall.Read(0x5, 0xc2000b8c00, 0x200, 0x200, 0x41dd04, ...)
/home/gopher/dvyukov/go/src/pkg/syscall/syscall_unix.go:132 +0x5a
os.(*File).read(0xc205ee9370, 0xc2000b8c00, 0x200, 0x200, 0x502640, ...)
/home/gopher/dvyukov/go/src/pkg/os/file_unix.go:174 +0x60
os.(*File).Read(0xc205ee9370, 0xc2000b8c00, 0x200, 0x200, 0x0, ...)
/home/gopher/dvyukov/go/src/pkg/os/file.go:95 +0x96
bytes.(*Buffer).ReadFrom(0xc20130f2a0, 0xc20007fba0, 0xc205ee9370,
0x0, 0x0, ...)
/home/gopher/dvyukov/go/src/pkg/bytes/buffer.go:169 +0x203
io.Copy(0xc20007fcf0, 0xc20130f2a0, 0xc20007fba0, 0xc205ee9370, 0x0, ...)
/home/gopher/dvyukov/go/src/pkg/io/io.go:331 +0xa3
os/exec.func·003(0x503520, 0xc20130f1c0)
/home/gopher/dvyukov/go/src/pkg/os/exec/exec.go:207 +0x60
os/exec.func·004(0xc201704ae0)
/home/gopher/dvyukov/go/src/pkg/os/exec/exec.go:276 +0x2a
runtime.goexit()
/home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1223
created by os/exec.(*Cmd).Start
/home/gopher/dvyukov/go/src/pkg/os/exec/exec.go:277 +0x732
goroutine 13 [finalizer wait]:
runtime.park(0x0, 0x0, 0x6400db)
/home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1175 +0x64
runfinq()
/home/gopher/dvyukov/go/src/pkg/runtime/mgc0.c:2182 +0x6d
runtime.goexit()
/home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1223
created by runtime.gc
/home/gopher/dvyukov/go/src/pkg/runtime/mgc0.c:1886
goroutine 1648 [runnable]:
runtime.exitsyscall()
/home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1388 +0x119
syscall.Syscall6()
/home/gopher/dvyukov/go/src/pkg/syscall/asm_freebsd_amd64.s:57 +0x61
syscall.wait4(0x33e0, 0xc205ee93d0, 0x0, 0xc201705360, 0x100000001, ...)
/home/gopher/dvyukov/go/src/pkg/syscall/zsyscall_freebsd_amd64.go:32 +0x7b
syscall.Wait4(0x33e0, 0x810ab1f54, 0x0, 0xc201705360, 0xc200ab8480, ...)
/home/gopher/dvyukov/go/src/pkg/syscall/syscall_bsd.go:137 +0x6c
os.(*Process).wait(0xc201704c60, 0x81092b110, 0x0, 0x0)
/home/gopher/dvyukov/go/src/pkg/os/exec_unix.go:22 +0xea
----- stack segment boundary -----
os.(*Process).Wait(0xc201704c60, 0xc200676b20, 0xe, 0xc201704aa0)
/home/gopher/dvyukov/go/src/pkg/os/doc.go:43 +0x25
os/exec.(*Cmd).Wait(0xc200abbdc0, 0x0, 0x0)
/home/gopher/dvyukov/go/src/pkg/os/exec/exec.go:310 +0x197
os/exec.(*Cmd).Run(0xc200abbdc0, 0xe, 0xc200ab8480)
/home/gopher/dvyukov/go/src/pkg/os/exec/exec.go:232 +0x52
main.func·002(0xc20129cd70, 0x5, 0x5, 0x5, 0xffffffffffffff01, ...)
/home/gopher/dvyukov/go/test/run.go:438 +0x18e
main.(*test).run(0xc20010be70)
/home/gopher/dvyukov/go/test/run.go:553 +0xd77
main.func·001()
/home/gopher/dvyukov/go/test/run.go:243 +0x27
runtime.goexit()
/home/gopher/dvyukov/go/src/pkg/runtime/proc.c:1223
created by main.runTests
/home/gopher/dvyukov/go/test/run.go:245 +0xa2

@andybalholm
Copy link
Contributor Author

Comment 91:

I was running four copies of while true; do time go run run.go; done 
and the OOM killer kicked in. My system had run out of swap space (4GB RAM + 1GB swap)!
It killed one of the invocations of 6g, but I don't know any way to tell what process it
was that was hogging that much memory. Now, with three of the run.go jobs still running,
I have 2GB free. So I suspect that it was another one of the transient errors that we're
chasing.
Besides the OOM, the messages I got so far from the test runs were:
run fixedbugs/bug207.go : wait : bad address
and
runoutput rotate0.go : incorrect output

@remyoudompheng
Copy link
Contributor

Comment 92:

I don't think you can run run.go 4 times in parallel without running out of memory.
Running index*.go and rotate*.go takes a lot of memory (>300MB) and having 16 of them in
parallel doesn't seem possible (assuming you have 4 cores). An OOM may cause the
compiler or linker to produce corrupt binaries.

@ianlancetaylor
Copy link
Contributor

Comment 93:

Andy, here is something else for you to try.  Update to tip.  Edit
src/pkg/runtime/mgc0.c.  Change the value of IgnorePreciseGC, near the top of the file,
from 0 to 1.  Rebuild go.  Use the rebuilt go to rebuild your program.  See if it
crashes.  This will help us understand whether the new precise GC in Go 1.1 is related
to this crash.  Thanks!

@andybalholm
Copy link
Contributor Author

Comment 94:

Well, I ran it since yesterday afternoon and only OOMed once. Probably that's when the
memory-hungry ones happened to run at the same time.
Make that twice�it just OOMed again.
As it was running overnight, it had a few more "wait: bad address" errors from files in
the fixedbugs directory.

@andybalholm
Copy link
Contributor Author

Comment 95:

OK. I'm running with conservative GC now. We'll see how it does.

@lexprfuncall
Copy link

Comment 96:

Andy, it turns out the change Ian referenced might not be sufficient to disable the
behavior that is causing the crashes.  Can you try patching CL 7569047?
Here is some background information: I am able to reproduce "wait: bad address" failures
(among others) at ToT.  These failures persist with IgnorePreciseGC set to 1.  However,
with CL 7569047 applied I have yet to observe a failure.

@andybalholm
Copy link
Contributor Author

Comment 97:

I'm running with 7569047 now.

@andybalholm
Copy link
Contributor Author

Comment 98:

Well, I got a crash again today, even with the patch.
(A caught panic, immediately followed by an uncaught one)

Attachments:

  1. crash Apr 13 1230.txt (56029 bytes)

@andybalholm
Copy link
Contributor Author

Comment 99:

The issue appears to have been fixed by
https://code.google.com/p/go/source/detail?r=0704ea89235b

@dvyukov
Copy link
Member

dvyukov commented Apr 19, 2013

Comment 100:

Status changed to Fixed.

@rsc rsc added this to the Go1.1 milestone Apr 14, 2015
@rsc rsc removed the go1.1 label Apr 14, 2015
@golang golang locked and limited conversation to collaborators Jun 24, 2016
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
Go1.1
Development

No branches or pull requests