infrequent crashes in daemon mode on armv7

I am running ipfs on a raspberry pi 2 and the daemon seems to crash somewhere between 5 mins and 1hour after i start it with ipfs daemon.

the kernel version and architecture uname reports is Linux 3.18.11-v7+ armv7l

the head of the stack trace is as follow

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x76dc18dc
...

I do not know what of the rest of this is relevant, but here it is https://gist.github.com/optimusbrine/e29ed9360d2eaa81a390

[chriscool]

The following links look relevant:

https://groups.google.com/forum/#!topic/golang-nuts/8gszDBRZh_4
apache/cassandra-gocql-driver#330

It looks like the daemon might be creating too many threads or using too much memory.
ulimit -u could help increase the thread limit but it's not really a fix.

I was able to curtail it to some degree by lowering the default stack size with ulimit -s (i think anyhow, the default was 8mb, but 500kb was sufficient).

What structure in the program do all of these threads relate to? I was thinking it might be swarm peers, but on two seperate runs where the peer count plateued around 40, the thread count maxed at 115 and 272 respectively.

It might be more reasonable to be able to limit the amount of these thread related structures instead.

[jbenet]

It might be more reasonable to be able to limit the amount of these thread related structures instead.

how do we do this in Go? I think go governs this with its own internal scheduler, but we may be able to tune it.

cc @whyrusleeping

[whyrusleeping]

@optimusbrine could you try running the daemon with IPFS_REUSEPORT=false ?

@jbenet I just meant that if the usage of native threads is somehow semantically tied to some data structure, that there be an option provided to limit the usage of that structure. Like if it was tied to peers, an option to let you specify that you only want to connect to N peers at max. It shouldn't need to be go specific unless that relationship cannot be determined. Edit I don't know enough about why go would spawn a bunch of native threads to know if this makes sense in this context.

@whyrusleeping Yes that seems to be keeping the thread count at exactly 10.

[whyrusleeping]

@optimusbrine okay, so the extra threads are being caused by our poor use of epoll in our reuseport implementation. let me know how things go running it with that env var set like that.

[jbenet]

@whyrusleeping that reuseport crap keeps on giving! if only we could use the internal runtime scheduler... golang/go#9661 is so annoying. have better ides on using epoll?

[whyrusleeping]

kqueue > epoll. all my kernel experience is on freeBSD so i'm not actually that familiar with epoll.

@whyrusleeping 18 threads at 15 hours up. I think it was at around 16 3 hours in, so it seems pretty stable. Thanks for the help guys.

[whyrusleeping]

@optimusbrine thanks for the feedback, do let us know if you experience any connectivity issues.

[bcomnes]

@optimusbrine I'm running into the exact same thing. Can you clarify what you did to help improve the situation?

EDIT: Added Environment=IPFS_REUSEPORT=false to my [service] section of the systemd service in hopes to prevent crashes.

@bcomnes that should be all you need. Running top, then showing threads with Shift+h should not show any more than about 20 threads being spawned by the ipfs daemon. Before setting the IPFS_REUSEPORT=false it was spawning hundreds.

[bcomnes]

Worked great! Thanks for the info.

[leni536]

I had the same problem on the following system:

$ uname -a
Linux debian-eee 3.16.0-4-686-pae #1 SMP Debian 3.16.7-ckt9-2 (2015-04-13) i686 GNU/Linux

Using IPFS_REUSEPORT=false worked fine for me too.

[whyrusleeping]

@leni536 thanks for the info. what kind of machine was that? and what is your normal file descriptor limit?

I'm glad disabling reuseport allows you to use ipfs, but it also limits the effectiveness of NAT traversal. We should look at fixing our implementation (if anyone knows about epoll stuff, we want your help!)