ipfs add hangs and stalls with large folders

[entr0p1]

Version information:

go-ipfs version: 0.4.8-
Repo version: 5
System version: amd64/linux
Golang version: go1.8

Type:

Bug

Severity:

High

Description:

When adding files to the repository, it gets to a seemingly arbitrary point (around 8-10GB?) in to the file transfer, then the progress bar disappears and the job stalls. I usually pipe the output into a text file to get a file and hash list, but have tried without and still no luck I'm afraid.

The directory being added is 24.91GB in size, and contains 101143 objects.

The actual content being stored can be found with an explanation on this page: climate-mirror/datasets#333

The command I'm using is:

ipfs add -r -p -w (FOLDER NAME)>(FOLDER NAME)_filelist.txt

I was originally running golang installed from the Yum repositories, which turned out to be an old version (v1.6.3). I've since removed this one and installed straight from their site (running v1.8.1 now) which has significantly improved the performance of the ipfs add overall.

# go version
go version go1.8.1 linux/amd64

The machine is a Virtual Machine on Citrix XenServer, running Oracle Linux 7 and the datastore is on an NFS volume directly attached to the VM, mounted via /etc/fstab on boot. The VM has 2x vCPU cores pinned to it and 8GB of memory. I can provide further details on the physical hardware if it helps.

I'm running SELinux but it's not had any effect on the functionality since the start, as far as I can tell.

I installed IPFS using ipfs-update and am currently on the latest version (v0.4.8).

I've just tried stopping the ipfs daemon and will see if that improves, so far it's looking like I've gotten further than before. I'll provide an update when it completes with the result

[Kubuxu]

Try enabling sharding ipfs config --json Experimental.ShardingEnabled true which allows for adding big directories.

[Kubuxu]

Also see: ipfs/notes#212

[entr0p1]

Update: It failed at 99.6%

Thanks for the super fast response @Kubuxu, I will give that a crack now and let you know the results. Fingers crossed

[entr0p1]

Got all the way through and failed with this error:

ERROR commands/h: unexpected EOF client.go:247

Tried a second time and it made it 1% in and threw the same error. I've restarted the IPFS daemon and get the same result still.

Stopped the daemon and it seems to be running now, will report back soon.

[entr0p1]

It bombed out again, this time saying "Killed". Trying again.

I've noted since turning on the sharding it seems to really use the RAM; to the point where it's filling swap as well. Unsure if that's related but thought it might be worth mentioning. Will post the results of the second attempt.

[Kubuxu]

Interesting observation regarding sharding.

cc @whyrusleeping

[entr0p1]

Yeah, no good I'm afraid

 8.95 GB / 24.91 GB [======================>----------------------------------------]  
Killed

It seems that it fills the RAM, then swap (other processes on the system have crashed I noticed) and then finally bombs out itself.

Let me know if you guys need any info from me, I'll do my best to give logs and such

[entr0p1]

Here's some visual data on the memory usage and how CPU looks around the time it peaks.

[whyrusleeping]

Hrm... I'll take a look at this after breakfast...

[whyrusleeping]

@dojobel If you don't mind, could you try running this branch: #3888 ?
I set it to flush the cached directories

[entr0p1]

Not at all, I've had a go (heh) but I must be noobing something up..

# make install
can't load package: package github.com/ipfs/go-ipfs/cmd/ipfs: cannot find package "github.com/ipfs/go-ipfs/cmd/ipfs" in any of:
	/usr/local/go/src/github.com/ipfs/go-ipfs/cmd/ipfs (from $GOROOT)
	/opt/go/src/github.com/ipfs/go-ipfs/cmd/ipfs (from $GOPATH)
bin/check_go_version 1.7
bin/check_go_path /opt/go/src/github.com/go-ipfs 
go-ipfs must be built from within your $GOPATH directory.
expected within '/opt/go' but got '/opt/go/src/github.com/go-ipfs'
make: *** [check_go_path] Error 1

I've stashed my original executable with ipfs-update stash and what I've done to get the branch is:

git clone https://github.com/ipfs/go-ipfs.git
cd go-ipfs
git checkout fix/add-mem-growth-hack

I've tried a make install straight in that folder and it told me it needs to run from under the $GOPATH. I moved the cloned folder into /opt/go/src/github.com/go-ipfs and end up receiving the above error. Have I missed something?

[entr0p1]

Yep, it was me being a noob indeed - had to move the folder from "/opt/go/src/github.com/go-ipfs" to "/opt/go/src/github.com/ipfs/go-ipfs".

Have gotten it to make install successfully. I'm running an ipfs add now, will report back when it completes and let you know if that fix worked.

[entr0p1]

It's stalled at 2.98%, 764.11MB / 24.91GB done.

The IPFS daemon is using a lot of memory since the import and is also using a fair bit of CPU (probably normal).

# ps aux | grep ipfs
root     23647 78.1 77.2 6646036 6107444 ?     Ssl  15:55  48:31 /opt/go/bin/ipfs daemon --enable-gc=true --manage-fdlimit=true
root     32524  1.1  0.7 340280 60736 pts/8    Sl+  16:20   0:25 ipfs add -r -p -w ftp.ncdc.noaa.gov

I've had to restart it as it was on the cusp of using all the RAM and crashing other processes. It's leveled out after the restart

# ps aux | grep ipfs
root     14158  5.4  1.0 362152 81700 ?        Ssl  16:59   0:06 /opt/go/bin/ipfs daemon --enable-gc=true --manage-fdlimit=true

I'm going to try importing without the Daemon running again and will report back

[entr0p1]

Doesn't look much better, it's currently 41.45% (10.33GB / 24.91GB) into the ipfs add. Going to have to cancel again to protect other processes I'm afraid

Edit: adding graphs

[entr0p1]

Hi guys,

Just a quick update - I managed to get these problem folders to import by turning off sharding again. Unsure if the patch you did had anything to do with it, but something is definitely improved. Will be trying with some much larger folders over the coming days/weeks.