IPFS and Gentoo Portage (distfiles)

[NiKiZe]

Just as #84 Gentoo could use the same concepts. Creating this issue to inform about and track progress.

Relevant Gentoo forum thread
Some info about Gentoo distfile mirrors; https://wiki.gentoo.org/wiki/Project:Infrastructure/Mirrors/Source
Don't know yet, but I expect everything to be 4-500GB of data.

Currently working on creating a gentoo distfiles mirror, using similar concepts to VictorBjelkholm/arch-mirror

Updates coming when I have succeed with initial sync and started testing.
Current WIP repo NiKiZe/Gentoo-distfiles-IPFS

[NiKiZe]

A few things that I have hit so far;

• Using ipfs add -w -r --nocopy --local ${DSTBASE}/* takes a long time for ~400GB of data, what is the best way to do this?, The best would be to have files being added during the rsync process, alternatively a flag that reuses existing hashes (--assume-clean-if-exists and would work with --nocopy) seems there is a few small files (timestamp files and symlinks) that does get updated, so those needs handling
• The generated directories are large and don't work, example:
  QmY6jEuqY3U5N9yxeBs6uoM63QnV2gjewXop9U4cQKauX8 has a distfiles directory with hash QmWvMmUjrH6neokv6RTqdgaYZSgxzr6hrjqaHosnDh6o7C which is 4821289 bytes in one block, I have republished that raw block as QmNzePsWN7kCF9fvbFBAnoVERWbCiVFjVEmr5EFqXFLtxL, Another example is QmfSAWwPTx5Kmre4bDF3zbkDuYXeNo9X7eTLvRUxyfr52B which can be found in QmPctnSoGCY8j19Lhxez3wqH2Ld55DzSqjnrbvNUuxoYgw. Using Sharding as described in Tips for adding large datasets into ipfs #212 might solve this, but still the client should give error (or at least warning) when this happens.
  I guess that is the size here that is the issue? But should go-ipfs allow to create so large blocks? (It would be better for it to throw an error until there is a fix) I get a feeling there already must be an issue created about this?, but can't find a good match (that is still open), created Adding large blocks/directories (4MiB+) should cause error kubo#5282
• If I would like to create a filelist for publish (to get actual file hashes) what is the best way, output from the add command is one alternative, but is there any way to re-display those hashes in a filestore other then rerunning a full add? (the idea is that it could be used as a file to hash mapping by clients) something similar to filestore ls --file-order | grep " 0$" but returning file hash instead of hash for first block. Also an recursive option to ipfs ls would be useful.
• Symlinks are not working, might not be major but still worth a mention. Also documented at Stop rewriting symlinks victorb/arch-mirror#1
• While running the above ipfs add the node seems to be unresponsive
• "Error: cannot manually publish while IPNS is mounted" Not possible to do ipfs name publish - this removes the possibility to run mirror, and use it on the same machine. (disable 'ipfs name publish' while /ipns is mounted kubo#964 (comment))

[lidel]

Hi, thank you for pushing this! 👍

• FYI go-ipfs 0.4.17-rc1 fixes some issues with sharding, give it a try!
  • it also ships with experiment called URLStore, which (if I understand it correctly) may be useful if you already have HTTP-based data repositories.
• Tips for adding large datasets into ipfs: Tips for adding large datasets into ipfs #212

[NiKiZe]

Please remember to read the below as things to make something awesome even better

FYI go-ipfs 0.4.17-rc1 fixes some issues with sharding, give it a try!

Still using 0.4.16 so it works with sharding locally, but the gateways are currently "broken" (ipfs/kubo#5270) - the main issue I had here was that ipfs add allows for creating a directory block that is larger then 4MiB and is then inaccessible by other nodes. This should probably cause an error instead.

Yes UrlStore seems interesting indeed, just need to hook it into existing sync process somehow.
Another thing that is interesting is if shards can be used/created on different servers but all ending up in one large browseable directory listing.

Most of #212 is followed so far, I'm still having major issues however with files that are deleted, or even more so symlinks that is modified (this is due to using filestore)

It's up and "running" on /ipns/QmescA7sGoc4yZEe3Gof7dYt2qkkxDEXQPT2z84MpjVu8o/
In total ~485GB of data.
Running add on the full set takes 1-2 hours (this is needed since rsync is used as the source)
ipfs filstore verify Have been running for 12h+ and are not done.
Either filestore needs an update here, or maybe there will be something available in ipfs files/mfs
(writeable fuse mount that rsync writes directly to, and all changes are published directly to the configured ipns)

One observation while exploring ipfs is that there is multiple commands that does almost the same thing, this is to an degree confusing, but even more so a time thief.

[Stebalien]

Still using 0.4.16 so it works with sharding locally, but the gateways are currently "broken" - the main issue I had here was that ipfs add allows for creating a directory block that is larger then 4MiB and is then inaccessible by other nodes. This should probably cause an error instead.

You are absolutely correct. Please file a bug in go-ipfs.

ipfs filstore verify Have been running for 12h+ and are not done.

Please file a bug in go-ipfs.

One observation while exploring ipfs is that there is multiple commands that does almost the same thing, this is to an degree confusing, but even more so a time thief.

That often happens due to backwards compatibility concerns. Eventually, we'd like to release a new command with an entirely new, thought out API (and make that the default for the 1.0 release). The current thinking is to make everything use something like "mfs". This should significantly reduce confusion as all files will get names and can be managed through a file system.

[NiKiZe]

Really thank you for your feedback!

ipfs filstore verify Have been running for 12h+ and are not done.

Please file a bug in go-ipfs.

It simply takes time because it is huge dataset (will try to collect actual figures), and I think that ipfs/kubo#4260 (comment) which includes --overwrite and rm mentions already is appropriate for this since we just need something smarter to handle changes in filestore.

Thinking about it right now I could use a ipfs filestore verify <filename> this would be called with all files that was modified in the last x hours. But that is also just a hack for the actual issue with keeping large filestores synced. (and might be a bad feature to have)

Ok update;
(time (ipfs filestore verify --local --file-order | grep -v ^ok)) is now around 1h, so that is acceptable, but still high when it is run in conjunction with add as well.

[Stebalien]

It simply takes time because it is huge dataset

I'd expect it to take time, but it shouldn't be slower than an add. Could you try ipfs/kubo#5286? Is that any faster? Note: It may actually be slower as I haven't benchmarked it.

[NiKiZe]

Started update at Thu Jul 26 06:57:20 CEST 2018
#rsync runs here
End: Thu Jul 26 06:58:15 CEST 2018 - spent on stage 0:00:55
#another rsync
Delete dl done: Thu Jul 26 06:58:16 CEST 2018 - spent on stage 0:00:01
ipfs filestore verify --local --file-order | grep -v ^ok
verify done: Thu Jul 26 07:49:53 CEST 2018 - spent on stage 0:51:43
ipfs repo gc
gc done: Thu Jul 26 07:50:14 CEST 2018 - spent on stage 0:00:11
ipfs add QmPrbM7rqrUVbf9Guhdhs2yo6WRAGrbE21N75RyttfyQWY done: Thu Jul 26 08:54:55 CEST 2018 - spent on stage 1:04:41

With verify --file-order this is on the same level in time as the add.
I think a big reason for it being slow without --file-order is BTRFS which is used on these disks. (it''s just horrible with random seeks)
Must also add that I have not done controlled tests here, just "random trying to get the mirror up and running"
I will try out that branch and see how it behaves, but I will need better testcase for that to be useful.

One weird thing IMHO, old versions of the the files, those that are changed, or are missing does not get removed by the verify command (logs of verify command is found at QmZ35fVbUUMoTcz5a24f17qHvguZVwBiZ5nxz3pUwnhRjq) when I run verify again I get the same output, isn't it supposed to remove those links?
This does also cause the daemon to spew

16:57:42.928 ERROR engine: tried to execute a task and errored fetching block: data in file did not match. gentoo-distfiles/releases/ia64/autobuilds/latest-install-ia64-minimal.txt offset 0 engine.go:141

and similar lines

[NiKiZe]

Just a quick note about time taken for verify ..
with 0.4.15 and running with --file-order it is done in around 56minutes (guestimating 70-120MB/s disk usage)
with 0.4.15 and running without --file-order I canceled after 2hours 30minutes (guestimating ~16-30MB/s disk usage)
after 6-to-7 repo migration I'm now running the branch commit 23f5cd4f0 done after 3hours 36minutes (seeing 25-33MB/s disk usage) Will leave it running over night.

But the whole reason to run verify was to clean out deleted files, and/or files that have changed contents (timstamps and symlinks) which it seems verify does not do, and I have missunderstood how to deal with this. (for this to be viable the ipfs filestore clean mentioned in ipfs/kubo#4260 (comment) is what we/I want)

[NiKiZe]

A smallish update.
Just using add with --nocopy for an rsyncd directory is currently not an option.
It among other things has the issue with not being able to list files and is to slow (with verify it takes at least 2 hours).
However there is mfs which does this quite nicely.

I have rewriten the sync script to loop over recently modified files, add them one by one with nocopy and then update relevant mfs nodes.
Next stage is to find all removed files, this is done by a separate rsync that gives a list of deleted files, and then a similar loop is done over these where files are removed from mfs.
NiKiZe/Gentoo-distfiles-IPFS@bc71758
With this I'm down to 2-3 minutes for doing a normal update of the repo mirror.

Since everything now is in mfs, it is easy to get filelists as well.
Now ipfs files ls just needs to get an --recursive option for that filelist to be easily created (but I can probably hack bash for that as well)