Replication

[davidar]

My new year's ipfs/archives resolution was to make it much easier for others to replicate the data we're collecting here, since that's the whole point of IPFS ;) . There's two issues here:

• Redundancy. Currently, many of the larger archives are only pinned to one or two machines being operated by the IPFS team. This is a very fragile situation, because sometimes bad things happen. We need content to be replicated over a larger number of nodes, preferably run by independent parties.
• Currently there's a lot of hashes spread through the issues in this repo, with no systematic way of accessing/pinning them. @eminence has already started addressing this in Archive webpage archive hub #26, but we need to extend this to all of the archives. In particular, we need to have a process in place for patching archives into a single tree, and point a domain like archives.ipfs.io to it. This way, all of the archives can be replicated simply by pinning the root hash.

Practically, we can't expect people to be pinning all of the archives (or even the entirety of a single large archive), so we need a system for (intelligently) distributing small pieces to different nodes. For example, if a user wants to donate X units of disk space, we need to be able to generate a list of hashes (with total size less than X) that are in the most need of being replicated (ie. those being provided by a small number of peers). @victorbjelkholm's PinCoop project might be able to help here?

CC: @jbenet

[d10r]

I would very much like to have the option to pin a specific project (or part of it).
Also I like the proposal to donate some resources and let the network decide what data specifically gets pinned, I would however combine those, e.g. let the user decide to dedicate 1 GB to OpenStreetMap.

Imo the user should further be able to monitor how much traffic is generated from that and have the possibility to throttle or pause distribution temporarily. This is however a feature which would probably make sense for ipfs in general, not only for pinned archives.

[davidar]

I knew I shouldn't've called this a resolution...