Implement `datastore`

[daviddias]

Almost missed blocks as there is no reference in the Spec (https://github.com/ipfs/specs/blob/fix-repo/repo/README.md). I'm guessing is that datastore is becoming the blocks as the transition form levelDB to flatfs happens.

@whyrusleeping @jbenet can I get some details on this?

To implement the fanout factor I'm going to use a blob-store that builds on top of fs-blob-store that knows how to do fanout (so that in the browser we don't have too)

[jbenet]

@diasdavid blocks is just the name we give to a serialized ipfs object. Raw data. You don't need it in your code necessarily.

Maybe this helps:

• a dag node is logically an object
• IPLD is the format for dag nodes, which implies serialization
• historically (from the ipfs paper) serialized objects were called blocks.
• in go-ipfs we have a blocks package that layers on top of the repo and datastore. It handles making keys just the hash of an object, and puting/getting it from a datastore (blockstore wraps a datastore, think of a blockstore as a "specific to ipfs" abstraction on top of datastore)
• func deserialize(block) object

There's a diagram somewhere, I'll look for it.

[jbenet]

(Ipld and multicodec are newer than blocks so some there's more overlap than we'd have if these abstractions were created from scratch together)

[jbenet]

@diasdavid does that make sense? lmk if not

[daviddias]

Trying to wrap my mind around what would be the lifecycle of an file when added to IPFS.

What data does datastore folder have then (currently and near future)? Blocks is making sense, but having a blocks folder and a datastore folder is not.

[jbenet]

What data does datastore folder have then (currently and near future)? Blocks is making sense, but having a blocks folder and a datastore folder is not.

conceptually, blocks/ is supposed to be inside datastore/ it's just that historically datastore/ was a leveldb instance, so blocks/ had to go outside. the fs-repo layout with leveldb in go is more historical than correct. we can do a migration to make it better. maybe we could:

• move datastore/ to leveldb/
• or, move datastore/ to datastore/leveldb and move blocks/ to datastore/blocks

[jbenet]

cc @whyrusleeping

[daviddias]

As talked on Sprint Meeting Dec 14, blocks will be 'datastore' and datastore with level db will be 'datastore-legacy'

[daviddias]

(probably going to ask the obvious) The objects stored on this datastore, are serialised versions of the protobufs, right?

Currently all the blocks have the .data extension, what is the migration plan for when we use IPLD + CBOR? Migrate and have .json objects, so we can both read from old repos and new repos at the same time?

Are we dropping protobufs soon enough that we can make the js impl only understand IPLD + CBOR?

[jbenet]

(probably going to ask the obvious) The objects stored on this datastore, are serialised versions of the protobufs, right?

Yes

Currently all the blocks have the .data extension, what is the migration plan for when we use IPLD + CBOR?

New objects are done with cbor. old objects still readable. see https://github.com/ipfs/go-ipld/tree/master/coding/pb

Migrate and have .json objects, so we can both read from old repos and new repos at the same time?

Turns out we will need to still have the protobuf implementation to read old objects. yeah annoying, but we'll get them on the wire from other people, or even if we get them in json, we'll need to be able to serialize with protobuf to hash and verify.

So we support protobuf in a "backwards compatible" kind of way. which is unfortunate, but not so bad with multicodec. read the discussions with @mildred over here:

• https://github.com/ipfs/go-ipld/issues
• https://github.com/ipfs/go-ipld/pulls
• important directive PR (not merged) Add @codec directive when decoding PB mdag instead of trying to detect it ipld/go-ipld-deprecated#13
• protobuf PR Make protobuf codec compatible with how links are represented in IPLD ipld/go-ipld-deprecated#11

Are we dropping protobufs soon enough that we can make the js impl only understand IPLD + CBOR?

No, we'll have to make it understand old objects. Sorry.

On the bright side, mafintosh wrote a very nice protobuf parser: https://github.com/mafintosh/protocol-buffers

---

I think we should use this whenever "breaking links" or "breaking old data" is a possibility:

• "Break links? THIS IS IPFS!!!!"

[jbenet]

that said, we could migrate the protobufs to .pb if useful. though the multicodec should give it

[daviddias]

With #20 we have a impl capable of reading/writing blocks into the datastore following the same folder structure go-ipfs does.