The `dag` API - One API to manipulate all the IPLD Format objects

[daviddias]

We need to come up with an API to manipulate IPLD Format objects.

Currently, go-ipfs master ships with a dag API that offers get and put methods, it doesn't expose yet a dag resolve API.

For reference, here are the help texts:

» ipfs dag --help
USAGE
  ipfs dag - Interact with ipld dag objects.

SYNOPSIS
  ipfs dag

DESCRIPTION

  'ipfs dag' is used for creating and manipulating dag objects.

  This subcommand is currently an experimental feature, but it is intended
  to deprecate and replace the existing 'ipfs object' command moving forward.


SUBCOMMANDS
  ipfs dag get <cid>         - Get a dag node from ipfs.
  ipfs dag put <object data> - Add a dag node to ipfs.

  Use 'ipfs dag <subcmd> --help' for more information about each command.

» ipfs dag get --help
USAGE
  ipfs dag get <cid> - Get a dag node from ipfs.

SYNOPSIS
  ipfs dag get [--] <cid>

ARGUMENTS

  <cid> - The cid of the object to get

DESCRIPTION

  'ipfs dag get' fetches a dag node from ipfs and prints it out in the specifed format.

» ipfs dag put --help
USAGE
  ipfs dag put <object data> - Add a dag node to ipfs.

SYNOPSIS
  ipfs dag put [--format=<format> | -f] [--input-enc=<input-enc>] [--] <object data>

ARGUMENTS

  <object data> - The object to put

OPTIONS

  -f,        --format string - Format that the object will be added as. Default: cbor.
  --input-enc         string - Format that the input object will be. Default: json.

DESCRIPTION

  'ipfs dag put' accepts input from a file or stdin and parses it
  into an object of the specified format.

While, in js-ipfs, we have a pretty much straight out copy of this API, defined as an interface at: https://github.com/ipfs/interface-ipfs-core/tree/master/API/dag and an resolve API exposed by the IPLD Resolver that goes as (simple as) follows:

.resolve(cid, path, callback)

Note: this function is capable of resolving through different formats.

We need to complete the dag API definition, taking into account the following issues

Current shortcomings

It is impossible to ensure that the right type is returned when using a non-strict IPLD Format

To help understand this issue, let's define as a strict IPLD Format something like dag-pb, eth-block, git-block and other Merkle Data Structures that have been predefined and that its format follows a structure. non-strict IPLD Formats are (so far, we have one main case) data structures like dag-cbor, which have no definition when it comes to the keys and the value types of its data.

When resolving through non-strict IPLD Formats, the entity that requests for a .resolve to happen can't tell which is the type of the value that is going to be returned. This problem can be somewhat mitigated due to some languages support to type inference (or others that don't have type systems at all), it is an unavoidable problem when we have to pass a node through a transport like http. Let's illustrate the issue

//Imagine we have an object that is stored in cbor that looks like
{
  name: 'fancy-music.mp3',
  data: new Buffer(<bytes of fancy-music.mp3>)
}

// This object can be serialized and deserialize as many times we want,
// since cbor has a 1:1 mapping with JSON, however, if a http client 
// requests this object, it will have to be JSON.stringify'ed in order to 
// pass through the wire and so, the previous object will be converted to:

"{ \"name\": \"fancy-music.mp3\", \"data\": { \"type\": \"Buffer\" \"data\": [<array of bytes>] }"

// Now, if we do JSON.parse, we get:

{
  name: 'fancy-music.mp3',
  data: {
    type: 'Buffer',
    data: [<array of bytes>]
}

Now the client, would have to know that in the context of this application, the data field is a Buffer and cast it manually, but this has to be application specific, which makes it specially hard to work with.

Another case, is what happens today, is that go-ipfs base64 any buffer it has to send and convers to a string, so in fact the returned object form a go-ipfs http-api would look on the wire bore like:

"{ \"name\": \"fancy-music.mp3\", \"data\": \"base64encodedArrayofBytes\" }"

This is ok for dag-pb, because we can easily cast since we always know that data in dag-pb needs to be a buffer.

The user needs to know which type is going to be returned when doing an ipld.resolve across multiple IPLD Formats

In a similar way, each time a CID/path gets resolved and a change of IPLD Format is perfomed, the receiver needs to know before hand what is going to be the data type of the returned object.

Proposed solutions

1:1 JSON mapping. In order to support the weird casting, every IPLD Format would require to have a 1:1 mapping with JSON (toJSON, fromJSON methods), which is something non trivial (even if we reduced the scope for 'every object needs to be created first in its native serialized format and then converted to JSON'.

Last mile resolve. Another suggestion would be to do a last mile resolve, where what gets passed on the wire is the last IPLD node serialized (in a block) and the client deserializes that node and resolves any remainderPath, being able to capture the right type for that object.

Boxing of values. We can also considere the boxing of values, where every value is passed around as a byte array inside a 'box' and that box also has a label saying its type, so that the consumer can properly cast it to the right type

Notes

We still haven't had the chance to have a long discussion about this dag API, this issue purpose is to collect ideas and get feedback on the proposals.

@whyrusleeping here is the brain dump including notes from our chat yesterday, please add anything that I missed :)

[jbenet]

(1) on the Buffer problem

This object can be serialized and deserialize as many times we want,
since cbor has a 1:1 mapping with JSON, however, if a http client
requests this object, it will have to be JSON.stringify'ed in order to
pass through the wire and so, the previous object will be converted to:

• It could be passed as raw cbor, base64 encoded. instead of json.
• It could be passed as EJSON (a JSON flavor that signals buffers with {"$binary": "<b64-data>"})
• We could standardize to use EJSON (or some similar variant) instead of pure JSON. I wouldn't be opposed to this because JSON's lack of binary buffer is a horrendous omission that keeps wasting my time.

(2) type returned on ipld.resolve

In a similar way, each time a CID/path gets resolved and a change of IPLD Format is perfomed, the receiver needs to know before hand what is going to be the data type of the returned object.

Sorry, I don't see why-- can you give a concrete example with the CID/path case?

1:1 JSON mapping

Last mile resolve. Another suggestion would be to do a last mile resolve, where what gets passed on the wire is the last IPLD node serialized (in a block) and the client deserializes that node and resolves any remainderPath, being able to capture the right type for that object.

Oh-- maybe i get it now. You mean you need the type because you're converting to JSON on the wire?

Yeah, if you convert to JSON on the wire, there HAS to be a 1:1 well-defined mapping. I would encourage maybe the use of EJSON or something that is super clear instead of JSON.

However-- you may be able to circumvent the problem in MANY use cases (probably not all). Just return the blocks-- all of them -- as serialized, and let the libraries de-serialize them. so the HTTP API should return raw blocks, and js-ipfs-api should deserialize them. (this wont cover all use cases, but most).

Another thing is that we have been discussing a proper RPC API over a socket for some time (over proper multiaddr sockets). This will be useful in go-ipfs to prevent the need of an HTTP endpoint, and being able to use a unix domain socket. Separately, this can be used with websockets to expose a high throughput, low request overhead API to js-ipfs-api -- it would avoid all the horrible HTTP requests. (it's what Facebook had to do to make chat work, way back in the day. that work helped motivate websockets.)

Boxing of values. We can also considere the boxing of values, where every value is passed around as a byte array inside a 'box' and that box also has a label saying its type, so that the consumer can properly cast it to the right type

Yeah-- if considering this, why not just send the raw blocks?

[jbenet]

@diasdavid i'm sorry for taking so long.

[daviddias]

Thank you @jbenet

It could be passed as raw cbor, base64 encoded. instead of json.

Same as the last mile resolved solution

It could be passed as EJSON (a JSON flavor that signals buffers with {"$binary": ""})
We could standardize to use EJSON (or some similar variant) instead of pure JSON. I wouldn't be opposed to this because JSON's lack of binary buffer is a horrendous omission that keeps wasting my time.

For all the readers: tl;dr; In HTTP, or data goes base64 encoded (text) or it goes in pure JSON (no Buffers).

So yeah, we can standardize that our HTTP API returns and expects EJSON and it should be parsed that way. //cc @hsanjuan

Oh-- maybe i get it now. You mean you need the type because you're converting to JSON on the wire?

JSON 1:1 mapping solution is different from last mile resolve.

Last Mile Resolve Issue (or the need to also signal the last CID)

If I have something like:

a(type: dag-cbor) -> b(type: dag-cbor) -> c(type: eth-block)

And if I try to fetch /<cid of a>/b/c, In a last mile resolve (sending the last block as a base64 encoded block), I would have to need from the http-api client, that the returning block is a eth-block and not a dag-cbor anymore like the CID I has in the beginning.

This can be solved with returning a JSON object like:

{
  cid: <last cid>
  block: <base 64 encoded>
}

[daviddias]

Another thought, I now realise that we really should standardise on EJSON for our endpoints, independently of the DAGAPI, it just makes things more nice to understand.

[jbenet]

• 👍 for EJSON (or something like it, that's sane. i think there are a couple other similar variants, so we should look at them before picking one and doing the work of changing all the things)

[daviddias]

Can we have a decision on this thread? Or, is there anything against moving forward with the last mile resolve?

@whyrusleeping @jbenet @nicola
?

[daviddias]

🌠 READ THIS - The Formalization of the DAG API

I had a very good chat yesterday with @whyrusleeping and found some new ways to simplify this a lot (but not 100%, there will be still some costs, but for 90% of the cases or more, it will be just right 👌🏽) and also, he convinced me to merge dag get and dag resolve.

So, the DAG API will have only commands/function calls (not including the workbench stuff, but that is another endeavor) and these are:

ipfs dag get <[cid, cid+path]>

The dag get call will be able to fetch nodes and resolve paths, the options for this command are:

Options

• --local-resolve - It will try to resolve everything it can within the scope of the first object, returning the last value it could + the remainerPath

Implementation details:

The http-api endpoint needs to reply with an header that specifies the content-type, e.g:

• dag-cbor, dag-pb, eth-block and so forth, if it is returning a full node
• json, cbor, ejson, octet-stream, string, number, etc, if it is returning a value within the node. This means that the local-resolver of each format needs to tell the type of the value of the thing it just resolved.

ipfs dag put <node>

The dag put call will store a single node at each time and return its respective CID

Options

• --format - The multicodec of the format (dag-pb, dag-cbor, etc)
• --hash-alg - The hash algorithm to be used
• --input-enc - The input encoding of how the object is being passed. This is just a requirement to make sense of a blob of characters from a CLI context.

Shortcomming: There is no clear way to pass a git object, or a ethereum block this way, unless we had conversations from and to json for all of them. Solution: Just add the git, ethereum and other IPLD compatible objects using ipfs block put. Important to note: The shortcomming is just a CLI thing, within a language runtime, it will be fine doing dag put of ethereum nodes, since we have the classes for them.

Other clarifications

• The interface-ipld-format in JS, is what in go is called ipld-node.

[nicola]

@diasdavid that sounds awesome!
Is there a plan to also merge Get and Resolve in the IPLD resolver?

[daviddias]

@nicola absolutely :)

[daviddias]

• The API defined for core on -- https://github.com/ipfs/interface-ipfs-core/issues/81#issuecomment-277271941 -- has been written as part of the interface-ipfs-core.
• For future spec design, follow: feat/public-api/cli - dag-api specs#150