IPNS+IPFS as a NoSQL database

[rethore]

It would be quite nice if we could have a library that would emulate a NoSQL database. So in essence offering an abstraction of IPNS+IPFS through methods such as insert, update, remove and find. From there we could easily build distributed web apps based on this as a back-end. So for instance meteor is using a combination of a client side minimongo library that syncs with a server side MongoDB. If we were to swap the MongoDB with this new library, and move most of the logic on the client side, we could achieve quite a bit already. Going further would require to run the server side as well locally, but it's not impossible, just less convenient. There is of course some problems arising of having an open access back-end database, but as long as people are not able to erase other peoples entries, their activity, if judge negative by the community, could be filtered out in a p2p fashion.

[jbenet]

@rethore yep, this is very possible. there are many systems that implement NoSQL dbs and even full SQL dbs on top of KV-stores. we should look at them and try them out.

one good route would be to start with things from the level nodejs ecosystem. there's a few SQL adaptors wrapping level.

some interesting links:

• https://www.npmjs.com/package/sql-engine
• https://github.com/LMDB/sqlightning
• https://sqlite.org/src4/doc/trunk/www/design.wiki
• https://sqlite.org/src4/doc/trunk/www/bt.wiki
• http://docs.oracle.com/cd/E17076_02/html/bdb-sql/dbsqlbasics.html#identicalusage

[whyrusleeping]

we should take a look at using the files api to build such a thing.

[jbshirk]

https://github.com/bigchaindb/bigchaindb
https://www.bigchaindb.com/features/
https://www.bigchaindb.com/whitepaper/ ipfs mentioned

[fazo96]

The problem is handling permissions. As far as writing and deleting it can all be implemented, but we need a system where each folder has different permissions. It will be possible when we can have more people allowed to publish a given IPNS record 👍

[Kubuxu]

Can a directory with subdirectories being /ipns/Qm.... be created?

[whyrusleeping]

@fazo96 btw, you can generate a key with this and use it to publish an ipns entry anytime without the daemon with this

This behaviour will be integrated into ipfs itself soon, as well as a mechanism for storing and managing keys. But those two programs should serve as a decent way to accomplish a few things now

[fazo96]

Wow that's very nice @whyrusleeping! I hope to see that in the HTTP API soon

[rethore]

thanks @jbshirk, I wasn't aware of this project. It does seem to complement IPFS and ethereum quite well. It would be nice to build a distribution stack that bundle those three projects together and integrate them nicely. Ideally the incentivization scheme should look at compensating for bandwidth, CPU and storage. A complete decentralised crypto-economy and internet stack. This is a really exciting prospects.

[r14c]

I'm in the early stages of developing a graph database on IPFS. The idea is to publish metadata (graphs, vertices, edges, schemas) via IPNS as RWORSets of IPFS hashes. There are some performance considerations, but I'm thinking that GPG could do well as the backend to a cryptographic RBAC system.

I'm in a bit of a time crunch getting funding and such, but I'm excited to finally get cracking on Infinite Automata and collaborating with y'all starting this winter.

[fdietze]

@tokyo-jesus I'd like to know more details of your project. Like how would a db schema look like? Can there be something like transactions? How to avoid race conditions?

[r14c]

Schemas consist of primitive types and named references to other schemas. CRDTs don't have a concept of locking, the state is only guaranteed to be eventually-consistent, which is enough for most needs, but not all. Long term, we'd like to enable strongly-consistent transactions through another distributed mechanism that can provide locks.

[jbenet]

@tokyo-jesus would be great! 👍

[jbshirk]

relevant: https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/