pub/sub - publish / subscribe

[jbenet]

We've known for some time we need to layer a pub/sub system on IPFS. We should try to reuse other things. The least work, the better. But it should be a simple protocol, easy to implement, well-layered, and meshing well with the rest of IPFS abstractions.

Requirements

• very, very fast
• flexible (maybe different topology-forming algorithms)
• multiple modalities (single publisher, multiple publishers, etc)
• support both encrypted and unencrypted streams (encrypted again, this is above the regular libp2p encryption -- and specific to the pub/sub group)
• support privately encrypted channels (ie user supplied keys)
• layers over IPRS to do discovery

We need to:

• do a survey of relevant {literature, protocols, and implementations}.
• decide on a protocol
• build it into libp2p.

I likely won't have time to do a proper survey until late Nov or Dec. If you'd like to speed this along, post links to great {papers, systems} here for me.

Relevant to research:

• https://github.com/ipfs/ipfs/issues/73 and #ipfs IRC logs.
• XMPP and matrix.org
• MQTT and other messaging queues.
• all the multicast research
• all the pub/sub research

[davidar]

Cc #42

[bharrisau]

It might be easier start with a basic [slow] implementation before doing the high performance multicast P2P thing. For example, this paper (https://www.cs.utexas.edu/~yzhang/papers/mad-info09.pdf) has two modes depending on how active the group is.

The basic implementation would be as simple as:

• Pick the IPNS address you want to subscribe to changes at
• Append signed message with nodeID and TTL/expiry to DHT at IPNS address
• Owner of IPNS address checks all subscriptions and sends a notifyIpnsUpdate message to each node

Encrypted streams with group-specific encryption, multicast and multiple publishers can then be deferred to the more advanced implementation.

[bharrisau]

I guess the basic could be made even simpler by using the backlink ideas (ipfs/ipfs#31) instead of a new interface to the routing. You then only need a new P2P message to notify a node of changes.

[spikebike]

I stumbled upon this "Decentralized Reddit using a DHT to store content and a blockchain to rank it"
https://news.ycombinator.com/item?id=10391996

The first comment seems particularly relevent:
liamzebedee 21 hours ago
Re: the hosting of topics/subreddits in the DHT, I've done quite a lot of research [1] into a very innovative yet not well known P2P publish-subscribe network design [2] from some Norweigan computer scientists that removes the role of hosting for nodes not interested in a topic, even designing a decentralised microblogging platform on top of it [3].

[1] http://liamz.co/wp-content/uploads/2015/03/Computer-Science-Extended-Essay_Liam-Edwards-Playne.pdf
[2] http://acropolis.cs.vu.nl/~spyros/www/papers/PolderCast.pdf
[3] BitWeav http://liamz.co/wp-content/uploads/2015/03/whitepaper.pdf

I'm reading the papers, but it seems like some very interesting discussion for pubsub. In particular Poldercast's finding a subset of nodes with a particular interest and only those nodes host that interest reminds me of IPFS's policy of not downloading anything unless you ask for it.

I'll read the papers before I make any specific pub/sub recommendations.

[jbenet]

@bharrisau agreed on starting simple to get something working and moving to more efficient constructions later.

@spikebike [1] link is broken.

[davidar]

@jbenet it should be

[1] http://liamz.co/wp-content/uploads/2015/03/Computer-Science-Extended-Essay_Liam-Edwards-Playne.pdf

[spikebike]

@davidar @jbenet thanks, right, fixed.

[rabble]

The xmpp stuff proved really hard in comparison to federation doing pubsubhubub.

[jbshirk]

Maybe not new to folks here, but new to me:
https://hackpad.com/Probabilistic-data-structures-7UPPH2soDvw

[jbshirk]

Now I see that discussion about the use of Bloom filters is already underway:
https://github.com/ipfs/ipfs/issues/31#issuecomment-55875124

[jbenet]

See also http://www.w3.org/TR/push-api/ (HT @nicola)