shard index lookup table for higher write throughput and availability #8
Description
At present the index lookup table must be present in a single database. This is great for read (at least when the table is sufficiently small), but bad for write scalability. One of the solution is to shard the index lookup table based on the the source-id (lets just say apply modulo and limit to N=8 shards). It needs to be think through really well though.