DynamoDB: Use eventually consistent reads for older data

[leth]

Eventually consistent reads cost half the throuput units of strongly consistent reads.

The data is eventually consistent across all storage locations, usually within one second or less.

We should be able to safely use eventually consistent reads for data that is sufficiently old!

[bboreham]

When chunks are flushed from an ingester they are immediately removed from the ingester's in-memory store. So "sufficiently old" would be a few seconds after the chunk was flushed. Unfortunately there is no way to know from the querying side when a chunk will be flushed, so you'd have to guess at something like 24 hours, being well past the max chunk age plus time a chunk should live in the flush queue.

On the other hand, if we made the ingester hold the chunk in memory for while after flushing we could use eventual consistency for all DynamoDB reads.

[bboreham]

Interestingly, we only set ConsistentRead on Get calls, not on queries.
And since we do the query first, my arguments about timing are void.

@tomwilkie you added this in #418; do you recall the thinking behind these choices?

[bboreham]

Also recently-flushed chunks will be in memcached. (Modulo #611)