Don't try and read/write all ingesters in a set; only do it if one fails/times out

[tomwilkie]

We should only write to the minimum required.

This should improve tail latencies & reduce load on the system.

[jml]

@tomwilkie Is this still relevant?

[tomwilkie]

Yes - we still write to all three; we should only try and write to two, as per Tiered Requests in http://www.cs.duke.edu/courses/cps296.4/fall13/838-CloudPapers/dean_longtail.pdf

[bboreham]

If we write to two ingesters (when configured for 3x replication), and one of them dies, now we have three potential replicas with only one of them holding the data. A subsequent read can accept the response from the two that don't have the data. This is also noted at #731

[csmarchbanks]

I agree that as the ingester stands we need to write to all 3 all the time. The read path could be changed to only do 2 queries, then send the third out later if one of the first two is slow/fails. Perhaps this issue should be changed to only the read path?