Don't try to write to "leaving" ingesters

[jml]

Currently, when identifying ingesters to write data to, we include those that are in the process of shutting down. These ingesters (correctly) refuse to accept new writes, as they are busy writing everything they have to dynamo. This causes writes to fail, which causes a loss of user data.

Instead, we should:

• mark "leaving" ingesters as such
• skip over them for writing, not counting them to the replica total
• include them querying, counting them to the replica total

[juliusv]

But then new data will already go to a new set of ingesters, so you need to consider a larger set of ingesters for querying when there are ones that are shutting down, so you get the old data from the shutting-down ones and the new data from the new ones.

[juliusv]

Then again, that's less of a problem if we start doing what we said originally: deploying ingesters slowly, one by one?

[jml]

Observed symptom: Error executing query: could only find 1 ingesters for query. Need at least 2 when trying to run a query on Weave Cloud. Retries don't affect result.