Distributor queries are weird when you have low numbers of ingesters

[bboreham]

While testing, I had zero ingesters owning tokens. In this situation, Distributor.Query will:

• if your query has a metric, error with "empty circle". This is a confusing message because we never talk about "circle", only "ring".
• if your query has no metric, continue with an empty set to queryIngesters() which will compute a negative number as the minimum required for success, hence return success with no data.

At least, the behaviour should be consistent. I presume minSuccess should never be negative but haven't researched what it should be.

At a higher level, I don't think an absence of ingesters should cause queries to fail. We can tell, without asking, that the empty set of ingesters holds an empty set of chunks, so the query should return whatever is in the persistent store.

[tomwilkie]

I hit this recently testing the Cassandra branch. Have opened #681 to resolve it IMO.

At a higher level, I don't think an absence of ingesters should cause queries to fail. We can tell, without asking, that the empty set of ingesters holds an empty set of chunks, so the query should return whatever is in the persistent store.

That sounds useful for other things, but this issue mainly arises from misconfiguration, right? In this case I think automatically only returning data from the chunk store might be a bit confusing. In normal operation, the ring should never be empty. Having a Cortex mode that only ran from chunk store (read-only) does sound cool though.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.