Limit per-user metric cardinality

[juliusv]

Even just by accident, it's really easy for a user to overload or hotspot an ingester right now, by either sending too many series with the same metric name, or just too many time series in general (especially when accidentally misusing a label with unbounded value cardinality).

We should have some way of limiting the user's number of series.

[juliusv]

There's a question here of whether we want to limit the total number of series per user or per ingester. Maybe both.

From a system safety perspective, we'll want to limit the number of series per ingester. But that is hard to communicate to the user. They will want to know how many series they can create overall, because they don't understand the uneven distribution of their series over ingesters (similarly to how we stumbled over Dynamo table throughput issues with uneven table shards).

Limiting the per-user series on each ingester would be technically easiest though, because the necessary state is readily available. Given that this is necessary for basic safety, we will want to have some limit here in any case, even if it's pretty high.

I'm not sure how we would track the total number of series for a user as another limit. The distributor would either have to get stats from the ingesters on every append (probably infeasible) or the distributor would have to track metric cardinality itself. Doing full cardinality tracking in the distributor would use too many resources, but maybe it could be done approximately with HyperLogLog.

[tomwilkie]

Limiting the per-user series on each ingester would be technically easiest though, because the necessary state is readily available. Given that this is necessary for basic safety, we will want to have some limit here in any case, even if it's pretty high.

Yes, I think this is the best place to start. We can always offer users this limit as a lower bound.

Doing full cardinality tracking in the distributor would use too many resources, but maybe it could be done approximately with HyperLogLog.

Thats an interesting idea; it would need some kind of moving average as well, as cardinality over time can be virtually unlimited - we close inactive series after 1h.

[rade]

What are we proposing to do when the limit is exceeded? Throw away the data cortex has received? How will a user know that (and why) this is happening?

[juliusv]

@rade Throw away the samples and return an error to the user. For the rate-limiting case, we do the same and return a 429 Too Many Requests, which wouldn't be fully accurate here. I don't know what the best HTTP response code would be, maybe 418 I'm a teapot :)

At first, this would require a user to meta-monitor their Prometheus scraper for failed remote writes, but eventually we could notify users automatically when we see that they are being continuously denied.

[tomwilkie]

We need to make it clear thats its the product of the label cardinality for a given metric thats problematic for us. If we detect such a metric, we should black list it, but not drop the entire batch.

[rade]

Getting something to show up in the Weave Cloud cortex UI would be nice. And, going totally meta... feed a metric into the instance's cortex, which the user can set an alert on :)

[rade]

This is all post MVP, obviously. Let's be safe before being nice.

[juliusv]

We could still store other samples, but should still return an HTTP error, because a 200 would not seem ok in that case. Then the user doesn't know which samples were stored, but I don't think we can do better.

Yeah, in the future we can have nice UI features and meta-metrics for this.

[rade]

Then the user doesn't know which samples were stored, but I don't think we can do better.

We can say what did/didn't happen in the response body. That won't be easy for the user to track down (would the sending prom even log it?), but it's better than nothing.

[juliusv]

The Prometheus server doesn't look at the response body at all, but yeah, theoretically a user could tcpdump it. It would not be super useful, since the series that were rejected are not special in any way, they just happened to be the first ones that were "one too much".

On a technical level, reporting these details back would require changing the gRPC response from the ingesters to include this information and the distributor to then merge it and send the failed series back to the user. Since I don't believe it'll ever be even seen by anyone, I doubt the value of that.

[rade]

The Prometheus server doesn't look at the response body at all

That seems wrong. Surely it should log any errors it gets back.

the series that were rejected are not special in any way, they just happened to be the first ones that were "one too much".

What makes them special is that cortex has thrown away some of the data. And that is of interest to users, I would have thought.

On a technical level, (it's complicated)

Fair enough. Not part of the MVP them. As I said, let's be safe before being nice.