pass user id metrics to prom eval metrics in ruler

[jtlisi]

Another PR involved in the unwinding of #1513

This PR passes a user wrapped register to evaluated rule groups so the userID will accompany each set of metrics a rule group generates.

Signed-off-by: Jacob Lisi [email protected]

[bboreham]

How many metrics are we talking about, per user?

[bboreham]

This change isn't mentioned in the commit message.

[jtlisi]

I'll will take this out 👍

I have some rule groups that take a long time to eval and having larger buckets can be nice

[jtlisi]

@bboreham each user will have 16 metrics, plus 3 for each rule group

[bboreham]

Seems like quite a lot of data (more than double the existing number of per-user metrics?).
Maybe not so bad since each user will only be on one ruler whereas other metrics come from distributor, ingester, etc., and are multiplied by the number of instances.

[jtlisi]

Also worth considering a number of these metrics will already exist without a user label. Specifically the 3 per rule group. But instead of just having the rulegroup label, after this PR they will have the user ID as an accompanying label. This also means in theory as things stand, if two users have a rulegroup with the same name, they will both utilize the same metrics.

[bboreham]

Do the metrics work for you, either before or after this change? See #1557.

[jtlisi]

@bboreham sadly the majority of these metrics are set in the run function we do not call for the RuleGroup in prometheus: https://github.com/prometheus/prometheus/blob/41151ca8dc069448515f48893b8631b9a3ad8df8/rules/manager.go#L279

I haven't put much thought into it but maybe some upstream refactoring can give us all of the evaluation related metrics. However, I don't thing we will ever be able to make much use of the latency/iteration metrics. That would mean letting the prometheus rule group itself handle scheduling and evaluation. That seems like a step backwards in our case since we already have the scheduling queue.

[bboreham]

I have wondered about passing all rules via temporary files inside the container filesystem, similar to how we do template files. This would enable us to use Prometheus' native rule manager and remove a lot of ruler code.

[jtlisi]

I'm closing this in favor of utilizing the prometheus rule manager #1571, which will embed and register these metrics by default. It will also actually record values for all of the metrics provided by the rule group.