`synapse_storage_transaction_time_bucket` prometheus metric has too high cardinality due to desc label

[daenney]

Description

The desc label on synapse_storage_transaction_time_bucket results in a very high cardinality metric. For a single instance, there's 248 variants of desc, multiplied by 15 buckets. This results in over 3k series for a single host.

Though this might be acceptable if you're only ingesting metrics for a single instance, for Prometheus instances that might be scraping multiple Synapses this quickly becomes a problem.

On one of our internal clusters this is resulting in over a million time series, which is about 5x the amount of time series of the next most problematic timeseries: synapse_http_server_response_time_seconds_bucket. Though this is not causing storage issues, it causes unnecessarily high CPU load and memory usage ballooning on the ingesters (so we have to run with much bigger instances) and querying these series become problematic.

Steps to reproduce

• Run lots of Synapse instances
• Scrape them all with a single Prometheus instance
• Prometheus gets sad

Version information

• Homeserver: any
• Version: any since this metric got introduced
• Install method: unrelated
• Platform: unrelated

[daenney]

For reference, based on promtool tsdb analyze on one of our big clusters:

Highest cardinality metric names:
1054110 synapse_storage_transaction_time_bucket
212280 synapse_http_server_response_time_seconds_bucket
78990 synapse_storage_query_time_bucket
75546 synapse_util_caches_cache_evicted_size

[richvdh]

I'd be pretty happy to drop this metric altogether. It's not used on our default grafana dashboard.

[richvdh]

While we're in the area, it might be worth noting that lots of prometheus metrics are duplicated: #11106.

It's not quite a problem on the same scale as synapse_storage_transaction_time_bucket, but those cache metrics are non-trivial.

[daenney]

I'd be pretty happy to drop this metric altogether. It's not used on our default grafana dashboard.

Yup, I noticed that too. Talking to @erikjohnston he was happy for us to stop ingesting this metric altogether, which strongly suggests we can drop it from the code base.