Add start/end time range support to label names and values APIs

[pracucci]

Prometheus has introduced (prometheus/prometheus#7288) the optional support to start and end parameters for the following API endpoints:

• /api/v1/labels
• /api/v1/label/<label_name>/values

We should do the same in Cortex.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

Still valid

[yeya24]

Hi, I am looking at this issue.

For block storage, if we want to support time ranges, do you think it still makes sense to keep the behavior in https://github.com/cortexproject/cortex/blob/master/pkg/ingester/ingester_v2.go#L637? Only query the head range.

[pracucci]

Thanks @yeya24 for your interested! We definitely want some help here.

Some preliminary thoughts:

1. This feature is tricky because large time ranges could OOM ingesters (and we don't want it). For this reason, we may want to introduce a max time range limit on the "end - start" period, which - as all other limits in Cortex - should be overridable by user (easy to build in Cortex).
2. We need to query both ingesters (as we do it right now) and blocks in the long-term storage (via store-gateway).

To keep the work easier, we may address (1) and (2) separately. Doing (1) first, and then (2). When the time range is not specified, I believe it's better to keep the current approach of querying the TSDB head only.

What do you think? /cc @pstibrany

[pstibrany]

1. This feature is tricky because large time ranges could OOM ingesters (and we don't want it). For this reason, we may want to introduce a max time range limit on the "end - start" period, which - as all other limits in Cortex - should be overridable by user (easy to build in Cortex).

With extra effort we could modify LabelValues operation on ingester to use streaming. This would reduce memory pressure on ingesters, and put it onto queriers instead. But they already must merge and deduplicate different incoming results together. This could be combined with splitting big range into multiple smaller ranges – which can generate duplicate labels, but we can leave that for querier to handle. [I don't think we would need range splitting when doing streaming.]

2. We need to query both ingesters (as we do it right now) and blocks in the long-term storage (via store-gateway).

👍

When the time range is not specified, I believe it's better to keep the current approach of querying the TSDB head only.

👍

[pstibrany]

One more comment... limiting user's time range may not be very useful. Even small ranges can contain large number of labels, and on the other hand extending the range may not increase returned labels too much.

We could instead have a limit for returned number of labels.

[pstibrany]

We could instead have a limit for returned number of labels.

Or two limits (but that's perhaps too low-level): one for ingesters, one for entire query.

[pracucci]

With extra effort we could modify LabelValues operation on ingester to use streaming.

But in TSDB you can't query label names/values in a streaming fashion. Am I wrong?

We could instead have a limit for returned number of labels.

But you would have probably already paid most of the cost of fetching label names/values in a distributed system until you realise the limit has been reached.

[pstibrany]

With extra effort we could modify LabelValues operation on ingester to use streaming.

But in TSDB you can't query label names/values in a streaming fashion. Am I wrong?

You're right. Somehow I was under impression that we could do that, but Querier API doesn't support that.

[gouthamve]

When you say Querier API, which API are you referring to? Because we should change it :)