Improve ES|QL rate performance for high-cardinality using local circuit breaker

[JonasKunz]

Removes a bottleneck in the ES|QL rate implementation which was found via profiling:

I benchmarked the rate function locally with a high cardinality (2k series) series on fine granular time buckets.
This means that for the aggregation there are many groups and each group only contains little to medium amounts of data.

The profiling showed that the main bottleneck for this use-case was in fact the memory accounting for the CircuitBreaker: We allocate multiple arrays for each group, which in turn means that we are hammering the underlying atomic counter with concurrent updates.

This PR fixes this by making sure that we use the LocalCircuitBreaker here instead: This circuit breaker pre-allocates larger chunks of memory from the atomic counter, causing the number of updates to be significantly reduced.
In my local benchmarks, this reduced the response time for the query by about ~30%.

Unfortunately those results don't exactly translate to our high cardinality CI benchmarks: Those benchmarks also have a high number of groups, but much more data per group, causing this problem to have a lower impact. For that reason we only see a 3~5% improvement here on CI for the high cardinality benchmarks.

[JonasKunz]

Buildkite benchmark this with tsdb-metricsgen-240m-highcardinality please

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 2cff7b7
• Baseline: 560f7b5 (env ID ec38a3f9-eb43-4d98-9362-50eab3e748e9)
• Contender: 2cff7b7 (env ID cf53bc4e-4904-4035-a09a-6a9ffc506cc9)
• Benchmark results

This build ran two tsdb-metricsgen-240m-highcardinality benchmarks to evaluate performance impact of this PR.

History

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[dnhatn]

I have two comments, but the changes look great. Thank you for the investigation @JonasKunz!

[dnhatn]

We should release localCircuitBreakerService if creating reducedStates fails.

[dnhatn]

I think we should pass the actual settings here, but we can do that in a follow-up.

[JonasKunz]

I wanted to, but I looked at DriverContext and it didn't expose the Settings unless I missed it.
If it's easy, I think we can fix it on this PR.
Do you have a suggestion on what we the best way to gain access to the real Settings here?

[dnhatn]

I think we are currently missing some wiring. We can move these settings to PlannerSettings and pass PlannerSettings to DriverContext. Let's address this in a follow-up.