Add a worker time-spent-idle counter to the ruler #727
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
blocked_workers
metric can't give us a good indication of how busy the ruler is, because work happens every 15s and we sample every ~15s.The
evalLatency
metric indicates when evaluations are late, i.e. when we have reached/exceeded our capacity, but doesn't indicate how close we are to it in normal circumstances.Using an idle time counter should give us an idea of how much slack we have in the queue processing; if it approaches 0, then we know our workers are at capacity.
Since counters are monotonic, this should allow us to measure utilisation without hitting the same sampling issue.