Horizontally Scalable Ruler #1258
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for working on this change! I'll be very happy to test this when it is ready :) @rsteneteg FYI, since we've discussed this as a scaling contention point in the past |
jtlisi
force-pushed
the
20190219_distributed_ruler
branch
from
db8fcc0 to
4c54de9
Compare
|
Does this also fix #310? |
|
Fixes #1213. |
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
tomwilkie
reviewed
Apr 6, 2019
|
This looks really good! Lots of small nits and we need to chat about how we handle config on Cortex. When you get this green & all the points address, give me a ping and we can do a final pass. |
jtlisi
force-pushed
the
20190219_distributed_ruler
branch
from
c5774e8 to
5a8d444
Compare
|
@tomwilkie I think this is ready for another look |
jtlisi
force-pushed
the
20190219_distributed_ruler
branch
from
94f7902 to
06db06f
Compare
gouthamve
approved these changes
May 5, 2019
jtlisi
force-pushed
the
20190219_distributed_ruler
branch
from
06db06f to
bdefa7b
Compare
jtlisi
force-pushed
the
20190219_distributed_ruler
branch
2 times, most recently
from
a35a3f4 to
1bbde65
Compare
…valuation of rule groups Signed-off-by: Jacob Lisi <[email protected]>
jtlisi
force-pushed
the
20190219_distributed_ruler
branch
from
1bbde65 to
12aec3d
Compare
|
I rebased this on top of master to work with a single binary. This should be ready for another look @tomwilkie |
|
LGTM! Thanks for you work on this Jacob. |
Closed
|
I have edited the description to mention some breaking changes to metrics. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR provides a flag that allows the ruler to shard ownership of which rule groups to evaluate by re-using the ring code. When enabled each ruler will act as a Lifecycler and interact with consul to create a ring.
TO DO
BREAKING CHANGES
Metrics deleted:
rules_queue_lengthMetrics renamed:
cortex_ingester_consul_heartbeats_totalcortex_member_consul_heartbeats_totalcortex_ingester_ring_tokens_ownedcortex_member_ring_tokens_ownedcortex_ingester_ring_tokens_to_owncortex_member_ring_tokens_to_owncortex_ring_ingester_ownership_percentcortex_ring_member_ownership_percentcortex_ring_ingesterscortex_ring_membersThe
ingesterlabel on some metrics is renamedmember.All of the above and
cortex_shutdown_duration_secondsadd anamelabel