Skip to content

Ruler doesn't report success/failure of rule evaluation #477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jml opened this issue Jul 4, 2017 · 3 comments · Fixed by #1571
Closed

Ruler doesn't report success/failure of rule evaluation #477

jml opened this issue Jul 4, 2017 · 3 comments · Fixed by #1571
Assignees
@jtlisi
Labels
component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. postmortem An issue arising out of a serious production issue

Comments

@jml
Copy link
Contributor

jml commented Jul 4, 2017

This is partly due to the underlying Prometheus library API. We should at least suggest some upstream changes.

Having this information would make debugging issues in a running cortex deployment much easier.
@jml jml self-assigned this Jul 15, 2017
@jml jml added the postmortem An issue arising out of a serious production issue label Jul 15, 2017
@leth
Copy link
Contributor

leth commented Aug 3, 2017 •
edited
Loading

Prometheus does expose a metric: prometheus_rule_evaluation_failures_total though this doesn't allow us to do any special error handling.

@jml
Copy link
Contributor Author

jml commented Sep 7, 2017

This would naturally fall out of #310 (comment)

Probably we can just put the prometheus metrics in our dashboards, making this not a Cortex bug.

@jml jml removed their assignment Dec 13, 2017
@jml
Copy link
Contributor Author

jml commented Dec 13, 2017

Would love to do this, but

  1. am obviously not doing it
  2. less likely to get to this with recent changes

@tomwilkie tomwilkie added the component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. label Jul 12, 2018
@jtlisi jtlisi mentioned this issue Aug 28, 2019
Merged
@jtlisi jtlisi mentioned this issue Sep 6, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jtlisi jtlisi

Labels
component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. postmortem An issue arising out of a serious production issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update Ruler to use upstream Prom Rule Manager grafana/cortex
4 participants
@leth @jml @tomwilkie @jtlisi