Time of day based alert routing/notification

[brian-brazil]

We've had numerous requests for routing alerts based on the time of day/week. This issue is to track those.

[tyrken]

So, @brian-brazil (@fabxc ?), could you provide any design guidance on how to implement said feature, as I think I'd like to suppress all 'severity: warning' alerts overnight rather than putting loads of time-based rule duplication into my Prometheus alert rules. I really don't want to do that as warnings are still valid & worth warning about if I go looking for current Alerts active - I just don't want to be woken up for them. They also shouldn't seem to resolve every evening & start again in the morning.

After being burnt wasting effort on #709 I want some suggestion up front of what might be accepted from the maintainers.

Indeed, it would be nice to see whether it is within the scope of AM. It'd be great to have this feature.

I would like to be able to have time of day, or day of week influence which receiver an alert is sent to. i.e. - during daytime/business hours, alerts might go to a slack channel, vs during night/weekends, same alerts might go to pagerduty, or on e-mail for the current on-call person.

From the Alertmanager perspective, it could be nice to use existing label matching to control routing to different receivers based on datetime.

i.e. -

- match_re:
      alertname: MyAlert
      day: '(Monday|Tuesday|Wednesday|Thursday|Friday)'
   receiver: slack_appteam
- match_re:
      alertname: MyAlert
      day: '(Saturday|Sunday)'
   receiver: pagerduty_appteam

Something similar for time of day? It's a little tricker, and in routing, it would be nice keep things simple... i.e. - match_re on a label like 'time_window: business_hours', but I'm not sure how to get that meaningful label in there from the alert manager perspective without some sort of relabeling within alertmanager itself, and prometheus passing along an alert date/time. I'm a Prometheus/Alertmanager newbie, so apologies in advance if I'm missing something obvious here.

An approach that generates meaningful date/time labels on the alerts, means those labels could also be used in inhibition as per @tyrken 's request to inhibit warnings for some or all alerts overnight.

One of my main drivers for this is to not introduce time based rule duplication in all my prometheus alerts, as that feels cumbersome.

[jmangs]

The ability to subdue events based on the day and/or time would be really useful. I don't know what the best approach is here, but for Sensu they follow a re-usable pattern tied to the handler (receiver): subdue-attributes

[etiennechabert]

+1 : Just here to says it would be a lovely feature to be able to sleep well during the night and have some alerts only during business hours / days

[stuartnelson3]

Adding https://golang.org/pkg/time/#Weekday for future reference. This could be implemented as a pipeline step that filters based on a defined day/time range. All times would be done in UTC.

[danielmotaleite]

This is a important missing feature! Either enable alerts during some time ranges, or allow recursive silent rules

Is there any workaround to silent staging/test/QA alerts during the night, but still receive then during the day?

[simonpasquier]

@danielmotaleite I have a solution for this based on inhibition rules that doesn't require any change to AlertManager. I'll post the blog post address here once it is out.