Alert manager silence feature is working partially

[arnavbose]

Hi,

We came across an issue with our alert manager recently. We are able to create silences successfully and are also able to see the created silences in the UI. The issue is that the silence is working partially.

Eg: Suppose there are 10 servers for a host with a label host_name. If I create a silence for the host_name, it should silence all alerts coming from all those 10 servers belonging to the host during the specified duration. What is happening is that sometimes 1 or 2 server alerts from the host are spilling into the slack channel. The rest majority of the alerts are silenced, but just a few of them at random are not silenced. Has this been observed previously in any version of alert manager? Was this a known issue in an older version of alert manager?

Thanks,
Arnav

[sam-celer]

Same here. Proof below. You can see the alerts in the silence impacted alerts but somehow they are also in alerts view being triggered.

Alert in the silence impacted alert

Alert also in the alerts view

Refreshing anything in the silence seems to somehow trigger a refresh and all alerts are then correctly silenced.

[sam-celer]

amtool reports the same issue.

Silence on client_name 'fern' currently active
7b023dad-0f95-4a68-b8a2-41d9d0df463c client_name~=fern 2020-11-21 05:00:00 UTC 2020-11-28 05:01:00 UTC 2020-11-21 04:59:17 UTC prometheus-silencer [fern]silence fern

Yet there are alerts with client_name fern
ConnectionUri="bin://localhost:11510" FromDisplayName="MarketMerchantMarketDataDestination(XCEL_SWEEP)" Status="RECONNECTING" ToDisplayName="CelerMarketMerchantModule" Username="marketdata_marketmerchant_api_1" alertname="appserver_api_socket" client_name="fern" environment="demo" host="fern-demo-lon-app-01" instance="fern-demo-lon-app-01:9273" job="celer-prod-monitoring" jolokia_agent_url="http://localhost:11001/jolokia" location="lon" server_purpose="client" severity="warning" title="API socket session marketdata_marketmerchant_api_1 in status RECONNECTING on fern-demo-lon-app-01" to="CelerMarketMerchantModule" action="TBC" description="API socket session marketdata_marketmerchant_api_1 in status RECONNECTING on fern-demo-lon-app-01" 2020-11-20 22:04:57 UTC 2020-11-21 09:03:57 UTC http://celer-prod-lon-monitoring-01:14000/graph?g0.expr=ApiSocketSessions_ConnectionConfiguredAsEnabled%7Benvironment%21%3D%22uat%22%7D+%3D%3D+1+and+ApiSocketSessions_ConnectionConfiguredAsEnabled%7BStatus%21~%22LOGGED_IN%22%2CUsername%21%3D%22staticdata_marketmerchant%22%7D&g0.tab=1

[roidelapluie]

Please fill the issue templace: https://github.com/prometheus/alertmanager/blob/master/.github/ISSUE_TEMPLATE.md

[sam-celer]

What did you do?
I am rotating silences weekly which might be what is triggering the issue.
Say silence 1 starts Saturday 10am with duration 1w 5mn, then the next Saturday a new silence is triggered at the same time 10am to last 1w 5mn so effectively matching alerts should be permanently silenced. I have the feeling that some alerts are getting through because of this (But it's odd that only some alerts are impacted and that refreshing the silence with any kind of updates makes it all go correct)
The silences are created via the API v2

What did you expect to see?
Alerts being silenced

What did you see instead? Under which circumstances?
Alerts are silenced but somehow are also triggered (They appear in both the silence impacted alerts view and the alerts view)

Environment

System information:
Linux 3.10.0-1127.19.1.el7.x86_64 x86_64

Alertmanager version:
alertmanager, version 0.21.0 (branch: HEAD, revision: 4c6c03e)
build user: root@dee35927357f
build date: 20200617-08:54:02
go version: go1.14.4

Prometheus version:
prometheus, version 2.21.0 (branch: HEAD, revision: e83ef207b6c2398919b69cd87d2693cfc2fb4127)
build user: root@a4d9bea8479e
build date: 20200911-11:35:02
go version: go1.15.2

Alertmanager configuration file:

global:
  resolve_timeout: 5m

route:
  group_by: ['title']
  group_wait: 30s
  group_interval: 2m
  repeat_interval: 30s
  receiver: celer-prometheus-general
  routes:
  - match:
      severity: critical
    receiver: 'celer-prometheus-critical'
    continue: true

-- pager duty receivers --

insert configuration here
Prometheus configuration file:
None relevant I think. Seems to impact any kind of alert

Logs:
N/A

[roidelapluie]

Are you sure that the start time / end time of the silence is correct? AFAIK alertmanager is using GMT

[sam-celer]

Yeah, I'm sure. The fact that I can see the triggered alerts in the silence view just confirms this. And the fact that just editing the silence (but not changing anything) makes it all go correct is another proof that the silence time is correct.
It should be impossible for a silence to be both in the alerts view and in the silence view, should it not?

[sam-celer]

Maybe i can explain a little bit more the scenario.
On the Saturday of week 1, a silence is created at 05:00am with client_name~=fern valid for a week and 5mns
On the Saturday of week 2, the same silence is created at 05:00am again valid for a week and 5ms.
So the matching alerts should be covered by silence 1 during week 1, covered by silence 1 and 2 during the 5mns overlap of the two silences and then covered by silence 2 during week 2 (etc... forever silencing these alerts)

[simonpasquier]

Can you share a list of alerts with labels that are silenced and a list of alerts that aren't silenced while they should be?
When you say that you create a new silence, I presume that you don't edit the existing silence?
Have you tried extending the overlap between the 2 silences a bit (say 15 minutes instead of 5 minutes)?
What happens if you don't edit the silences? Does the issue resolve eventually?

[sam-celer]

The issue happens every weekend so let me get you the list once this coming weekend passes and the issue re-occurs.
That's correct, a new silence is created and the old one expires on its own (This is all done through the API, not manually)
I have tried extending the overlap to 10mns but the same issue happens.
I have waited several hours (4/5 hours) to see whether the issue would resolve itself but it never did. Maybe it would at some point but that would not solve my problem anyway as I don't want any alerts to be triggered.

[sam-celer]

Attached are all the silences and alerts for this weekend occurrence (They should all be silenced). Again updating any silence (even a silence not related to these alerts) rectifies the situation and all alerts disappear.
As an example, you can see below that the alert is in both the alerts view and the silences view which should be impossible.

alerts.pdf
silences.pdf

[sam-celer]

Hi,

Did you guys manage to look into this further?

Thanks,

Sam.

[simonpasquier]

Apologies for the delayed reply, I have some idea of why it may happen. Say that you've created a pending silence that overlaps with an existing silence. It seems that after the current silence expires, Alertmanager won't take into account the pending silence until the silence is updated because the Mutes() method doesn't look at silences in the future.

[sam-celer]

What do you call a 'pending' silence? Because the two silences are both active at the same time. There isn't one pending.

[simonpasquier]

A pending silence for me is a silence created with a start date in the future. E.g. it's not active when created but it will be at some later point.