Missing and empty label queries act differently to Prometheus

[Skepfyr]

What did you do?

Query alertmanager for alerts with the label foo=~"bar|" or foo=~".*"

What did you expect to see?

I expected the alerts with the label foo set to bar and the alerts with foo unset to be returned. This is the behaviour when querying the ALERTS metric through prometheus.

What did you see instead? Under which circumstances?

The alerts with the foo label unset weren't returned for any of the regex matchers (although they were returned for a query like foo="")

This behaviour makes providing dropdown filters for alert labels in Grafana impossible as far as I can tell, as it will always filter out the alerts which don't have that label set.

Environment

• System information: Linux 4.4.225-1.el7.elrepo.x86_64 x86_64
• Alertmanager version: 0.16.1
• Prometheus version: 2.14.0
• Alertmanager configuration file:

global:
      resolve_timeout: 5m

    receivers:
    - name: 'default-receiver'
    - name: 'SNMP-Alert-Sender'
      webhook_configs:
      - send_resolved: true
        url: 'http://localhost:25443/v1/target/www.xxx.yyy.zzz/162'
      - send_resolved: true
        url: 'http://localhost:25443/v1/target/www.xxx.yyy.zzz/162'
      - send_resolved: true
        url: 'http://localhost:25443/v1/target/www.xxx.yyy.zzz/162'

    route:
      receiver: 'default-receiver'
      group_wait: 30s
      group_interval: 1m
      group_by: ['...']
      repeat_interval: 1h
      routes:
      - receiver: 'SNMP-Alert-Sender'
        repeat_interval: 1h
        continue: true
        group_by: ['...']

[simonpasquier]

Indeed we need to be consistent with Prometheus which treats a missing label and a label with an empty value the same way.

[vears91]

Hi, I'd like to attempt to fix this