Improve watchdog alert

This expression fires the Watchdog alert also if the TSDB is up to date and therefore checks the functionality of the full stack.

With `vector(1)`, only alertmanager needs to be functional to fire. So In case of a full TSDB storage, the Watchdog still fires and the lack of new metrics goes unnoticed.

Signed-off-by: Andreas Tellenbach <[email protected]>

Author: a-Tell

manifests/kubePrometheus-prometheusRule.yaml

@@ -32,7 +32,7 @@ spec:
           "DeadMansSnitch" integration in PagerDuty.
         runbook_url: https://runbooks.prometheus-operator.dev/runbooks/general/watchdog
         summary: An alert that should always be firing to certify that Alertmanager is working properly.
-      expr: vector(1)
+      expr: present_over_time(prometheus_tsdb_head_max_time[1m]) != 0
       labels:
         severity: none
     - alert: InfoInhibitor