Alerting system stops triggering after short uptime

[S3bt3r]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Steps to Reproduce

1. Restart the alerting service/system.
2. Trigger a test alert condition immediately (verify it fires correctly).
3. Wait for a short period.
4. Trigger the alert condition again.
5. Observe: The alert status does not update, and no notification is sent, effectively failing to detect the new breach.

Expected Result

The alerting system initially works correctly after a system restart but silently stops evaluating rules or triggering notifications after a short period. During the failure state, no alerts are fired even when metrics significantly exceed defined thresholds.

Observed Behavior:

• After Restart: Alerts trigger correctly for a brief window.
• After Short Uptime: Alert execution stalls. Status remains "Resolved" and "Last Triggered" shows no recent activity, despite active incidents (e.g., >350 errors/hr vs >100 threshold).
• Workaround: Restarting the service temporarily restores functionality.

Actual Result

Product Area

Alerts

Link

No response

DSN

No response

Version

25.12.1

[linear]

ENG-6359

[getsantry]

Auto-routing to @getsentry/product-owners-alerts for triage ⏲️

[sentaur-athena]

@aldy505 we cannot reproduce this. Can you please take a look to see if there is an issue with the customers self hosted configurations?

[S3bt3r]

I did some research myself and found the following.

The snub-subscription service seemed to have connection problems with Clickhouse and Kafka. This was resolved after restarting the service.

Error before restart:

snuba-subscription-consumer-events-1  | 2026-01-27 17:57:42,036 Connection was closed, reconnecting.
snuba-subscription-consumer-events-1  | 2026-01-27 17:57:42,058 Error on socket shutdown: [Errno 107] Transport endpoint is not connected
snuba-subscription-consumer-events-1  | 2026-01-27 20:22:20,303 Error on clickhouse:9000 ping: Unexpected EOF while reading bytes
snuba-subscription-consumer-events-1  | %3|1769493517.661|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.59:9092 failed: Connection refused (after 1ms in state CONNECT, 33 identical error(s) suppressed)

The subscription service has a problem where it can't find several detectors, but I don't know exactly what this means. The problem persists even after restarting the service.

Errors that persist after restart:

subscription-consumer-events-1  | 07:43:22 [INFO] arroyo.processing.strategies.run_task_with_multiprocessing: Starting init MultiprocessingPool class
subscription-consumer-events-1  | 07:43:22 [INFO] arroyo.processing.strategies.run_task_with_multiprocessing: Starting maybe_create_pool function
subscription-consumer-events-1  | 07:43:22 [INFO] arroyo.processing.strategies.run_task_with_multiprocessing: Finished maybe_create_pool function
subscription-consumer-events-1  | 07:43:22 [INFO] arroyo.processing.strategies.run_task_with_multiprocessing: Finished init MultiprocessingPool class
subscription-consumer-events-1  | 07:43:25 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='events-subscription-results'), index=0): 70314}
subscription-consumer-events-1  | 07:43:25 [INFO] arroyo.processing.processor: Member id: 'rdkafka-f4cce5b9-ea3d-40e3-b8f0-28b19e2fcb6d'
subscription-consumer-events-1  | 07:43:25 [INFO] arroyo.processing.strategies.run_task_with_multiprocessing: Starting maybe_create_pool function
subscription-consumer-events-1  | 07:43:25 [INFO] arroyo.processing.strategies.run_task_with_multiprocessing: Finished maybe_create_pool function
subscription-consumer-events-1  | 07:43:25 [INFO] arroyo.backends.kafka.consumer: Paused partitions after assignment: set()
subscription-consumer-events-1  | 07:43:30 [INFO] arroyo.processing.strategies.run_task_with_multiprocessing: Finished parallel_worker_initializer function with pid 750, elapsed time 6.765867
subscription-consumer-events-1  | 07:45:41 [INFO] sentry.incidents.subscription_processor: Detector not found (subscription_id=35)
subscription-consumer-events-1  | 07:45:41 [ERROR] sentry.incidents.subscription_processor: No detector found for subscription, skipping subscription processing (subscription_id=35 project_id=19)
subscription-consumer-events-1  | 07:45:41 [INFO] sentry.incidents.subscription_processor: Detector not found (subscription_id=38)
subscription-consumer-events-1  | 07:45:41 [ERROR] sentry.incidents.subscription_processor: No detector found for subscription, skipping subscription processing (subscription_id=38 project_id=46)
subscription-consumer-events-1  | 07:45:41 [INFO] sentry.incidents.subscription_processor: Detector not found (subscription_id=33)
subscription-consumer-events-1  | 07:45:41 [ERROR] sentry.incidents.subscription_processor: No detector found for subscription, skipping subscription processing (subscription_id=33 project_id=36)
subscription-consumer-events-1  | 07:45:41 [INFO] sentry.incidents.subscription_processor: Detector not found (subscription_id=36)
subscription-consumer-events-1  | 07:45:41 [ERROR] sentry.incidents.subscription_processor: No detector found for subscription, skipping subscription processing (subscription_id=36 project_id=17)
subscription-consumer-events-1  | 07:45:41 [INFO] sentry.incidents.subscription_processor: Detector not found (subscription_id=37)
subscription-consumer-events-1  | 07:45:41 [ERROR] sentry.incidents.subscription_processor: No detector found for subscription, skipping subscription processing (subscription_id=37 project_id=21)
subscription-consumer-events-1  | 07:45:41 [INFO] sentry.incidents.subscription_processor: Detector not found (subscription_id=34)
subscription-consumer-events-1  | 07:45:41 [ERROR] sentry.incidents.subscription_processor: No detector found for subscription, skipping subscription processing (subscription_id=34 project_id=37)