ingest-occurrences and ingest-monitor silent hang → MAXPOLL eviction loop on 26.4.2

[Abdulaziz-Moafa]

Self-Hosted Version

26.4.2

CPU Architecture

x86_64

Docker Version

25.0.1

Docker Compose Version

2.35.1

Machine Specification

• My system meets the minimum system requirements of Sentry

Installation Type

Upgrade from 25.11.0 to 26.4.2

Steps to Reproduce

1. Run a self-hosted Sentry 26.4.2 install with active cron monitors.

2. both the ingest-monitors and ingest-occurrences containers become unhealthy.

3. Check Kafka consumer-group state. Lag starts growing, and after a while both groups show "no active members".

I don't have a clean synthetic reproducer — Once the services enters this state it does not recover on its own. Restart, force-recreate, and offset-reset all fail to fix it permanently.

Expected Result

• ingest-monitors reads check-ins, updates monitor state. and become healthy
• ingest-occurrences reads occurrences, and become healthy

Actual Result

Both consumers are caught in a poll-timeout / no-progress loop and don't recover on their own.

The pattern is the same on both:

1. The container starts, the consumer joins Kafka, and "New partitions assigned" is logged.
2. After max-poll-interval-ms elapses, librdkafka kicks it out with a MAXPOLL error and the consumer leaves the group.
3. CURRENT-OFFSET never moves. Lag grows.

For ingest-occurrences,

11:33:57 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='ingest-occurrences'), index=0): 14419083}
11:33:57 [INFO] arroyo.processing.processor: Member id: 'rdkafka-3acdb193-f381-4426-a2fd-19ee32a59bd9'
11:33:57 [INFO] arroyo.backends.kafka.consumer: Paused partitions after assignment: set()
%4|1778586639.061|MAXPOLL|rdkafka#consumer-1| [thrd:main]: Application maximum poll interval (1000000ms) exceeded by 50ms (adjust max.poll.interval.ms for long-running message processing): leaving group

For ingest-monitors the pattern is similar, MAXPOLL fires:

11:07:53 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='ingest-monitors'), index=0): 5361426}
11:07:53 [INFO] sentry.monitors.consumers.monitor_consumer: monitors.consumer.check_in_closed (guid='77c6299352cf4e4dafe9bdc0f3d0ac30' slug='updateworkerarrivalstatus111001' status=1)
... [~40 check_in_closed lines over 4 seconds] ...
%4|1778584373.597|MAXPOLL|rdkafka#consumer-1| [thrd:main]: Application maximum poll interval (300000ms) exceeded by 289ms (adjust max.poll.interval.ms for long-running message processing): leaving group

Digging into the monitors container further, I found this. The embedded sentry.utils.pubsub Kafka producer can't talk to the broker:

%5|1778613304.136|REQTMOUT|sentry.utils.pubsub#producer-3| [thrd:kafka:9092/1001]: kafka:9092/1001: Timed out MetadataRequest in flight (after 1000ms, timeout #0)
%4|1778613305.139|FAIL|sentry.utils.pubsub#producer-3| [thrd:kafka:9092/1001]: kafka:9092/1001: ApiVersionRequest failed: Local: Timed out: probably due to broker version < 0.10 (after 1001ms in state APIVERSION_QUERY)
%4|1778613306.273|FAIL|sentry.utils.pubsub#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: ApiVersionRequest failed: Local: Timed out: probably due to broker version < 0.10 (after 1000ms in state APIVERSION_QUERY)

Consumer-group state:

Consumer group 'ingest-monitors' has no active members.
Consumer group 'ingest-occurrences' has no active members.

GROUP TOPIC PART CURRENT-OFFSET LOG-END-OFFSET LAG
ingest-monitors ingest-monitors 0 5362916 5365054 2138
ingest-occurrences ingest-occurrences 0 14419083 14445920 26837

Event ID

No response

[heyitsmdr]

Just want to add that this exact scenario is happening to us as well, right after we upgraded to 26.4.2. I've been trying to find a workaround but haven't yet.

In our case, the ingest-monitor container does a little bit of processing, but it ultimately hits the MAXPOLL error, and even though it does some processing, it never lowers the kafka lag on the ingest-monitor topic.

22:54:57 [INFO] sentry.monitors.consumers.monitor_consumer: monitors.consumer.check_in_closed (guid='4236e511e5e94480b97147e0447347e0' slug='loancron' status=1 updated_status=1)
22:54:57 [INFO] sentry.monitors.consumers.monitor_consumer: monitors.consumer.check_in_closed (guid='ecee42aea2374eff93e4a4ae3e67e4ac' slug='loancron' status=1 updated_status=1)
22:54:57 [INFO] sentry.monitors.consumers.monitor_consumer: monitors.consumer.check_in_closed (guid='eb65c02571ac455fa09138a0cb6e549e' slug='competitioncron' status=1 updated_status=1)
22:54:57 [INFO] sentry.monitors.consumers.monitor_consumer: monitors.consumer.check_in_closed (guid='ad5f38660f5d4853a85d5155cc07339c' slug='emailer' status=1 updated_status=1)
22:54:57 [INFO] sentry.monitors.consumers.monitor_consumer: monitors.consumer.check_in_closed (guid='8aebf99be1cd4c80a5cf44c8819ebe4c' slug='shipmentscron' status=1 updated_status=1)
22:54:57 [INFO] sentry.monitors.consumers.monitor_consumer: monitors.consumer.check_in_closed (guid='c3999f9a475643839c64971e1f4675c7' slug='headlines' status=1 updated_status=1)
22:54:57 [INFO] sentry.monitors.consumers.monitor_consumer: monitors.consumer.check_in_closed (guid='f0e313eb98fa4db8b6d27737fb00735a' slug='tradecron' status=1 updated_status=1)
%4|1778799323.258|MAXPOLL|rdkafka#consumer-1| [thrd:main]: Application maximum poll interval (30000ms) exceeded by 6ms (adjust max.poll.interval.ms for long-running message processing): leaving group

I've tried increasing the max poll interval as well and it always hits MAXPOLL on whatever the increased value is.

[aldy505]

For adjusting the max.poll.interval, please refer to this PR: https://github.com/getsentry/self-hosted/pull/4313/changes

To quickly recover the container, and to make it just process latest events, try to reset the offset to latest: https://develop.sentry.dev/self-hosted/troubleshooting/kafka/#offset-out-of-range-error

[heyitsmdr]

I did try changing the max poll interval. No matter what it is set to, it always hits the max and shows the "leaving group" message. Even when set to several minutes.

I also did reset the offsets to latest. I confirmed lag was back to 0. That didn't seem to make a difference. I reset the offsets and then restarted the ingest-monitors container and I still saw the leaving group/MAXPOLL message. Lag also never went down -- even when there were a few things in the topic and it does process them when the container first starts, however, the current offset never changes.

[aldy505]

I brought this issue on Sentry's internal Slack. I'll let you now next Monday.