Upgrade to 26.2.1: Issue feed no new issues...

[stumbaumr]

Self-Hosted Version

26.2.1

CPU Architecture

x86_64

Docker Version

28.0.4, build b8034c0ed70494a90c133461d145cd072d920d7c

Docker Compose Version

5.1.0

Machine Specification

• My system meets the minimum system requirements of Sentry

Installation Type

Upgrade from 25.10

Steps to Reproduce

Took me a while to fix this so I wanted to leave this here in case someone else experiences this problem:

• Upgrade from 25.10 to 26.2.1
• First cron updates stopped taking in state updates
• Then no more updates in the issue feed
• Tried restarts and redeploys by cleaning up to no avail

Expected Result

After the Upgrade new issues should be processed and appear in the feed.

Actual Result

No new issues appeared, then started debugging.

Here the result(s):

1. Relay project configs never generated (immediate symptom)
   The taskworker was running but consuming zero tasks. All tasks queued in the taskworker Kafka topic had application="" (set by Sentry 25.x). The new 26.2.1 taskworker sends GetTaskRequest(application='sentry') when fetching work — the taskbroker filter mismatch meant it found nothing and entered a slow backoff loop.
2. Huge task backlog from the old version
   Every event ingested by the old Sentry (25.10) created save_event_transaction tasks in the taskworker Kafka topic. These expired tasks were never processed and blocked the consumer offset.

This was the fix I went for:
Manually populated Redis with project configs for all project keys using compute_projectkey_config(), unblocking relay immediately.
Then stopped taskbroker + taskworker and deleted all stale tasks from the SQLite database (DELETE FROM inflight_taskactivations)
Reset the taskworker Kafka consumer group offset to latest (skipping those obsolete tasks)
Restarted both services — new tasks generated by 26.2.1 including application='sentry' were then processed.

This is fixed for me - so no idea if I should immediately close it or maybe let it should linger for some time to make it easier for other people to find it.

Event ID

No response

[wodCZ]

Thank you for this 🙏

I've had a similar issue - "taskworker was running but consuming zero tasks". I don't understand kafka nor sentry that much, so I couldn't check tasks queued in kafka to see it's tags or something. I only saw growing lag in redpanda on the taskworker topic with no change on the current offset.

I've first tried stopping taskbroker + taskworker and updated the offset to latest (trying to skip any broken tasks) but after starting them again, no change.
Then I stopped them again, opened sqlite3 /var/lib/docker/volumes/sentry-self-hosted_sentry-taskbroker/_data/taskbroker-activations.sqlite and ran delete from inflight_taskactivations, started everything again and voila, tasks started processing again.

Could you please further elaborate on the first step regarding the relay? In relay's logs I'm getting lots of

WARN relay_server::services::projects::source::upstream: error fetching project state 0378232551a7174cdb36e08cdc81627a: deadline exceeded errors=0 pending=260 tags.did_error=false tags.was_pending=true tags.project_key=0378232551a7174cdb36e08cdc81627a

Not sure whether that's actually a problem, but from the Usage stats overview in Sentry it seems that many projects are suspiciously quiet. But maybe it's just the lag from taskworker.

Edit: added the full warn line from relay. The pending=260 is worrying me, but again, no idea what that means and whether that relates to suspiciously quiet activity in our instance.

Edit 2: To add some context.

I've had issues since early morning that tasks were not processing, even though everything looked alive. At the time we were running 25.10.0.
Then I tried restart, which seemed to process few issues, but got stuck again after a minute or two.
Then I started upgrade to 26.2.1, ran into different issue with seaweedfs migration (https://discord.com/channels/621778831602221064/1478731251904020510) - eventually aborted the seaweed migration and restarted install.sh, booted sentry, but unfortunately tasks were not processing even though all services seemed happy in their logs.

[aldy505]

Thank you for submitting the issue. I wonder what's the actual root cause for this.

Looping @markstory if you have some ideas.

[stumbaumr]

Next time I will just delete all file systems except for PostGreSQL. The history of the shell I used is unfortunately gone - the above text is a destillation from my browser history...
I used compute_projectkey_config() from https://github.com/getsentry/sentry/blob/master/src/sentry/tasks/relay.py

[markstory]

Looping @markstory if you have some ideas.

The change in application is a rough bump. There are a couple solutions to it:

1. Go into each broker pod and use sqlite3 to update any activations that have been stored. UPDATE inflight_taskactivations SET application = 'sentry' should do the trick.
2. Delete the pending activations like wodCZ did.

The pending=260 is worrying me, but again, no idea what that means and whether that relates to suspiciously quiet activity in our instance.

That metric indicates the number of stored tasks in sqlite. Do you have any old containers that could be producing tasks without application set?

[Paulkm2006]

Hi,

I encountered the same issue while upgrading from 25.10 to 26.1.2. My logs for relay also indicates a error while trying to fetch

relay-1  | 2026-03-05T04:39:00.461963Z  WARN relay_server::services::projects::source::upstream: error fetching project state 84f19e405ba***684: deadline exceeded errors=0 pending=266 tags.did_error=false tags.was_pending=true tags.project_key=84f19e405ba***684

and a bunch of this error in taskworker

taskworker-1  | 04:40:53 [WARNING] sentry.workflow_engine.processors.detector: ensure_association_with_detector_detector_not_found (group_id=30 group_type=1 project_id=6)

Following @wodCZ 's suggestion I deleted any pendinginflight_taskactivationsbut nothing seemed to work. I also tried to reset taskworker's Kafka offset to latest by executing

docker compose exec kafka kafka-consumer-groups \
  --bootstrap-server kafka:9092 \
  --group taskworker \
  --all-topics \
  --reset-offsets \
  --to-latest \
  --execute

Then finally, I tried to trigger a manual migration using docker compose exec web sentry upgrade --noinput, which generated the following output

* Unknown config option found: 'uptime.snuba_uptime_results.enabled'
Checking for migrations to run for default
No migrations to run for default
Creating missing DSNs
Correcting Group.num_comments counter
04:44:00 [INFO] sentry.outboxes: Executing outbox replication backfill
04:44:00 [INFO] sentry.outboxes: Processing sentry.ControlOutboxs...
04:44:00 [INFO] sentry.outboxes: Processing sentry.RegionOutboxs...
04:44:00 [INFO] sentry.outboxes: done

After all these and a compose restart, everything seemed to started working. However, I'm not sure which step helped for the final mitigation, and what is the true underlying cause of it.