Skip to content

fix: await processEvent before acking RabbitMQ message to restore backpressure#490

Merged
Uxio0 merged 1 commit into
mainfrom
uxio/pla-1250-big-number-of-events-blocks-the-event-loop
Apr 14, 2026
Merged

fix: await processEvent before acking RabbitMQ message to restore backpressure#490
Uxio0 merged 1 commit into
mainfrom
uxio/pla-1250-big-number-of-events-blocks-the-event-loop

Conversation

@Uxio0
Copy link
Copy Markdown
Member

@Uxio0 Uxio0 commented Apr 14, 2026

Summary

  • subscribeToEvents callback was not async and did not await func(), causing channel.ack() to fire before any HTTP requests were made
  • This made AMQP_PREFETCH_MESSAGES completely ineffective — every message was acked instantly, regardless of delivery status
  • Result: ~170k events in 15 min × 10 webhooks × up to 5s timeout = ~9,000–11,000 concurrent pending connections at steady state, causing pod instability

Fix

Made the consume callback async, awaits func(), and only acks in finally. Errors are caught and logged; always ack (no requeue — HTTP-level retries are the right layer for transient failures).

Follow-up

Increase AMQP_PREFETCH_MESSAGES from 10 to 100. With the fix applied, prefetch is the real concurrency dial: throughput = prefetch / avg_webhook_latency. At 100 prefetch × 10 webhooks = 1,000 concurrent HTTP connections at steady state, sustaining ~189 events/sec with avg webhook latency under ~530ms.

Closes PLA-1250

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f750555468

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/datasources/queue/queue.provider.ts
…kpressure

Unawaited func() caused channel.ack() to fire synchronously before any
HTTP requests were made, making AMQP_PREFETCH_MESSAGES completely
ineffective and creating unbounded concurrent connections under load.
@Uxio0 Uxio0 force-pushed the uxio/pla-1250-big-number-of-events-blocks-the-event-loop branch from f750555 to 20669a6 Compare April 14, 2026 11:38
@Uxio0 Uxio0 merged commit d3d83ea into main Apr 14, 2026
9 checks passed
@Uxio0 Uxio0 deleted the uxio/pla-1250-big-number-of-events-blocks-the-event-loop branch April 14, 2026 12:15
@github-actions github-actions Bot locked and limited conversation to collaborators Apr 14, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants