-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Labels
type: bugA code related bug.A code related bug.
Description
A note for the community
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Problem
The blackhole sink documentation states it supports end-to-end acknowledgements, but the sink does not actually implement event finalizer handling. This causes sources that depend on acknowledgements (like aws_s3 with SQS) to never delete messages from the queue.
What happens:
- Configure aws_s3 source with SQS notifications
- Route events through a transform to a blackhole sink with acknowledgements.enabled: true
- Events are processed successfully (blackhole logs show events collected)
- SQS messages are never deleted - they reappear after visibility timeout expires
- Queue depth grows indefinitely
Looking at src/sinks/blackhole/sink.rs, the sink consumes events but never updates finalizer status. Compare to src/sinks/console/sink.rs:38-54 which properly handles finalizers.
Configuration
sources:
s3_source:
type: aws_s3
sqs:
queue_url: "https://sqs.us-east-2.amazonaws.com/xxx/my-queue"
sinks:
blackhole:
type: blackhole
inputs: [s3_source]
acknowledgements:
enabled: true
Version
0.50.0
References
References:
- Component spec requiring all sinks support acks: docs/specs/component.md
- Sink tutorial showing correct finalizer pattern: docs/tutorials/sinks/1_basic_sink.md
- Original acks implementation epic: Implement end-to-end record acknowledgement #1107
Metadata
Metadata
Assignees
Labels
type: bugA code related bug.A code related bug.