Skip to content

blackhole sink does not implement end-to-end acknowledgements despite documentation claiming support #24281

@sanjams2

Description

@sanjams2

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

The blackhole sink documentation states it supports end-to-end acknowledgements, but the sink does not actually implement event finalizer handling. This causes sources that depend on acknowledgements (like aws_s3 with SQS) to never delete messages from the queue.

What happens:

  • Configure aws_s3 source with SQS notifications
  • Route events through a transform to a blackhole sink with acknowledgements.enabled: true
  • Events are processed successfully (blackhole logs show events collected)
  • SQS messages are never deleted - they reappear after visibility timeout expires
  • Queue depth grows indefinitely

Looking at src/sinks/blackhole/sink.rs, the sink consumes events but never updates finalizer status. Compare to src/sinks/console/sink.rs:38-54 which properly handles finalizers.

Configuration

sources:
    s3_source:
      type: aws_s3
      sqs:
        queue_url: "https://sqs.us-east-2.amazonaws.com/xxx/my-queue"

  sinks:
    blackhole:
      type: blackhole
      inputs: [s3_source]
      acknowledgements:
        enabled: true

Version

0.50.0

References

References:

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugA code related bug.

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions