Support concurrent links with stream filtering #14255
Merged
+278
−84
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What?
If a receiver performs stream filtering with AMQP property filters or AMQP SQL filter expressions, the following downsides can occur:
As an example, let's assume a receiver attaches to the start of a multi GB stream providing a link credit of 2. Let's assume only the very first message matches the filter. In this case, RabbitMQ scans the entire stream without processing other links on the same session, and sends the matched message only once the scan completed (after many seconds or even minutes).
Instead, we want other links to be processed concurrently and the receiver might want to start processing the first matched message while RabbitMQ continues filtering the stream.
This commit fixes these two downsides.
How?
After a threshold of consecutively unmatched messages, the session "pauses" filtering on that link temporarily by:
resume_filtering
to itself, andAny other Erlang messages then have a chance to be processed by the session before the filtering on that link is resumed by the
resume_filtering
Erlang message.Once the end of the stream is reached or link credit is exhausted, the
credit_reply
will be returned fromrabbit_stream_queue
torabbit_amqp_session
.An alternative solution would be to use separate Erlang processes for filtering links as they can be CPU bound and also block for disk I/O.