Creating independent sources/pipelines within Vector

[ichait-bgl]

Hi there,

Currently we have a single AWS SQS source, from which we're creating multiple pipelines each with a different sink. For example the SQS source goes through a remap(A), route, filter and another remap(B) before going into an OpenSearch sink.
Remap(B) is also used as an input to remap(C) which goes into an HTTP sink.

We've noticed that a failure in the HTTP sink endpoint impacts processing of the OpenSearch sink. I'm assuming this is because we haven't specified the buffer.when_full behaviour, and so it's using 'block' by default - which is causing back-pressure and 'stopping' the source - similar to this post. Hopefully setting buffer.when_full = drop_newest will allow other branches to continue processing in the event of a sink failure on that sink.

Just confirming there isn't a better way, within Vector, to create 'independent' branches from a single source i.e. a way to create a 'duplicate' independent source?

Other option we're looking at is to use SNS to fan-out into multiple SQS queues and then have a pipeline for each within Vector.

Thanks in advance!

[pront]

Just confirming there isn't a better way, within Vector, to create 'independent' branches from a single source i.e. a way to create a 'duplicate' independent source?

Hi @ichait-bgl, can you share your config please? Generally, the link you shared above is the answer to this. But looking at the specific config will help me understand your specific setup better.

Other option we're looking at is to use SNS to fan-out into multiple SQS queues and then have a pipeline for each within Vector.

Sounds like a promising workaround.