Consumer leaves the group when combined processing+backoff time is higher than `max.poll.interval.ms`

[truepele]

In what version(s) of Spring for Apache Kafka are you seeing this issue?

2.8.x -2.9.x

Describe the bug
On error handling (via FallbackBatchErrorHandler) consumer leaves the group when combined processing+backoff time is higher than max.poll.interval.ms.

To Reproduce

• Configure DefaultErrorHandler with a fixed backoff interval which is few seconds below the max.poll.interval.ms
• Implement some processing code which has a delay for several seconds (such that the delay + the configured backoff time is higher than max.poll.interval.ms) and then throws an exception.
• Run the application

Expected behavior

The failed processing should be retried after the backoff delay without consumer leaving the consumer group

Actual behaviour
Consumer leaves the group due to inactivity for longer than max.poll.interval.ms

[garyrussell]

That is expected - each individual back off interval + the batch processing time must be less than the max.poll.interval.ms.

You need to increase the interval, or reduce the back off to ensure that the aggregate of the backoff and processing time don't exceed the interval.

[garyrussell]

While we could add an additional poll() between processing and the backoff, that would make this one use case inconsistent with all others; where the processing + backoff aggregate is the time between polls.

[garyrussell]

I see you issued a PR. Thanks.

I have approved the PR build, but it will fail checkstyle due to changed import orders.

We will discuss internally tomorrow if we can live with this inconsistency.

[truepele]

each individual back off interval + the batch processing time must be less than the max.poll.interval.ms

Hi @garyrussell
Thank you for looking into this!
May I ask why in the first place this was decided that way? I do not have enough context to think of use cases where such behaviour could be useful. Processing time not always can be predicted accurately, which makes it hard choosing a good backoff interval (or max.poll.interval.ms). Today I can increase max.poll.interval.ms, but tomorrow my processing time increases and I get consumers leaving the group again...

[garyrussell]

May I ask why in the first place this was decided that way?

There was no specific design decision but, as I mentioned, this is consistent with all other error handling. This error handler is unique in that the consumer is paused, so there would be no harm in doing another poll; in other cases, the consumer is not paused, so we can't poll it.

After sleeping on it, I have no issues accepting this change, but I don't consider it a bug.

[garyrussell]

That said, the general advice is to use a generous allowance when setting max.poll.records, max.poll.interval.ms - all kinds of glitches can cause unexpected rebalances if the window is too tight. It's best to make the margin wide enough to ensure a rebalance is extremely unlikely to happen.