kafka_deliveryAttempt header with initial value 2 in non-blocking retries

[jgslima]

In what version(s) of Spring for Apache Kafka are you seeing this issue?
2.9.1

Describe the bug
It is about the value of the Header "kafka_deliveryAttempt" (KafkaHeaders.DELIVERY_ATTEMPT).
When used in combination with non-blocking retries, from the 2nd non-blocking attempt, it starts with the value 2 instead of 1.

Example: in a configuration with 2 blocking retries (Backoff with 2 maxAttempts) and a RetryTopicConfiguration with .maxAttempts(3), one would expect the value of the Headers in each listener invocation to be:

kafka_deliveryAttempt=1, retry_topic-attempts=null
kafka_deliveryAttempt=2, retry_topic-attempts=null
kafka_deliveryAttempt=3, retry_topic-attempts=null

kafka_deliveryAttempt=1, retry_topic-attempts=2
kafka_deliveryAttempt=2, retry_topic-attempts=2
kafka_deliveryAttempt=3, retry_topic-attempts=2

kafka_deliveryAttempt=1, retry_topic-attempts=3
kafka_deliveryAttempt=2, retry_topic-attempts=3
kafka_deliveryAttempt=3, retry_topic-attempts=3

But instead we have:

kafka_deliveryAttempt=1, retry_topic-attempts=null
kafka_deliveryAttempt=2, retry_topic-attempts=null
kafka_deliveryAttempt=3, retry_topic-attempts=null

kafka_deliveryAttempt=2, retry_topic-attempts=2
kafka_deliveryAttempt=2, retry_topic-attempts=2
kafka_deliveryAttempt=3, retry_topic-attempts=2

kafka_deliveryAttempt=2, retry_topic-attempts=3
kafka_deliveryAttempt=2, retry_topic-attempts=3
kafka_deliveryAttempt=3, retry_topic-attempts=3

To Reproduce
Please see above.

Expected behavior
Please see above.

[jgslima]

Please do not mind in trying to come up with a workaround solution. This is a small bug which we can live with until there is the actual fix.

[garyrussell]

This is weird; it works fine in a JUnit integration test, but I see this behavior in a Boot app (unless I run it in a debugger, in which case it works there too). I thought maybe a race condition, but this state is maintained in a ThreadLocal.

Investigating...

[garyrussell]

It seems to be another side effect of making RuntimeException retryable. It doesn't happen with, say, IllegalStateException.

Still trying to figure it out.

[garyrussell]

I figured it out; will have a PR tomorrow - but there is no work around.