traceId is not propagated on kafka

[codecraft21]

Describe the bug
The traceId is not propagated to Kafka.
Spring Boot 2.5.2
Spring Cloud 2020.0.3
Spring Cloud Sleuth OTel 1.0.0-M8

Sample
See https://github.com/codecraft21/demo-tracing
master branch: generated from start.spring.io + added Sleuth OTel according to its doc
When calling the / endpoint, another endpoint is called (traceId is propagated on http) and then a kafka message is produced.
The consumer has no traceId.

2021-06-26 19:25:12.209  INFO [,b062f82be4a37b87f96916d298f9a41a,296ab729c3b3ed95] 10856 --- [nio-8080-exec-3] c.codecraft.demo.tracing.DemoController  : Handling home
2021-06-26 19:25:12.212  INFO [,b062f82be4a37b87f96916d298f9a41a,0f3000fc63a58f61] 10856 --- [nio-8080-exec-8] c.codecraft.demo.tracing.DemoController  : generated uuid 19008618-ced3-439a-aa47-28575f21ec7e
2021-06-26 19:25:12.213  INFO [,b062f82be4a37b87f96916d298f9a41a,296ab729c3b3ed95] 10856 --- [nio-8080-exec-3] c.codecraft.demo.tracing.DemoController  : produced kafka 19008618-ced3-439a-aa47-28575f21ec7e
2021-06-26 19:25:12.216  INFO [,,] 10856 --- [container-0-C-1] ch.codecraft.demo.tracing.AsyncService   : received 'some data 19008618-ced3-439a-aa47-28575f21ec7e' with headers: RecordHeaders(headers = [], isReadOnly = false)

brave branch: same setup but without OTel, only Sleuth default. And in that case the traceId is propagated to Kafka.

2021-06-26 19:26:17.916  INFO [,ee08855bbd16bed9,ee08855bbd16bed9] 21112 --- [nio-8080-exec-5] c.codecraft.demo.tracing.DemoController  : Handling home
2021-06-26 19:26:17.918  INFO [,ee08855bbd16bed9,d56b43ff83467b4d] 21112 --- [nio-8080-exec-2] c.codecraft.demo.tracing.DemoController  : generated uuid 510314f5-64a0-4005-b171-6759483bf065
2021-06-26 19:26:17.919  INFO [,ee08855bbd16bed9,ee08855bbd16bed9] 21112 --- [nio-8080-exec-5] c.codecraft.demo.tracing.DemoController  : produced kafka 510314f5-64a0-4005-b171-6759483bf065
2021-06-26 19:26:17.922  INFO [,ee08855bbd16bed9,cf0d4734f3375490] 21112 --- [container-0-C-1] ch.codecraft.demo.tracing.AsyncService   : received 'some data 510314f5-64a0-4005-b171-6759483bf065' with headers: RecordHeaders(headers = [], isReadOnly = false)

[marcingrzejszczak]

Can you check Cloud 2021.0.0-SNAPSHOT, Boot 2.5.2 and Otel 1.1.0-SNAPSHOT ?

[codecraft21]

Same effect, not working:

2021-06-28 17:32:00.012  INFO [,623d05548aa118482be8d4a9685e2fb5,d24e00d4673946c9] 22712 --- [nio-8080-exec-5] c.codecraft.demo.tracing.DemoController  : Handling home
2021-06-28 17:32:00.015  INFO [,623d05548aa118482be8d4a9685e2fb5,3101e2ae10500709] 22712 --- [nio-8080-exec-4] c.codecraft.demo.tracing.DemoController  : generated uuid da0122d9-8639-49a6-bf66-f003cf91e0c2
2021-06-28 17:32:00.016  INFO [,623d05548aa118482be8d4a9685e2fb5,d24e00d4673946c9] 22712 --- [nio-8080-exec-5] c.codecraft.demo.tracing.DemoController  : produced kafka da0122d9-8639-49a6-bf66-f003cf91e0c2
2021-06-28 17:32:00.020  INFO [,,] 22712 --- [container-0-C-1] ch.codecraft.demo.tracing.AsyncService   : received 'some data da0122d9-8639-49a6-bf66-f003cf91e0c2' with headers: RecordHeaders(headers = [], isReadOnly = false)

See the latest branch

[marcingrzejszczak]

Hmm @AndersClausen - is this anything related to what you've been working on?

[AndersClausen]

@marcingrzejszczak we shall have a look mate

[codecraft21]

Spring Kafka creates kafka producers and applies post processor, see https://github.com/spring-projects/spring-kafka/blob/main/spring-kafka/src/main/java/org/springframework/kafka/core/DefaultKafkaProducerFactory.java#L741-L748
It works for Brave because it has a org.springframework.cloud.sleuth.brave.instrument.messaging.KafkaFactoryBeanPostProcessor but I could not find an equivalent for OTel. Debugging DefaultKafkaProducerFactory with OTel showed an empty postProcessors list, thus no instrumentation when producing.

[AndersClausen]

@marcingrzejszczak the problem described above is about spring-kafka and our changes applies on the kafka client directly, so shouldn't have an effect on what's described by @codecraft21 - however, we shall double-check and get back to you all.

[codecraft21]

Spring Kafka creates kafka client producers instances directly, how could "your changes" be applied?
Can you give me a pointer to those "changes"?

[flavium31]

@codecraft21 we have recently implemented Kafka instrumentation which decorates the Kafka clients directly, i.e. picks up Producer and Consumer beans from the context and decorates them. AFAIK, spring-kafka does not expose these producers and consumers directly, rather it uses factories to create and manage them under the hood in KafkaTemplate.

The Kafka instrumentation I'm talking about is here:
Autoconfig: https://github.com/spring-cloud/spring-cloud-sleuth/tree/3.1.x/spring-cloud-sleuth-autoconfigure/src/main/java/org/springframework/cloud/sleuth/autoconfig/instrument/kafka
Implementation: https://github.com/spring-cloud/spring-cloud-sleuth/tree/3.1.x/spring-cloud-sleuth-instrumentation/src/main/java/org/springframework/cloud/sleuth/instrument/kafka

[flavium31]

@codecraft21 In addition to my comment above, as part of the same Kafka instrumentation we expose a reactive ProducerFactory and ConsumerFactory to be used with reactor-kafka. Perhaps a similar approach could work for spring-kafka? I see that spring-kafka can also take custom producer and consumer factories. But I'm not sure that would be the right way to do it, or whether spring-kafka has some other abstractions that could be (or already are) instrumented, @marcingrzejszczak ?

[codecraft21]

@flavium31 Thanks for the details.

As I mentioned above: you could call addPostProcessor on spring kafka factories, like brave instrumentation does: https://github.com/spring-cloud/spring-cloud-sleuth/blob/main/spring-cloud-sleuth-brave/src/main/java/org/springframework/cloud/sleuth/brave/instrument/messaging/KafkaFactoryBeanPostProcessor.java#L41

[codecraft21]

Any update on this?

[flavium31]

@marcingrzejszczak if the approach suggested above by @codecraft21 sounds good, we can prepare a PR for this

[marcingrzejszczak]

LGTM please open a PR