Propagate opentelemetry spans across remote connections.

[rread]

The current span context is extracted and serialized with the message, and deserialized in read_request_raw() and saved in thread local storage. The generated RemoteService now creates a new span based on the message variant name, and sets the new span's parent to the extracted context.

Fixes #71

[Frando]

Thanks! The feature sounds useful.

• With this PR the wire format of a protocol changes based on feature flags for irpc. We can't do this unfortunately: You would never want your wire messages to change because some dependency enabled a feature on irpc which will - due to feature unification - affect all crates in the workspace dependency tree. So, we need a different approach here. I'll have to do some thinking how best to do this.

• The PR makes the code generation dependant on feature flags on the proc macro crate. I think this is a no-go: Due to feature unification you can never know for sure if and where a feature got enabled. So this should be based on configuration via attributes on the rpc_requests macro instead.

• The opentelemetry integration should be behind a separate feature use-tracing-opentelemetry or similar

[rread]

Thanks for your review. I updated this to make OpenTelemetry span propagation opt-in per service instead of being globally enabled via feature flags.

Changes

• Added span_propagation attribute to the rpc_requests macro - services that want span context propagation now need to explicitly opt in: #[rpc_requests(message = MyMessage, span_propagation)]
• Moved OpenTelemetry deps behind a new use-tracing-opentelemetry feature instead of bundling them with rpc
• Wire format is now controlled per-service via Service::SPAN_PROPAGATION - services without span_propagation skip the context wrapper
• Removed the spans and rpc feature flags from irpc-derive - the generated code now calls a helper function that handles the cfg logic internally

[rread]

Fixed the CI errors.

[jgrund]

Any chance this can make the next release? I'd like to use this to get distributed traces of my app

[rklaehn]

Any chance this can make the next release? I'd like to use this to get distributed traces of my app

I will let @Frando push the button. But thanks a lot, this is very neat.

[Frando]

Thanks for your continued work on this. The impl now looks much better!

• I'm wondering if the thread-local storage based approach is correct? irpc is usually run in a multi-threaded tokio runtime, so things can move between threads. Wouldn't that mean that the span context can end up being stored on a different thread than where it's being read from?
  I'll think a bit if we can pass the span context without a thread local, that would be preferably IMO.

• irpc-iroh should also get the use-tracing-opentelemetry feature, if enabled it should propagate the feature to irpc

• I think we can rename the feature flag to just tracing-opentelemetry if we set resolver = 2 in the Cargo.toml. Edit: I think we need to switch to edition 2024 for that

• Please add docs for the span_propagation flag to the docs of the rpc_requests macro (in irpc/src/lib.rs)

• We should add an end-to-end test to verify this all works as intended, and so that we don't break the feature accidentally in the future.

[Frando]

I tried to get this to work, but did not succeed. Maybe I don't know enough about the interaction between tracing and opentelemetry.

I pushed a branch that contains a test example, and adds more debug logging:
https://github.com/rread/irpc/compare/rread/tracing...n0-computer:irpc:Frando/span-propagation-test?expand=1

If you check out this Frando/span-propagation-test branch, you can run my test with much debug logging with:

RUST_LOG=irpc=trace,irpc_iroh=trace cargo test -p irpc-iroh --lib --all-features -- span_propagation --nocapture

I would have expected to get the req_id = 23 from the client spans somewhere in a server span. However, I couldn't find it anywhere in the debug output (the test does not actually test anything for now).

Or did I get something wrong what this feature enables?

[rread]

Thanks for the feedback, I'll work on the suggestions and take a look at your test.

IIRC resolver 2 only requires edition 2021.