Controlling context propagation boundary

[pmm-sumo]

What are you trying to achieve?

There are several use cases where two organisations employing OpenTelemetry communicate with each other, such as: any 3rd party API calls, synthetics monitoring, webhooks, etc. In some cases (e.g. synthetics monitoring) it is anticipated that the called organisation records the identifier of the originating call and samples the trace (if OpenTelemetry is employed) so it could be later used for diagnosing purposes.

When standard trace context propagation approach is being employed, this leads to several side effects, which might or might not be anticipated:

• spans recorded on both of them share the same trace ID,
• each of the organisations has access only to spans recorded on their side; which also means that callee (Organization 2) is not having access to the root span,
• the originating organisation sampling decision will be passed,
• baggage might be unwillingly shared between the organisations, which might cause leaking sensitive information

 Organization 1         Organization 2
    (caller)               (callee)
    
 ┌───────────┐          ┌────API────┐
 │           │          │           │
 │ Service A ├────?────►│ Service B │
 │           │          │           │
 └─────┬─────┘          └─────┬─────┘
       │                      │
     Spans                  Spans
       │                      │
 ┌─────▼─────┐          ┌─────▼─────┐
 │ Backend 1 │          │ Backend 2 │
 └───────────┘          └───────────┘

One way to approach it is simple to leave the status quo. I.e. assume that it's fine to deal with the listed side effects and consider it should be a duty of the organisation making the call to external service that the baggage must not contain sensitive information.

However, perhaps the Tracing API/SDK could be extended to handle such case gracefully and e.g. explicitly filter baggage context as well as provide means for controlling how the trace context should be propagated when making a call to an external organisation. There are several approaches how to handle this, to start with some:

1. Drop trace context in such case and instead identify the caller somehow via baggage. This information could be incorporated on the callee side and persisted, e.g. via storing as a span attribute.
2. Drop trace context and generate a new trace ID for the callee (essentially, include a new context without parent span ID). That way, the caller would have the information that allows to identify the trace on callee side.
3. Leverage tracestate (or trace-flags) and include additional information there. Perhaps a special field could describe that the request is passing the boundary which requires starting a new trace and using linking capability to store information about the relationship with the callee.
4. Define a boundary on the callee side (e.g. via API), so whenever context is being extracted there, instead of continuing the trace, start a new one and store context as a linked span
5. ...

Additional context.

Perhaps this should followup with OTEPS describing a proposal

Related issues: #1255, #1337, #867, #510

[eyjohn]

Thanks for posting this Przemek,

One thing to consider that I'd like to add is data sensitivity, in some cases, I do not wish third parties who may continue the trace to be aware of the true origin on the request and related request by being able to correlate either on a traceId/spanId/TraceState basis. In fact, we may want to obscure the traceIds by generating a new one at the boundary point or having a transformation scheme. In any case, I think this point should be called out during the formation of any recommendations/semantic conventions regarding cross-organisation tracing.

[Oberon00]

4. Define a boundary on the callee side (e.g. via API), so whenever context is being extracted there, instead of continuing the trace, start a new one

For this approach, see also #1188 "Support restarting the trace with a different trace ID"

(quite a lot related issues here)

[Oberon00]

Also for

• each of the organisations has access only to spans recorded on their side; which also means that callee (Organization 2) is not having access to the root span,

Not only the root span is a problem but any parent span can be. I consider this a design problem of W3C traceparent, but it can be worked around by storing the parent span ID in the tracestate under a unique key, see #366 (comment)

[reyang]

Discussed during the 04/23 triage meeting:

1. we want to see if there is a relatively high demand for this
2. if there is a high demand, we want to see if this is something that should be addressed by the W3C TraceContext specification, rather than having OpenTelemetrying providing a workaround / additional layer.
3. the same question might apply to other propagation protocols (e.g. Zipkin B3).

[yurishkuro]

we want to see if this is something that should be addressed by the W3C TraceContext specification

I doubt it. W3C TC defines the format, not the implementation. It calls out that baggage is meant to propagate only within trust boundaries (https://github.com/w3c/baggage/blob/master/baggage/privacy.md):

The main purpose of this header is to provide additional system-specific information to other systems within the same trust-boundary. The baggage header may contain any opaque value in any of the keys. As such, the baggage header can contain user-identifiable data. Systems MUST ensure that the baggage header does not leak beyond defined trust boundaries and they MUST ensure that the channel that is used to transport potentially user-identifiable data is secured.

This is not to say that OTEL SDK is the place to enforce trust boundaries. It's certainly possible to implement in the SDK but since it would require special configuration it is less reliable than other solutions, like not allowing your backend services to talk to external services without going through some DMZ which can sanitize the requests.

[pmm-sumo]

we want to see if this is something that should be addressed by the W3C TraceContext specification

I doubt it. W3C TC defines the format, not the implementation. It calls out that baggage is meant to propagate only within trust boundaries (https://github.com/w3c/baggage/blob/master/baggage/privacy.md):

This is more in context of open-telemetry/semantic-conventions#1230 , but I think W3C TraceContext could be extended with one of the following (among other solutions):

• ability to set parent-id to zeroes, which would identify that a new trace-id should start (and the caller would know what will be this trace ID) - that would solve the problem of having parentless traces in the example I posted above
• defining a new flag (e.g. external) which could identify a hint that the context passes a boundary - that would allow to handle it accordingly on the other side - e.g. start a new trace and link the one provided in traceparent rather than continuing it; this could actually work in conjuction with Support restarting the trace with a different trace ID #1188 (i.e. such behavior could be controlled via a hint, client library config or both)

The main purpose of this header is to provide additional system-specific information to other systems within the same trust-boundary. The baggage header may contain any opaque value in any of the keys. As such, the baggage header can contain user-identifiable data. Systems MUST ensure that the baggage header does not leak beyond defined trust boundaries and they MUST ensure that the channel that is used to transport potentially user-identifiable data is secured.

This is not to say that OTEL SDK is the place to enforce trust boundaries. It's certainly possible to implement in the SDK but since it would require special configuration it is less reliable than other solutions, like not allowing your backend services to talk to external services without going through some DMZ which can sanitize the requests.

I think that maybe the risk of leaking the baggage should be emphasized somehow in the docs. Also, I think that at least auto-instrumentation based libraries might want to eventually provide some means of controlling it

[brunobat]

We have an issue on the Java instrumentation project that relates to this: open-telemetry/opentelemetry-java-instrumentation#8038.
This has concerning security implications. Could you please increase the priority of this issue?