Consider adding activities for potentially long running network operations

[davidfowl]

Edit by @antonfirsov: The plan is to implement this feature request by introducing a separate Activity for each pooled connection, and link those to the HTTP Request Activity via ActivityLink. See #93832 (comment).

---

While diving into more distributed traces it would be interesting to break down outbound HTTP request time into a few more pieces to understand exactly where that time is spent. I think it would be interesting to explore adding an Activity for socket connection attempts and DNS resolution.

This activity could be coupled to HTTP traffic, but I think it would make more sense to push this down into the Socket/Dns APIs. That would make it work for all sorts of client APIs that use the underlying socket API (redis, various database drivers etc).

cc @noahfalk @samsp-msft

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

While diving into more distributed traces it would be interesting to break down outbound HTTP request time into a few more pieces to understand exactly where that time is spent. I think it would be interesting to explore adding an Activity for socket connection attempts and DNS resolution.

This activity could be coupled to HTTP traffic, but I think it would make more sense to push this down into the Socket/Dns APIs. That would make it work for all sorts of client APIs that use the underlying socket API (redis, various database drivers etc).

Author:	davidfowl
Assignees:	-
Labels:	area-System.Net
Milestone:	-

[MihaZupan]

I think instrumenting DNS/Sockets with more information would be fine, but I want to point out that it could become misleading / not match user expectations. Related: #63159 (comment)

You will rarely see a "perfect" trace like this

◄────────────────────HTTP Request────────────────────►
  ◄──DNS──►
           ◄──Socket──►
                       ◄──TLS──►
                                  ◄──Dependency──►

When a request comes in it may kick off a new connection attempt, but then get picked up by a different connection that became available before the new connection is established.
As a result, you could see traces like

◄─────────HTTP Request───────────►
    ◄──────DNS─────►
      ◄──Backend dependency─►
                    ◄────Socket────►
                                    ◄─────TLS────►

Or even

◄─────────HTTP Request───────────►
      ◄──Backend dependency─►
                                       ◄──────DNS─────►
                                                       ◄────Socket────► ...

---

From the HTTP perspective, you could avoid some confusion by inserting a different activity instead

◄────────────────────────HTTP Request───────────────────────────►
  ◄──Waiting for an available connection──►
                                             ◄──Dependency──►

That highlights the issue that the HTTP request is decoupled from the establishment of new connections.

[davidfowl]

From the HTTP perspective, you could avoid some confusion by inserting a different activity instead

This seems ideal for the HTTP request case. It seems then it's possible that we'd also want to disable the dns and socket activities for the HTTP case? I'd be interested to see if we can get a similar trace for connection establishment in other clients and see if a similar problem exists (presumably it does if there's smart connection pooling going on).

[samsp-msft]

This feels like something that should be done with Events on activities. Putting too many extra activities for this sub-operations will be expensive and noisy.
I wonder if the Activity class should have a verbosity setting that would be inherited from its parent, and then code can query the current activity for it, and use that to determine whether to add extra data. That could be extended to be somewhat similar to ILogger, with methods to add event data that will be conditional on the requested verbosity level.
The advantage of using Events on the Activity is that they include a timestamp, which makes sense for things like DNS or creating a connection. They could be skipped if the connection already exists.

[davidfowl]

Putting too many extra activities for this sub-operations will be expensive and noisy.

I agree in general but not for expensive networking operations.

The advantage of using Events on the Activity is that they include a timestamp, which makes sense for things like DNS or creating a connection. They could be skipped if the connection already exists.

Lets explore this then, do you have something more concrete we could investigate and visualize in tools?

[noahfalk]

Lets explore this then, do you have something more concrete we could investigate and visualize in tools?

I think Sam is proposing that you visualize this list of span events that is included in the OTLP proto:

https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto#L232

I assume when you look at it right now it would be empty, but if you want to experiment with putting some data in there you can use the Activity.AddEvent API.

I wonder if the Activity class should have a verbosity setting that would be inherited from its parent, and then code can query the current activity for it, and use that to determine whether to add extra data

This sounds an awful lot like Activity.IsAllDataRequested property (doc guidance). In general Activity events aren't designed to scale to a high verbosity level but a handful of messages with key timings should work fine.

[davidfowl]

Lets take @MihaZupan's scenario, would be add begin/end events for these operations and visualize those as named events on the overall activity timeline? e.g. HttpClient does the following:

Activity.Current.AddEvent(new("wait for connection"));
await GetConnectionAsync();
Activity.Current.AddEvent(new("resolved connection"));

Is that the idea?

[antonfirsov]

@davidfowl is there a specific customer ask behind this proposal (eg, from Aspire)?

[davidfowl]

We were debugging connection slowness with aspire and more details would have helped speed that up.

[samsp-msft]

Lets take @MihaZupan's scenario, would be add begin/end events for these operations and visualize those as named events on the overall activity timeline? e.g. HttpClient does the following:

Activity.Current.AddEvent(new("wait for connection"));
await GetConnectionAsync();
Activity.Current.AddEvent(new("resolved connection"));

Is that the idea?

I would suggest that for the events you don't want begin/end items, just the ones that have been achieved, so you'd get something like:

• DNS address resolved
• Socket opened
• TLS Negotiated
• Request sent

If any of them are skipped it doesn't so much matter. But if there is a DNS item in there you can see how that relates to the overall time for the span.

[davidfowl]

I like the idea of using events and also doing it in HttpClient so that we have a better view of the http connection time (based on #93832 (comment))