Additional metrics for .NET Core and ASP.NET Core

[rob2244]

I have talked internally with the IC3 platform team, and we've come up with a list of additional metrics we'd love to see in .NET 8 which would help us (and others) better understand, monitor and fine tune our services.

Http Connection Pool Metrics

Ideally these metrics would be available per FQDN:Port combination, broken down by type (i.e. HTTP1.1/HTTP2) and there would be some way to track which http client each pool belongs to.

• total number of connections
• number of idle connections
• average idle connection time
• number of connections evicted from pool due to idle timeout
• number of connection timeouts
• average time to connect
• queue length
• average time spent in queue
• number of http 2 keep alive pings which didn't receive a response
• average response time of http 2 keep alive pings

Rate Limiting Middleware Metrics

It would be great if we could have some way of telling these apart by specific rate limiter, as I know that in .NET 7 you can have multiple rate limiters configured. Maybe if we could name each rate limiter then get the metrics output by the name? Or maybe it could be by the endpoint which the rate limiter protects?

• Number of requests kicked out by limiter
• For limiters with queues, length of queue and average time spent in queue
• For Token based limiters the number of unclaimed tokens in the pool and the average rate at which tokens are removed from the pool
• For Fixed window limiters the number of requests seen per window
• For sliding window limiters, the number of requests seen in a window, the number of requests seen in a segment and the number of requests that appeared in multiple segments

Kestrel Metrics

• Number of client requests cancelled/client disconnects (not sure if this one is possible or not)

Memory Metrics

• Private Bytes (MB) as tracked by this issue

Thanks!

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I have talked internally with the IC3 platform team, and we've come up with a list of additional metrics we'd love to see in .NET 8 which would help us (and others) better understand, monitor and fine tune our services.

Http Connection Pool Metrics

Ideally these metrics would be available per FQDN:Port combination, broken down by type (i.e. HTTP1.1/HTTP2) and there would be some way to track which http client each pool belongs to.

• total number of connections
• number of idle connections
• average idle connection time
• number of connections evicted from pool due to idle timeout
• number of connection timeouts
• average time to connect
• queue length
• average time spent in queue
• number of http 2 keep alive pings which didn't receive a response
• average response time of http 2 keep alive pings

Rate Limiting Middleware Metrics

It would be great if we could have some way of telling these apart by specific rate limiter, as I know that in .NET 7 you can have multiple rate limiters configured. Maybe if we could name each rate limiter then get the metrics output by the name? Or maybe it could be by the endpoint which the rate limiter protects?

• Number of requests kicked out by limiter
• For limiters with queues, length of queue and average time spent in queue
• For Token based limiters the number of unclaimed tokens in the pool and the average rate at which tokens are removed from the pool
• For Fixed window limiters the number of requests seen per window
• For sliding window limiters, the number of requests seen in a window, the number of requests seen in a segment and the number of requests that appeared in multiple segments

Kestrel Metrics

• Number of client requests cancelled/client disconnects (not sure if this one is possible or not)

Memory Metrics

• Private Bytes (MB) as tracked by this issue

Thanks!

Author:	rob2244
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I have talked internally with the IC3 platform team, and we've come up with a list of additional metrics we'd love to see in .NET 8 which would help us (and others) better understand, monitor and fine tune our services.

Http Connection Pool Metrics

Ideally these metrics would be available per FQDN:Port combination, broken down by type (i.e. HTTP1.1/HTTP2) and there would be some way to track which http client each pool belongs to.

• total number of connections
• number of idle connections
• average idle connection time
• number of connections evicted from pool due to idle timeout
• number of connection timeouts
• average time to connect
• queue length
• average time spent in queue
• number of http 2 keep alive pings which didn't receive a response
• average response time of http 2 keep alive pings

Rate Limiting Middleware Metrics

It would be great if we could have some way of telling these apart by specific rate limiter, as I know that in .NET 7 you can have multiple rate limiters configured. Maybe if we could name each rate limiter then get the metrics output by the name? Or maybe it could be by the endpoint which the rate limiter protects?

• Number of requests kicked out by limiter
• For limiters with queues, length of queue and average time spent in queue
• For Token based limiters the number of unclaimed tokens in the pool and the average rate at which tokens are removed from the pool
• For Fixed window limiters the number of requests seen per window
• For sliding window limiters, the number of requests seen in a window, the number of requests seen in a segment and the number of requests that appeared in multiple segments

Kestrel Metrics

• Number of client requests cancelled/client disconnects (not sure if this one is possible or not)

Memory Metrics

• Private Bytes (MB) as tracked by this issue

Thanks!

Author:	rob2244
Assignees:	-
Labels:	area-System.Net.Security, untriaged
Milestone:	-

[MihaZupan]

Oops, changed the labels on the wrong issue

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I have talked internally with the IC3 platform team, and we've come up with a list of additional metrics we'd love to see in .NET 8 which would help us (and others) better understand, monitor and fine tune our services.

Http Connection Pool Metrics

Ideally these metrics would be available per FQDN:Port combination, broken down by type (i.e. HTTP1.1/HTTP2) and there would be some way to track which http client each pool belongs to.

• total number of connections
• number of idle connections
• average idle connection time
• number of connections evicted from pool due to idle timeout
• number of connection timeouts
• average time to connect
• queue length
• average time spent in queue
• number of http 2 keep alive pings which didn't receive a response
• average response time of http 2 keep alive pings

Rate Limiting Middleware Metrics

It would be great if we could have some way of telling these apart by specific rate limiter, as I know that in .NET 7 you can have multiple rate limiters configured. Maybe if we could name each rate limiter then get the metrics output by the name? Or maybe it could be by the endpoint which the rate limiter protects?

• Number of requests kicked out by limiter
• For limiters with queues, length of queue and average time spent in queue
• For Token based limiters the number of unclaimed tokens in the pool and the average rate at which tokens are removed from the pool
• For Fixed window limiters the number of requests seen per window
• For sliding window limiters, the number of requests seen in a window, the number of requests seen in a segment and the number of requests that appeared in multiple segments

Kestrel Metrics

• Number of client requests cancelled/client disconnects (not sure if this one is possible or not)

Memory Metrics

• Private Bytes (MB) as tracked by this issue

Thanks!

Author:	rob2244
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[MihaZupan]

Thank you for the suggestions!

average idle connection time

How would this be defined? The average idle-time across all currently idle connections? The average idle-time of connections that were woken up to serve a request within the current interval?

number of connection timeouts

Are you referring to the number of connection attempts that hit the ConnectTimeuot, or the number of requests that hit the request timeout before obtaining a connection?

number of http 2 keep alive pings which didn't receive a response

This seems rather niche, what were you looking to do with this information?

average time spent in queue

Akin to counters like http11-requests-queue-duration, just with dimensions?
Note that this counter currently tracks the average time spent on queue for requests that ended up on the queue (not all requests).
Was that the sort of information you were looking for, or just an average across all requests to track how much latency the request queue is adding to an average request?

average response time of http 2 keep alive pings

Is the purpose of this to track the average RTT or something else?

Rate Limiting Middleware Metrics

cc: @BrennanConroy

Kestrel Metrics

cc: @davidfowl

[davidfowl]

cc @samsp-msft @adityamandaleeka

[rob2244]

@MihaZupan

• For the average idle time: I was thinking the latter, i.e.

The average idle-time of connections that were woken up to serve a request within the current interval

Although I see how both of these could be useful in different ways.

• For the number of connection timeouts: the number of connection attempts that hit the ConnectTimeout

• Regarding the Http 2 metrics, for some of these metrics I was trying to think what might help devs outside of IC3 (with these specifically I was thinking it might help troubleshoot or serve as an early warning to http 2 issues). I don't think we have any use for this metric currently. If any of these don't make sense or there's already easier ways to troubleshoot/track them, feel free to disregard.

• Yes exactly, the current requests-queue-duration metric is great, it would just be adding the relevant dimensions.

[MihaZupan]

Triage: Re: networking, most of these metrics make sense. We'll need to figure out how we want to approach adding dimensions as part of #79371. Optimistically moving to 8.0.