Skip to content

Add EventSource/EventCounter tracing and metrics for Kestrel #4769

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
analogrelay opened this issue Nov 1, 2017 · 4 comments
Closed

Add EventSource/EventCounter tracing and metrics for Kestrel #4769

analogrelay opened this issue Nov 1, 2017 · 4 comments
Assignees
@davidfowl
Labels
area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions feature-kestrel feature-yarp This issue is related to work on yarp
Milestone
Test failures

Comments

@analogrelay
Copy link
Contributor

Some of the events/counters to add (some may already exist, others may be valuable too, discuss below):

  • Connections - These should probably be emitted by default, when the source is enabled (Informational level).
    • ConnectionStart event & counter - Triggered on a new connection.
    • ConnectionStop event & counter - Triggered on graceful termination of a connection.
    • ConnectionAbort event & counter- Triggered when a connection is terminated abnormally.
    • ConnectionReject event & counter - Triggered when a connection is terminated because a limit was exhausted.
    • ConnectionTimeOut event & counter- Triggered when a connection is terminated by the server because it timed out.
    • ConnectionDuration counter - Measures the length of time a connection is established for
    • ConcurrentConnections counter - Measures the number of concurrent connections (recorded each time a connection is established/terminated).
  • Requests - These should probably be emitted by default, when the source is enabled (Information Level).
    • RequestStart event & counter - Triggered when a request is started
    • RequestStop event & counter - Triggered when a request is completed (includes response status code)
    • RequestAbort event & counter - Triggered when a request is terminated abnormally
    • RequestTimeOut event & counter - Triggered when a request is times out
    • RequestFailure event & counter - Triggered when an exception escapes from the middleware pipeline.
    • RequestsSucceeded counter - Counts the number of requests that ended with 2xx-class status codes
    • RequestsRedirected counter - Counts the number of requests that ended with 3xx-class status codes (this does include 304 Not Modified, but the 3xx class is called "Redirection" in the spec, and isn't a 304 just a redirection to the cache? :P)
    • RequestsRejected counter - Counts the number of requests that ended with 4xx-class status codes
    • RequestsFailed counter - Counts the number of requests that ended with 5xx-class status codes
    • RequestsUpgraded counter - Counts the number of requests that ended with the 101 Switching Protocols status code.
    • RequestDuration counter - Measures the length of time a request takes to process. (May need to be measured in Kestrel and servers because of flushing)
  • Network Details - These should definitely be behind a Keyword or Verbose trace level.
    • RequestBytesRead counter - Counts the number of bytes read for HTTP requests.
    • ResponseBytesWritten counter - Counts the number of bytes written for HTTP responses.

It might be worth coordinating across HttpSysServer and IISIntegration to share code. Also, benchmarking the impact of disabled EventSources will be important. Obviously, avoiding regression of benchmarking scenarios is an extremely high priority, we may need to have some tricks ready for that.

See https://gist.github.com/anurse/af1859663ac91c6cf69c820cebe92303 for some guidance on adding EventSources and EventCounters to ASP.NET projects.
@muratg
Copy link
Contributor

muratg commented Feb 14, 2018

Backlogging based on the discussion with @DamianEdwards the other day.

@aspnet-hello aspnet-hello transferred this issue from aspnet/KestrelHttpServer Dec 13, 2018
@aspnet-hello aspnet-hello added this to the Backlog milestone Dec 13, 2018
@MolallaComm
Copy link

FWIW, I'd like to see this incorporated into a app.UseMetrics("/metrics") helper similar to how app.UseHealthChecks() works. Then it would be easy to build a grafana dashboard for all our kestrel based apps.

@analogrelay analogrelay added ✔️ Resolution: Answered Resolved because the question asked by the original author has been answered. and removed ✔️ Resolution: Answered Resolved because the question asked by the original author has been answered. labels Jan 8, 2020
@analogrelay analogrelay added the feature-yarp This issue is related to work on yarp label Apr 30, 2020
@analogrelay analogrelay removed this from the Backlog milestone Apr 30, 2020
@analogrelay analogrelay added this to the Next sprint planning milestone May 1, 2020
@analogrelay analogrelay mentioned this issue May 6, 2020
Closed
@benaadams
Copy link
Member

For the Requests counters, aren't they better done by Hosting and add to the other hosting counters (which already cover some of them) .

Number of open WebSockets also would be handy

davidfowl added a commit that referenced this issue May 9, 2020
@davidfowl
Added some kestrel event counters
f3e71e9 
- Connection queue length - This is the amount of connections accepted and queued in the thread pool.
- Connection count - The number of connections
- Total connections - The total number of connections ever connected.
- Connection Rate - Connections per second

Contributes to #4769
@davidfowl davidfowl mentioned this issue May 9, 2020
Merged
davidfowl added a commit that referenced this issue May 12, 2020
@davidfowl
Added some kestrel event counters (#21649)
5a0c097 
* Added some kestrel event counters
- Connection queue length - This is the amount of connections accepted and queued in the thread pool.
- Connection count - The number of connections
- Total connections - The total number of connections ever connected.
- Connection Rate - Connections per second

* Added TLS counters
- Current TLS handshakes
- Total TLS handshakes
- Failed TLS handshakes
- TLS handshake per second

* Added HTTP/2 queue length counter

* Improve the event information
- Add TLS version to handshake events
- Add HTTP version to request queue events
- Renamed HTTP/2 request queue length to http request queue


Contributes to #4769
@davidsh davidsh mentioned this issue Jun 4, 2020
Closed
@pranavkm pranavkm removed the fowler label Jun 15, 2020
@davidsh davidsh mentioned this issue Jun 17, 2020
Merged
@samsp-msft samsp-msft mentioned this issue Jul 2, 2020
Closed
@BrennanConroy
Copy link
Member

We think we're done here

@ghost ghost locked as resolved and limited conversation to collaborators Aug 23, 2020
@amcasey amcasey added area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions and removed area-runtime labels Jun 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@davidfowl davidfowl

Labels
area-networking Includes servers, yarp, json patch, bedrock, websockets, http client factory, and http abstractions feature-kestrel feature-yarp This issue is related to work on yarp
Projects
None yet
Milestone
Test failures
Development

No branches or pull requests
9 participants
@analogrelay @davidfowl @pranavkm @benaadams @MolallaComm @muratg @BrennanConroy @amcasey @aspnet-hello