Caching per-request objects on reused HttpContexts

[JamesNK]

ASP.NET Core 3 adds HttpContext reuse - Reuse HttpContext object per HTTP/1 connection and HTTP/2 stream

App frameworks and middleware often have their own objects that are allocated every request. Instead of app frameworks/middleware caching with a shared object pool (creating thread contention), what opportunities are there for to cache lightweight per-request objects on HttpContext?

Because cached objects would eventually be on every HttpContext this functionality should only be used for lightweight objects, and not something large like buffers.

e.g. generated gRPC methods have objects passed to them like a stream reader, stream writer and context - Task SayHellos(HelloRequest request, IServerStreamWriter<HelloReply> responseStream, ServerCallContext context). These objects are mostly wrappers over the underlying HttpContext.

• Before: IServerStreamWriter and ServerCallContext are allocated every request
• After: IServerStreamWriter and ServerCallContext are fetched from the HttpContext. App framework will create and cache them if they don't exist, or reset the existing instances for the current HttpContext.

[JamesNK]

@davidfowl

[benaadams]

Because cached objects would eventually be on every HttpContext this functionality should only be used for lightweight objects, and not something large like buffers.

Could have a cache that drops any object from the pool if it wasn't requested during that request; so it would only live on for 1 request (or a parameterised number e.g. if not used for x Requests then drop)

[davidfowl]

I think resettable features is the feature here. A feature can opt into controlling creation and reset.

[JamesNK]

Could have a cache that drops any object from the pool if it wasn't requested during that request; so it would only live on for 1 request (or a parameterised number e.g. if not used for x Requests then drop)

That's a clever idea. Rather than putting that logic in Kestrel it could be something a feature could track internally

e.g. each time the feature is reset it could increment an "unused" counter by 1. When it is used then "unused" counter goes back to zero. If counter exceeds a certain number then as part of the reset the heavy objects are culled (or could a feature remove itself from its HttpContext?).

I think resettable features is the feature here. A feature can opt into controlling creation and reset.

What happens to features when the HttpContext is reused today?

I don't know if this is a thing to worry about but imagine a web app that uses lots of middleware and app frameworks. Eventually there would be a lot of resettable features in the mix on the HttpContext. Could the time spent each request resetting those features have an impact?

[JamesNK]

Profiling gRPC and the largest allocation is the gRPC context. Caching this would almost halve allocated memory. Savings would be larger in streaming methods where the stream abstractions could also be cached.

[jkotalik]

@JamesNK is there anything left here that we'd consider caching or does most of the perf work we did in 3.1/5.0 cover this?

[JamesNK]

This is still useful IMO