MEAI Caching Support

[PederHP]

Some APIs, in particular the Anthropic API, use explicit caching, where content parts need to be marked for caching. The cost savings are 90% - so in agentic loops with a lot of tool use, the difference between caching and not caching is immense in terms of cost.

Currently one has to do something like this:

/// <summary>
/// Wraps content with Anthropic cache control for prompt caching.
/// Cache breakpoints should be placed at the end of "stable" content.
/// </summary>
private AIContent WithCacheControl(AIContent content, TTL? ttl = null)
{
    ContentBlockParam? rawContent = content switch
    {
        TextContent tc => new TextBlockParam
        {
            Text = tc.Text ?? string.Empty,
            CacheControl = new CacheControlEphemeral { TTL = ttl }
        },
        DataContent dc when dc.MediaType?.StartsWith("image/") == true => new ImageBlockParam
        {
            Source = new(new Base64ImageSource
            {
                Data = Convert.ToBase64String(dc.Data?.ToArray() ?? []),
                MediaType = dc.MediaType
            }),
            CacheControl = new CacheControlEphemeral { TTL = ttl }
        },
        // All other content types need to be manually handled and mapped!
        _ => null
    };

    if (rawContent != null)
    {
        content.RawRepresentation = rawContent;
    }
    return content;
}

This is extremely awkward. And I'm not even sure it is possible to do for System-level content, depending on how the mapping to the top-level property is done (the Anthropic API doesn't have system messages in the messages array).

I think a shared abstraction for Caching would be good so that provider-specific IChatClients can support this. Something like a property on AIContent perhaps?

I'll probably propose an extension to the Anthropic C# SDK to handle this in the meanwhile, but I think it's a valid top-level concern, as Gemini also supports explicit caching.

See: anthropics/anthropic-sdk-csharp#73

[stephentoub]

I think a shared abstraction for Caching would be good so that provider-specific IChatClients can support this. Something like a property on AIContent perhaps?

What would you propose? Even between Anthropic and Gemini the controls exposed vary. I'm not against the idea of adding support to the core abstractions if there's a reasonable way to handle it robustly in a provider-agnostic manner.

It doesn't necessarily have to be on the exact content instance. It could, for example, be a dedicated content type that's added immediately after the target content, in which case it could be provider specific without complicating the model. The IChatClient implementation then peeks ahead at the next content to see if they should be combined logically. (This is how, for example, the google nuget package handles thought signatures on arbitrary content, by adding an additional reasoning content node after the target.)

AdditionalProperties can also be used, e.g.

TextContent tc = ...;
tc.AdditionalProperties = new() { ["CacheControl"] = new CacheControlEphemeral { TTL = ttl };

and then IChatClient implementation just looks for additional property information there.

[PederHP]

Yes, the AdditionalProperties approach is how I've implemented it currently.

I like the idea of a dedicated content type as that will be compatible with a wider array of caching approaches. Prefix-based caching (the style Anthropic uses) can use it as a marker. I looked more into Gemini's style of caching and it's quite complicated, and seems to allow infix caching, which I don't understand as that should be fundamentally incompatible with transformer models.

But a CacheMarkerContent with a TTL (which is in both Gemini and Anthropic) property seems a reasonably generic abstraction. For Gemini one would have to place two markers and set a name on them. This would allow the ChatClient to handle the cache creation and referencing complexity.

Maybe adding this just to support something that's only available in two APIs is a bit much, but the cost savings from using caching are massive, so making sure caching has good ergonomics is important. But maybe it's enough to have the extension in the SDKs.

It just feels off that something which can easily reduce agent costs by 50% to 80% for highly agentic use-cases isn't convenient to use. It's difficult to find a good caching strategy as that is domain-specific, but it should be easy to implement the caching strategy.

I guess I'm leaning towards not adding this after all, now that I've looked into the Gemini caching mechanisms, but I think we should try to make as accessible as possible in the Anthropic-based IChatClient.