[API Proposal]: Add 'AdditionalCacheKeyItems' property to DistributedCachingChatClient and DistributedCachingEmbeddingGenerator

[darthtrevino]

Background and motivation

For our work on GraphRag internally, we normally use the model-name as part of our cache key in our Python-based codebases since different models will yield dramatically different results.

It appears that there's already a mechanism where the DistributedCachingChatClient and DistributedCachingEmbeddingGenerator can be configured by users: via the configure argument within UseDistributedCache. So this can just be a property on the client classes that's visible to developers.

API Proposal (Microsoft.Extensions.AI)

class Test {
  public void test() {
    string modelName = "gpt-4o";
    ChatClientBuilder builder = createChatClientBuilder()
       .UseDistributedCache(configure: options =>
        {
            options.AdditionalCacheKeys = [modelName]
        });
  }
}

API Usage

See example above

Alternative Designs

No response

Risks

No response

[stephentoub]

We could certainly consider adding it as an extra property.

You shouldn't be blocked, though, as you can already derive from DistributedCachingChatClient/EmbeddingGenerator and override the GetCacheKey method, e.g.

protected override string GetCacheKey(IEnumerable<ChatMessage> messages, ChatOptions? options, params ReadOnlySpan<object?> additionalValues) =>
    base.GetCacheKey(messages, options, [.. additionalValues, modelName]);

For reference, the ResponseCachingChatClient in the Evaluations library does that here:

extensions/src/Libraries/Microsoft.Extensions.AI.Evaluation.Reporting/CSharp/ResponseCachingChatClient.cs

Lines 128 to 129 in f99c625

	protected override string GetCacheKey(IEnumerable<ChatMessage> messages, ChatOptions? options, params ReadOnlySpan<object?> additionalValues)
	=> base.GetCacheKey(messages, options, [.. additionalValues, .. _cachingKeys]);

[stephentoub]

@shyamnamboodiripad / @peterwald , would it help or hurt or be a nop for you if there were exposed as an additional public property on DistributedXx?

[peterwald]

I don't think it would make a difference for us. I can't think of any reason it would hurt.

[shyamnamboodiripad]

Agreed - this should be a nop for ResponseCachingChatClinet specifically.

Only minor consideration I can think of is that having more than one mechanism could make debugging slightly more difficult since there would be more than one place to check to understand what things can impact the caching (e.g., when multiple nested Distributed*Clients are in play and / or when some of these clients are defined in 3P assemblies etc.).

But looks like the proposed change would also eliminate the need to inherit Distributed*Client if the only reason to inherit was to override GetCacheKey... So, it would still probably help more than it hurts overall.