fix: cache token double counting at message_start

[xiaolin593]

Description
The Anthropic streaming protocol reports cache tokens (cache_read_input_tokens, cache_creation_input_tokens) in two events:

• message_start — contains the initial usage snapshot
• message_delta — contains the final/authoritative totals

Previously, the message_start handler set both input tokens and cache tokens from the initial event. The message_delta handler then accumulated cache tokens on top, resulting in double counting — cache tokens were applied once from message_start and again from message_delta.

This fix removes the cache token writes from the message_start handler, deferring all cache token accounting to message_delta where the authoritative values are available. Input tokens are still initialized from message_start as before.

** Detailed Example **
message_start
time=2026-06-16T18:37:53.012Z level=DEBUG msg="response body processing" is_upstream_filter=false request="response_body:{body:\"event: message_start\\ndata: {\\\"type\\\":\\\"message_start\\\",\\\"message\\\":{\\\"model\\\":\\\"claude-sonnet-4-20250514\\\",\\\"id\\\":\\\"<request_id>\\\",\\\"type\\\":\\\"message\\\",\\\"role\\\":\\\"assistant\\\",\\\"content\\\":[],\\\"stop_reason\\\":null,\\\"stop_sequence\\\":null,\\\"stop_details\\\":null,\\\"usage\\\":{\\\"input_tokens\\\":317,\\\"cache_creation_input_tokens\\\":4425,\\\"cache_read_input_tokens\\\":0,\\\"cache_creation\\\":{\\\"ephemeral_5m_input_tokens\\\":4425,\\\"ephemeral_1h_input_tokens\\\":0},\\\"output_tokens\\\":1}} }\\n\\n\"} metadata_context:{}"

input_tokens: 317
cache_creation_input_tokens: 4425
cache_read_input_tokens: 0
output_token: 1

message_delta
time=2026-06-16T18:37:57.878Z level=DEBUG msg="response body processing" is_upstream_filter=false request="response_body:{body:\"event: message_delta\\ndata: {\\\"type\\\":\\\"message_delta\\\",\\\"delta\\\":{\\\"stop_reason\\\":\\\"end_turn\\\",\\\"stop_sequence\\\":null,\\\"stop_details\\\":null},\\\"usage\\\":{\\\"input_tokens\\\":317,\\\"cache_creation_input_tokens\\\":4425,\\\"cache_read_input_tokens\\\":0,\\\"output_tokens\\\":274} }\\n\\n\"} metadata_context:{}"

input_tokens: 317
cache_creation_input_tokens: 4425
cache_read_input_tokens: 0
output_tokens: 274

response body processed

time=2026-06-16T18:37:57.884Z level=DEBUG msg="response body processed" is_upstream_filter=false response="response_body:{response:{header_mutation:{} body_mutation:{body:\"data: {\\\"id\\\":\\\"<request_id>\\\",\\\"choices\\\":[],\\\"created\\\":1781635073,\\\"model\\\":\\\"claude-sonnet-4@20250514\\\",\\\"object\\\":\\\"chat.completion.chunk\\\",\\\"usage\\\":{\\\"prompt_tokens\\\":9167,\\\"completion_tokens\\\":274,\\\"total_tokens\\\":9441,\\\"prompt_tokens_details\\\":{\\\"cache_creation_input_tokens\\\":8850}}}\\n\\ndata: [DONE]\\n\\n\"}}} dynamic_metadata:{fields:{key:\"io.envoy.ai_gateway\" value:{struct_value:{fields:{key:\"ai_service_backend_name\" value:{string_value:\"gateway/<placeholder>\"}} fields:{key:\"backend_name\" value:{string_value:\"gateway/<placeholder>/route/claude-sonnet-4/rule/0/ref/2\"}} fields:{key:\"cacheCreationInputTokenUsage\" value:{number_value:8850}} fields:{key:\"cachedInputTokenUsage\" value:{number_value:0}} fields:{key:\"inputTokenUsage\" value:{number_value:9167}} fields:{key:\"model_name_override\" value:{string_value:\"claude-sonnet-4@20250514\"}} fields:{key:\"outputTokenUsage\" value:{number_value:274}} fields:{key:\"response_model\" value:{string_value:\"claude-sonnet-4@20250514\"}} fields:{key:\"route_name\" value:{string_value:\"gateway/claude-sonnet-4\"}} <OMITTED>}}}}}"

prompt_tokens = 9167
completion_tokens = 274
total_tokens = 9441
prompt_detail -> cache_creation_input_tokens: 8850 <-------------------- 2x of cache_creation_input_tokens: 4425

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.74%. Comparing base (78391c6) to head (9b22d15).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2241      +/-   ##
==========================================
- Coverage   84.74%   84.74%   -0.01%     
==========================================
  Files         144      144              
  Lines       21210    21206       -4     
==========================================
- Hits        17975    17971       -4     
  Misses       2153     2153              
  Partials     1082     1082              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[xiaolin593]

/gemini review

[gemini-code-assist]

Code Review

This pull request prevents the double-counting of Anthropic cache tokens by removing the logic that sets cached input tokens and cache creation input tokens during the message_start event, as these are authoritatively handled during the message_delta event. A new unit test has been added to verify this behavior. The reviewer suggested enhancing this test by adding assertions for total input and output tokens to ensure they are also correctly calculated.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

To make the test more robust and ensure that the total input tokens (prompt tokens) and output tokens are also correctly calculated and not double-counted, consider adding assertions for the total input and output tokens.

	cachedTokens, cachedSet := p.tokenUsage.CachedInputTokens()
	require.True(t, cachedSet)
	require.Equal(t, uint32(5), cachedTokens, "cache_read tokens must not be double-counted")

	cacheCreationTokens, cacheCreationSet := p.tokenUsage.CacheCreationInputTokens()
	require.True(t, cacheCreationSet)
	require.Equal(t, uint32(2), cacheCreationTokens, "cache_creation tokens must not be double-counted")

	inputTokens, inputSet := p.tokenUsage.InputTokens()
	require.True(t, inputSet)
	require.Equal(t, uint32(17), inputTokens, "total input tokens must be correctly calculated")

	outputTokens, outputSet := p.tokenUsage.OutputTokens()
	require.True(t, outputSet)
	require.Equal(t, uint32(8), outputTokens, "output tokens must be correctly set")

[PatilHrushikesh]

i think SetInputTokens should also be moved to MessageDelta block, so all token related thing happens at one place

[xiaolin593]

Closed PR as work had been accomplished in #2239 per discussion.