Repository navigation
fix(observability): count cached tokens once when usage mixes formats - #4968
Merged
Merged
Conversation
A usage object with Chat Completions prompt_tokens and Anthropic-style cache_read_input_tokens or cache_creation_input_tokens had the cache added to an input count that already includes it, so tokens_in could exceed tokens_total. Decide the prompt size by usage format in one helper: prompt_tokens wins when present, cache counts are added only to Anthropic Messages input_tokens, and Responses input_tokens is used as is. Closes #4721. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
marcromeyn
force-pushed
the
romeyn/fix-cached-token-count
branch
from
October 23, 2026 11:15
834c456 to
a29f000
Compare
11 tasks
Contributor
Author
|
/ok to test df2c4e8 |
marcromeyn
enabled auto-merge
October 23, 2026 15:15
ananthsub
approved these changes
Oct 26, 2026
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40346067209 |
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40348629008 |
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40350661461 |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
extract_token_statsfell back toprompt_tokenswheninput_tokenswas absent and then added top-levelcache_read_input_tokensandcache_creation_input_tokenswhenever they were present. A Chat Completionsprompt_tokenscount already includes cached tokens, so a mixed usage object counted the cache twice:{prompt_tokens: 100, completion_tokens: 20, total_tokens: 120, cache_read_input_tokens: 80}gavetokens_in=180against a total of 120.The prompt size is now decided in one helper,
_prompt_size, whose order of checks is the rule:prompt_tokens(Chat Completions) includes cached tokens and wins whenever present, also when a gateway addsAnthropic-style cache fields beside it.
input_tokenswithout top-level cache fields (Responses) includes cached tokens and is used as is.input_tokenswithcache_read_input_tokens/cache_creation_input_tokens(Anthropic Messages) excludes them,so they are added. A fully cached response without
input_tokensuses the cache sum; zero cache counts alonegive
None, not 0.extract_token_statscalls the helper; its other outputs are unchanged. The only behaviour change is themixed-format case.
cache_creation_tokensis still reported separately, and missing usage still returnsNone.Closes #4721 (part of #4720).
Validation
pytest tests/unit_tests/test_model_calls.py tests/unit_tests/test_base_responses_api_model.py tests/unit_tests/test_anthropic_converter_egress.py: 138 passed.with and without cache hits, fully cached, the two Fix cached-token double counting in mixed-format usage responses #4721 mixed cases, malformed counts, missing usage). Against the
previous code exactly the two Fix cached-token double counting in mixed-format usage responses #4721 rows fail. Line and branch coverage of
_prompt_sizeis complete.tokens_in=180; after: 100.ruff checkandruff format --checkon the changed files: clean.Rollout evidence
N/A: the change is confined to token normalization of a usage object and has no effect on rollout or verifier
behaviour; the unit tests exercise each usage shape directly.
Compatibility
Token totals for mixed-format usage objects decrease to the correct value. No other usage shape changes. No
benchmark scores change.
Linear: FEP-1833 (part of FEP-1829)