Skip to content

fix(observability): count cached tokens once when usage mixes formats - #4968

Merged
marcromeyn merged 2 commits into
mainfrom
romeyn/fix-cached-token-count
Oct 26, 2026
Merged

marcromeyn merged 2 commits into
mainfrom
romeyn/fix-cached-token-count

Conversation

@marcromeyn

Copy link
Copy Markdown
Contributor

What and why

extract_token_stats fell back to prompt_tokens when input_tokens was absent and then added top-level
cache_read_input_tokens and cache_creation_input_tokens whenever they were present. A Chat Completions
prompt_tokens count already includes cached tokens, so a mixed usage object counted the cache twice:
{prompt_tokens: 100, completion_tokens: 20, total_tokens: 120, cache_read_input_tokens: 80} gave
tokens_in=180 against a total of 120.

The prompt size is now decided in one helper, _prompt_size, whose order of checks is the rule:

  1. prompt_tokens (Chat Completions) includes cached tokens and wins whenever present, also when a gateway adds
    Anthropic-style cache fields beside it.
  2. input_tokens without top-level cache fields (Responses) includes cached tokens and is used as is.
  3. input_tokens with cache_read_input_tokens / cache_creation_input_tokens (Anthropic Messages) excludes them,
    so they are added. A fully cached response without input_tokens uses the cache sum; zero cache counts alone
    give None, not 0.

extract_token_stats calls the helper; its other outputs are unchanged. The only behaviour change is the
mixed-format case. cache_creation_tokens is still reported separately, and missing usage still returns None.

Closes #4721 (part of #4720).

Validation

Rollout evidence

N/A: the change is confined to token normalization of a usage object and has no effect on rollout or verifier
behaviour; the unit tests exercise each usage shape directly.

Compatibility

Token totals for mixed-format usage objects decrease to the correct value. No other usage shape changes. No
benchmark scores change.

Linear: FEP-1833 (part of FEP-1829)

@copy-pr-bot

copy-pr-bot Bot commented Oct 23, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:core Shared APIs, servers, telemetry, health, and registries bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer labels Oct 23, 2026
A usage object with Chat Completions prompt_tokens and Anthropic-style
cache_read_input_tokens or cache_creation_input_tokens had the cache added
to an input count that already includes it, so tokens_in could exceed
tokens_total. Decide the prompt size by usage format in one helper:
prompt_tokens wins when present, cache counts are added only to Anthropic
Messages input_tokens, and Responses input_tokens is used as is.

Closes #4721.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test df2c4e8

@marcromeyn
marcromeyn added this pull request to the merge queue Oct 26, 2026
@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40346067209

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40348629008

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40350661461

Merged via the queue into main with commit f837b83 Oct 26, 2026
66 of 69 checks passed
@marcromeyn
marcromeyn deleted the romeyn/fix-cached-token-count branch October 26, 2026 16:00

This branch was previously deployed

1 inactive deployment
public — df2c4e89 Deployed Oct 23, 2026 by copy-pr-bot[bot] via release / finalize / notify #6838
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix cached-token double counting in mixed-format usage responses

3 participants