Skip to content

fix(observability): report OpenAI cache writes as cache_creation_tokens - #5277

Merged
marcromeyn merged 3 commits into
mainfrom
romeyn/cache-write-tokens-main
Nov 5, 2026
Merged

marcromeyn merged 3 commits into
mainfrom
romeyn/cache-write-tokens-main

Conversation

@marcromeyn

@marcromeyn marcromeyn commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

What and why

OpenAI usage reports prompt-cache writes in a detail field: usage.input_tokens_details.cache_write_tokens in
the Responses API and usage.prompt_tokens_details.cache_write_tokens in Chat Completions. The field names come
from the OpenAI API reference (Chat CompletionUsage.prompt_tokens_details) and the openai-python types
responses/response_usage.py (InputTokensDetails) and completion_usage.py (PromptTokensDetails). The
prompt caching guide
states that writes are part of input_tokens: ordinary input = input_tokens - cached_tokens - cache_write_tokens.

extract_token_stats in nemo_gym/observability/model_calls.py read only Anthropic's top-level
cache_creation_input_tokens, so every model-call record for an OpenAI model had cache_creation_tokens: null, even
when the provider reported the count (for example gpt-6-luna returns
input_tokens_details: {cached_tokens: 0, cache_write_tokens: 0} on every call).

Change: when cache_creation_input_tokens is absent, extract_token_stats reads cache_write_tokens from
input_tokens_details or prompt_tokens_details. _prompt_size is not changed: the OpenAI count is a subset of the
prompt count and is never added to tokens_in. The comment on ModelCallRecord and the _prompt_size docstring now
say this. A malformed count (not a non-negative integer) stays None. The model-call capture docs page describes
the per-call cache fields and where they come from.

Other places that read cached tokens:

  • _cache_signal (cache-read count) already reads cached_tokens from both detail groups. No change.
  • TrajectoryTokenStats, the ng_perf token fields and aggregate_model_call_records carry cached_tokens but no
    cache-write count for any provider, including Anthropic. Adding it would add a field to the trajectory schema, a
    key to ng_perf and to the run metrics. Follow-up: carry cache_creation_tokens through TrajectoryTokenStats,
    ng_perf (cache_write_prompt_tokens or similar) and aggregate_model_call_records.
  • nemo_gym/openai_utils.py NeMoGymResponseInputTokensDetails and the Chat Completions to Responses usage
    conversion in responses_converter.py / litellm_model build input_tokens_details with cached_tokens only.
    For OpenAI models served through openai_model this does not matter, because capture stores the provider
    response as returned (see the smoke test). A converted Chat Completions response drops the write count.
    Follow-up if converted responses need it.

No issue: found while checking the capture record of a gpt-6-luna smoke run.

Validation

  • At 559a3c2, with the telemetry extra: pytest tests/unit_tests/test_model_calls.py tests/unit_tests/test_fern_docs_links.py: 83 passed (24 subtests).
  • More table rows: Anthropic's top-level cache write wins over the OpenAI detail field (an explicit 0 included), and negative or boolean write counts stay None in both shapes. Converted Chat Completions responses dropping the write count is the follow-up below (GYM-1205).
  • pytest tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_base_responses_api_model.py tests/unit_tests/test_atif_export.py:
    892 passed, 12 skipped.
  • Four new rows in the existing _USAGE_SHAPES table test: responses nothing cached (the gpt-6-luna shape:
    cache_creation_tokens is 0, not None), responses cache read and write (the OpenAI docs example: 12000
    input tokens with 4000 read and 7000 written; tokens_in stays 12000), chat completions cache read and write,
    and responses malformed cache write. Without the change to model_calls.py, the first three fail and the
    malformed row passes (it guards against trusting a string count).
  • ruff check, ruff format and pre-commit run --files on the three changed files: clean.

Rollout evidence

gym env start --resources-server example_single_tool_call --model-type openai_model ++observability_enabled=true ++model_call_capture_dir=...
with policy_model_name: gpt-6-luna, then gym eval run --no-serve --agent example_single_tool_call_simple_agent --limit 2 --num-repeats 1 --max-output-tokens 512
with the same capture settings. 2 rollouts, both reward 1.0, 2 healthy. Each rollout made 2 model calls. All 4
captured call records have cache_creation_tokens: 0 (with the previous code: null) and cached_tokens: 0. The raw
captured usage is, for example, {"input_tokens": 66, "input_tokens_details": {"cached_tokens": 0, "cache_write_tokens": 0}}.
These prompts are under the 1,024-token caching minimum, so 0 is correct.

To get a nonzero count, two identical calls of about 3,100 tokens were sent directly to the same openai_model server
(POST /v1/responses) and their usage was normalized:

call raw input_tokens_details tokens_in cache_creation_tokens cached_tokens
1 {cached_tokens: 0, cache_write_tokens: 3123} 3126 3123 0
2 {cached_tokens: 3123, cache_write_tokens: 0} 3126 0 3123

tokens_in equals input_tokens in both calls, so the write count is a subset and is not added. Chat Completions
was not smoke-tested against a live endpoint; the table test covers it.

Compatibility

cache_creation_tokens on model-call records for OpenAI usage changes from null to the reported count. No other
token field changes, tokens_in and tokens_total included. Anthropic usage is unchanged: its top-level field takes
precedence. No benchmark scores change.

Linear: FEP-1837 (part of FEP-1829).

@copy-pr-bot

copy-pr-bot Bot commented Oct 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test 5133ead

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:core Shared APIs, servers, telemetry, health, and registries bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer labels Oct 29, 2026

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

LGTM — no concerns found. Reviewed all three changed files and the affected capture/attachment paths. The fallback preserves reported zero, rejects malformed counts, retains Anthropic precedence, and leaves prompt and total token counts unchanged. The regression cases and documentation match those semantics.

Static source review only; tests were not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test 9010ae3

@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test 559a3c2

OpenAI reports prompt-cache writes in input_tokens_details.cache_write_tokens
(Responses) and prompt_tokens_details.cache_write_tokens (Chat Completions).
Token normalization only read Anthropic's top-level
cache_creation_input_tokens, so model-call records for OpenAI models always
had cache_creation_tokens=None even when the provider reported the count.
Read the OpenAI detail field when the Anthropic one is absent. The OpenAI
count is a subset of input_tokens / prompt_tokens, so tokens_in is
unchanged.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
Add usage-table rows for Anthropic's top-level cache write winning over the
OpenAI detail field (an explicit 0 included), and for negative and boolean
write counts in the Responses and Chat Completions shapes, which stay None.

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…tions usage

Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
@marcromeyn
marcromeyn force-pushed the romeyn/cache-write-tokens-main branch from 9c60e52 to 2cf8e0b Compare November 4, 2026 10:57
@marcromeyn

Copy link
Copy Markdown
Contributor Author

/ok to test 2cf8e0b

@ffrujeri ffrujeri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. Small, correct fix. Anthropic precedence, a reported 0, and malformed counts are covered. tokens_in does not change.

flowchart LR
  U[usage] --> A{"cache_creation_input_tokens valid? (Anthropic)"}
  A -- yes --> R["cache_creation_tokens = value, added to tokens_in"]
  A -- no --> B{"input_tokens_details or prompt_tokens_details .cache_write_tokens valid? (OpenAI, new)"}
  B -- yes --> S["cache_creation_tokens = value, tokens_in unchanged"]
  B -- no --> N["None"]
Loading

@marcromeyn
marcromeyn added this pull request to the merge queue Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41925729486

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 4, 2026
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Nov 5, 2026
@marcromeyn
marcromeyn added this pull request to the merge queue Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42071667470

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42073148775

Merged via the queue into main with commit fee5c88 Nov 5, 2026
76 of 79 checks passed
@marcromeyn
marcromeyn deleted the romeyn/cache-write-tokens-main branch November 5, 2026 17:32

This branch was previously deployed

1 inactive deployment
public — 2cf8e0bb Deployed Nov 4, 2026 by marcromeyn via release / finalize / notify #9951
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants