Repository navigation
fix(observability): report OpenAI cache writes as cache_creation_tokens - #5277
Conversation
|
/ok to test 5133ead |
|
🌿 Preview your docs: https://nvidia-preview-pr-5277-2cf8e0bbb8ce.docs.buildwithfern.com/nemo/gym |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
LGTM — no concerns found. Reviewed all three changed files and the affected capture/attachment paths. The fallback preserves reported zero, rejects malformed counts, retains Anthropic precedence, and leaves prompt and total token counts unchanged. The regression cases and documentation match those semantics.
Static source review only; tests were not executed.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
|
/ok to test 9010ae3 |
|
/ok to test 559a3c2 |
OpenAI reports prompt-cache writes in input_tokens_details.cache_write_tokens (Responses) and prompt_tokens_details.cache_write_tokens (Chat Completions). Token normalization only read Anthropic's top-level cache_creation_input_tokens, so model-call records for OpenAI models always had cache_creation_tokens=None even when the provider reported the count. Read the OpenAI detail field when the Anthropic one is absent. The OpenAI count is a subset of input_tokens / prompt_tokens, so tokens_in is unchanged. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
Add usage-table rows for Anthropic's top-level cache write winning over the OpenAI detail field (an explicit 0 included), and for negative and boolean write counts in the Responses and Chat Completions shapes, which stay None. Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
…tions usage Signed-off-by: Marc Romeyn <mromeijn@nvidia.com>
9c60e52 to
2cf8e0b
Compare
|
/ok to test 2cf8e0b |
ffrujeri
left a comment
There was a problem hiding this comment.
Approve. Small, correct fix. Anthropic precedence, a reported 0, and malformed counts are covered. tokens_in does not change.
flowchart LR
U[usage] --> A{"cache_creation_input_tokens valid? (Anthropic)"}
A -- yes --> R["cache_creation_tokens = value, added to tokens_in"]
A -- no --> B{"input_tokens_details or prompt_tokens_details .cache_write_tokens valid? (OpenAI, new)"}
B -- yes --> S["cache_creation_tokens = value, tokens_in unchanged"]
B -- no --> N["None"]
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41925729486 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42071667470 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42073148775 |
What and why
OpenAI usage reports prompt-cache writes in a detail field:
usage.input_tokens_details.cache_write_tokensinthe Responses API and
usage.prompt_tokens_details.cache_write_tokensin Chat Completions. The field names comefrom the OpenAI API reference (Chat
CompletionUsage.prompt_tokens_details) and the openai-python typesresponses/response_usage.py(InputTokensDetails) andcompletion_usage.py(PromptTokensDetails). Theprompt caching guide
states that writes are part of
input_tokens: ordinary input =input_tokens - cached_tokens - cache_write_tokens.extract_token_statsinnemo_gym/observability/model_calls.pyread only Anthropic's top-levelcache_creation_input_tokens, so every model-call record for an OpenAI model hadcache_creation_tokens: null, evenwhen the provider reported the count (for example
gpt-6-lunareturnsinput_tokens_details: {cached_tokens: 0, cache_write_tokens: 0}on every call).Change: when
cache_creation_input_tokensis absent,extract_token_statsreadscache_write_tokensfrominput_tokens_detailsorprompt_tokens_details._prompt_sizeis not changed: the OpenAI count is a subset of theprompt count and is never added to
tokens_in. The comment onModelCallRecordand the_prompt_sizedocstring nowsay this. A malformed count (not a non-negative integer) stays
None. The model-call capture docs page describesthe per-call cache fields and where they come from.
Other places that read cached tokens:
_cache_signal(cache-read count) already readscached_tokensfrom both detail groups. No change.TrajectoryTokenStats, theng_perftoken fields andaggregate_model_call_recordscarrycached_tokensbut nocache-write count for any provider, including Anthropic. Adding it would add a field to the trajectory schema, a
key to
ng_perfand to the run metrics. Follow-up: carrycache_creation_tokensthroughTrajectoryTokenStats,ng_perf(cache_write_prompt_tokensor similar) andaggregate_model_call_records.nemo_gym/openai_utils.pyNeMoGymResponseInputTokensDetailsand the Chat Completions to Responses usageconversion in
responses_converter.py/litellm_modelbuildinput_tokens_detailswithcached_tokensonly.For OpenAI models served through
openai_modelthis does not matter, because capture stores the providerresponse as returned (see the smoke test). A converted Chat Completions response drops the write count.
Follow-up if converted responses need it.
No issue: found while checking the capture record of a
gpt-6-lunasmoke run.Validation
telemetryextra:pytest tests/unit_tests/test_model_calls.py tests/unit_tests/test_fern_docs_links.py: 83 passed (24 subtests).Nonein both shapes. Converted Chat Completions responses dropping the write count is the follow-up below (GYM-1205).pytest tests/unit_tests/test_rollout_collection.py tests/unit_tests/test_base_responses_api_model.py tests/unit_tests/test_atif_export.py:892 passed, 12 skipped.
_USAGE_SHAPEStable test:responses nothing cached(thegpt-6-lunashape:cache_creation_tokensis 0, notNone),responses cache read and write(the OpenAI docs example: 12000input tokens with 4000 read and 7000 written;
tokens_instays 12000),chat completions cache read and write,and
responses malformed cache write. Without the change tomodel_calls.py, the first three fail and themalformed row passes (it guards against trusting a string count).
ruff check,ruff formatandpre-commit run --fileson the three changed files: clean.Rollout evidence
gym env start --resources-server example_single_tool_call --model-type openai_model ++observability_enabled=true ++model_call_capture_dir=...with
policy_model_name: gpt-6-luna, thengym eval run --no-serve --agent example_single_tool_call_simple_agent --limit 2 --num-repeats 1 --max-output-tokens 512with the same capture settings. 2 rollouts, both reward 1.0, 2 healthy. Each rollout made 2 model calls. All 4
captured call records have
cache_creation_tokens: 0(with the previous code:null) andcached_tokens: 0. The rawcaptured usage is, for example,
{"input_tokens": 66, "input_tokens_details": {"cached_tokens": 0, "cache_write_tokens": 0}}.These prompts are under the 1,024-token caching minimum, so 0 is correct.
To get a nonzero count, two identical calls of about 3,100 tokens were sent directly to the same
openai_modelserver(
POST /v1/responses) and their usage was normalized:input_tokens_detailstokens_incache_creation_tokenscached_tokens{cached_tokens: 0, cache_write_tokens: 3123}{cached_tokens: 3123, cache_write_tokens: 0}tokens_inequalsinput_tokensin both calls, so the write count is a subset and is not added. Chat Completionswas not smoke-tested against a live endpoint; the table test covers it.
Compatibility
cache_creation_tokenson model-call records for OpenAI usage changes fromnullto the reported count. No othertoken field changes,
tokens_inandtokens_totalincluded. Anthropic usage is unchanged: its top-level field takesprecedence. No benchmark scores change.
Linear: FEP-1837 (part of FEP-1829).