Skip to content

Count Auto Ontology's tokens per question through NeMo Gym - #44

Merged
yuvalshkolar merged 2 commits into
mainfrom
gym/count-auto-ontology-tokens
Oct 14, 2026
Merged

yuvalshkolar merged 2 commits into
mainfrom
gym/count-auto-ontology-tokens

Conversation

@yuvalshkolar

Copy link
Copy Markdown
Collaborator

Why

In the Auto Ontology arm, Gym only saw the policy-model call, whose output the arm ignores. Auto Ontology's own ~19 LLM calls per question went straight to the endpoint, so Gym counted none of the tokens a question actually uses.

What changes

  • Route Auto Ontology through Gym (ontology_sql_eval/gym/model_routing.py, arm YAML, run_gym_arm.sh). Two inference_provider model servers stand in for Auto Ontology's reasoning and non-reasoning models. Their endpoint, model and key come from Auto Ontology's own resolve(), exported by run_gym_arm.sh as AUTO_ONTOLOGY_GYM_*. Those variables are plumbing, not new settings, and keep the key out of argv. The resources server points Auto Ontology at these servers before importing it.
  • Tag each call with its question (tag_rollout_requests). Gym files captured calls by the /ng-rollout/<id> URL prefix. Auto Ontology builds clients in many places, using httpx (ChatOpenAI) and requests (ChatNVIDIA), so the prefix is added at the HTTP layer, and only to requests aimed at the routed Gym servers. A call made outside any rollout logs a warning instead of silently going uncounted.
  • Report per-question tokens (captured_usage, verify). verify reads its rollout's captured calls back and returns agent_llm_calls, agent_input_tokens, agent_cached_input_tokens, agent_output_tokens, agent_total_tokens and agent_llm_calls_without_usage. These land on each rollout row and in the key metrics as mean/agent_*, and each question logs one line. The ignored policy call stays separate as mean/input_tokens / mean/output_tokens.
  • Enable capture for both arms: +observability_enabled=true, with captures in <output dir>/model_call_capture/.

Works around a Gym bug

Gym's extract_token_stats (nemo-gym 0.7.0) counts cached prompt tokens twice on the inference API's Claude responses. Their usage is Chat Completions–shaped (prompt_tokens already includes the cache) but also carries Anthropic's top-level cache_*_input_tokens. For example, 2,972 prompt tokens are recorded as 5,940. captured_usage therefore takes each call's input as the provider's total_tokens − completion_tokens. Gym's own ng_model_call_capture.metrics.tokens_in in the rollouts file is still inflated. A report for the Gym team covers this and three related gaps.

Testing

  • pytest tests/ — 59 passed. New tests cover the URL rewrite, both HTTP stacks, idempotence, and usage totals, including the double-counted case.
  • E2E with the NVIDIA API key: run_gym_arm.sh auto_ontology_sql on BIRD bird-1484 (answered correctly). Result: 13 LLM calls, 33,920 in (25,761 cached), 3,004 out, 36,924 total. This matches the sum of the raw provider usage objects in the capture file exactly, and no call missed its rollout tag.

🤖 Generated with Claude Code

The Auto Ontology arm ignores Gym's policy model and makes its own ~19 LLM
calls per question straight to the endpoint, so Gym counted none of them.
Those calls now go through two Gym model servers built from Auto Ontology's
own model resolution, and each request carries its rollout's /ng-rollout/<id>
prefix, so Gym's model-call capture files it under the question.

verify reads the rollout's captured calls back and reports agent_llm_calls,
agent_input_tokens, agent_cached_input_tokens, agent_output_tokens and
agent_total_tokens as rollout fields, which Gym aggregates into mean/agent_*
key metrics. run_gym_arm.sh turns capture on for both arms.

Input tokens are taken as total minus output: Gym's tokens_in double-counts
cached prompt tokens on the inference API's Claude responses, whose usage is
Chat Completions-shaped but also carries Anthropic's cache_*_input_tokens.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves the conflict in auto_ontology_sql/app.py's agent-crash path:
main (#45) now scores it through _without_prediction (BEAVER's official
rules), and this branch records the crash's token usage there. Keep both:
record usage first, since _without_prediction responds through
_response, which reads carried at call time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yuvalshkolar
yuvalshkolar merged commit d78656a into main Oct 14, 2026
2 checks passed
yuvalshkolar added a commit that referenced this pull request Oct 14, 2026
Brings in #44 (Auto Ontology token counting through Gym). No conflicts:
app.py and run_gym_arm.sh merged cleanly, with both this branch's Ray
uv-hook fix and #44's capture and routing blocks in run_gym_arm.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants