Repository navigation
Count Auto Ontology's tokens per question through NeMo Gym - #44
Merged
Merged
Conversation
The Auto Ontology arm ignores Gym's policy model and makes its own ~19 LLM calls per question straight to the endpoint, so Gym counted none of them. Those calls now go through two Gym model servers built from Auto Ontology's own model resolution, and each request carries its rollout's /ng-rollout/<id> prefix, so Gym's model-call capture files it under the question. verify reads the rollout's captured calls back and reports agent_llm_calls, agent_input_tokens, agent_cached_input_tokens, agent_output_tokens and agent_total_tokens as rollout fields, which Gym aggregates into mean/agent_* key metrics. run_gym_arm.sh turns capture on for both arms. Input tokens are taken as total minus output: Gym's tokens_in double-counts cached prompt tokens on the inference API's Claude responses, whose usage is Chat Completions-shaped but also carries Anthropic's cache_*_input_tokens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
polazilber
approved these changes
Oct 14, 2026
Resolves the conflict in auto_ontology_sql/app.py's agent-crash path: main (#45) now scores it through _without_prediction (BEAVER's official rules), and this branch records the crash's token usage there. Keep both: record usage first, since _without_prediction responds through _response, which reads carried at call time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
liavnave
approved these changes
Oct 14, 2026
This was referenced Oct 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
In the Auto Ontology arm, Gym only saw the policy-model call, whose output the arm ignores. Auto Ontology's own ~19 LLM calls per question went straight to the endpoint, so Gym counted none of the tokens a question actually uses.
What changes
ontology_sql_eval/gym/model_routing.py, arm YAML,run_gym_arm.sh). Twoinference_providermodel servers stand in for Auto Ontology's reasoning and non-reasoning models. Their endpoint, model and key come from Auto Ontology's ownresolve(), exported byrun_gym_arm.shasAUTO_ONTOLOGY_GYM_*. Those variables are plumbing, not new settings, and keep the key out of argv. The resources server points Auto Ontology at these servers before importing it.tag_rollout_requests). Gym files captured calls by the/ng-rollout/<id>URL prefix. Auto Ontology builds clients in many places, using httpx (ChatOpenAI) and requests (ChatNVIDIA), so the prefix is added at the HTTP layer, and only to requests aimed at the routed Gym servers. A call made outside any rollout logs a warning instead of silently going uncounted.captured_usage,verify).verifyreads its rollout's captured calls back and returnsagent_llm_calls,agent_input_tokens,agent_cached_input_tokens,agent_output_tokens,agent_total_tokensandagent_llm_calls_without_usage. These land on each rollout row and in the key metrics asmean/agent_*, and each question logs one line. The ignored policy call stays separate asmean/input_tokens/mean/output_tokens.+observability_enabled=true, with captures in<output dir>/model_call_capture/.Works around a Gym bug
Gym's
extract_token_stats(nemo-gym 0.7.0) counts cached prompt tokens twice on the inference API's Claude responses. Their usage is Chat Completions–shaped (prompt_tokensalready includes the cache) but also carries Anthropic's top-levelcache_*_input_tokens. For example, 2,972 prompt tokens are recorded as 5,940.captured_usagetherefore takes each call's input as the provider'stotal_tokens − completion_tokens. Gym's ownng_model_call_capture.metrics.tokens_inin the rollouts file is still inflated. A report for the Gym team covers this and three related gaps.Testing
pytest tests/— 59 passed. New tests cover the URL rewrite, both HTTP stacks, idempotence, and usage totals, including the double-counted case.run_gym_arm.sh auto_ontology_sqlon BIRDbird-1484(answered correctly). Result: 13 LLM calls, 33,920 in (25,761 cached), 3,004 out, 36,924 total. This matches the sum of the raw providerusageobjects in the capture file exactly, and no call missed its rollout tag.🤖 Generated with Claude Code