You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Address Auto-Ontology integration gaps in NeMo Gym #4720
Auto-Ontology's evaluation integration exposed gaps in Gym's token accounting, external-client integration, and setup experience. Yuval Shkolar documented the accounting findings and implemented application-side workarounds in Auto-Ontology Eval PR #44. The integration README documents the setup friction.
Track the Gym changes needed to remove those workarounds and support the BIRD-Interact use case. Each child should describe a reusable Gym improvement, with Auto-Ontology as the motivating integration.
RL training integration is a separate design follow-up. Capturing model calls or completing interactive evaluation does not by itself establish training readiness.
Thanks @anwithk for writing these up, and @yuvalshkolar for the detailed findings and workarounds in Auto-Ontology Eval PR #44. I've taken over Gym observability, and all five children now have fixes. I'll open them a few at a time and link each PR here.
The prompt size is decided per usage format: prompt_tokens (Chat Completions) already includes the cache and wins whenever present, so Anthropic-style cache fields beside it are no longer added again. A table test covers every usage shape, including both cases from the issue.
Every captured call is counted once by the role that owns it (agent, verifier, unattributed) in a new per-rollout ng_captured_calls. ng_perf keeps its meaning and its docs now say it covers agent turns only. On a judge benchmark this shows the judge calls that ng_perf left out.
Same PR: perf_summary.captured_calls adds the rollout summaries into run totals by role, computed by Gym from small per-rollout summaries (no payloads sent to /aggregate_metrics). Missing usage and runs without capture are reported as such, never as zero.
Model and resources servers accept an x-ng-rollout-id header in place of the /ng-rollout/<id>/ prefix. SDKs can pass it per call (extra_headers= in the OpenAI SDK); for clients that can't, add_rollout_id_header works as an httpx event hook or in a requests adapter. No global patching.
A local server directory with code but no requirements.txt/pyproject.toml is refused with both paths named instead of silently replaced by an installed server; only a pyproject.toml declaring nemo-gym counts as a Gym checkout; --agent explains server type vs instance name. New docs section "Servers in your own repository".
With these in, the Auto-Ontology integration should be able to drop its verify() usage copying, the cache workaround and the global HTTP patching. Once they merge, it would be great if you could confirm against your setup.
Two follow-ups:
Cache writes: OpenAI now reports cache writes in input_tokens_details.cache_write_tokens / prompt_tokens_details.cache_write_tokens, which Gym didn't read. A small follow-up PR reports them as cache_creation_tokens (they are already inside the prompt count, so not added to it).
Embedding tokens (@yuvalshkolar's comment): embedding calls that go through a Gym model server would fit as their own role in ng_captured_calls. Calls made directly to an embedding provider aren't captured. Which applies to your setup?
@GBX-Max1220, thanks for offering the test matrix on #4721/#4722: the role summary PR covers complete and partial captures, duplicate references, missing usage and capture-disabled runs. Reviews or extra cases on it are very welcome.
BIRD-Interact (#4160) is tracked separately and isn't part of these changes.
Thanks @anwithk for writing these up, and @yuvalshkolar for the detailed findings and workarounds in Auto-Ontology Eval PR #44. I've taken over Gym observability, and all five children now have fixes. I'll open them a few at a time and link each PR here.
Issue Fix PR #4721 cached tokens double-counted The prompt size is decided per usage format: prompt_tokens (Chat Completions) already includes the cache and wins whenever present, so Anthropic-style cache fields beside it are no longer added again. A table test covers every usage shape, including both cases from the issue. #4968 #4722 calls outside agent turns Every captured call is counted once by the role that owns it (agent, verifier, unattributed) in a new per-rollout ng_captured_calls. ng_perf keeps its meaning and its docs now say it covers agent turns only. On a judge benchmark this shows the judge calls that ng_perf left out. next #4723 run-level capture totals Same PR: perf_summary.captured_calls adds the rollout summaries into run totals by role, computed by Gym from small per-rollout summaries (no payloads sent to /aggregate_metrics). Missing usage and runs without capture are reported as such, never as zero. next #4724 rollout tagging for external clients Model and resources servers accept an x-ng-rollout-id header in place of the /ng-rollout/<id>/ prefix. SDKs can pass it per call (extra_headers= in the OpenAI SDK); for clients that can't, add_rollout_id_header works as an httpx event hook or in a requests adapter. No global patching. #4969 #4725 external-repository setup A local server directory with code but no requirements.txt/pyproject.toml is refused with both paths named instead of silently replaced by an installed server; only a pyproject.toml declaring nemo-gym counts as a Gym checkout; --agent explains server type vs instance name. New docs section "Servers in your own repository". #4970
With these in, the Auto-Ontology integration should be able to drop its verify() usage copying, the cache workaround and the global HTTP patching. Once they merge, it would be great if you could confirm against your setup.
Two follow-ups:
Cache writes: OpenAI now reports cache writes in input_tokens_details.cache_write_tokens / prompt_tokens_details.cache_write_tokens, which Gym didn't read. A small follow-up PR reports them as cache_creation_tokens (they are already inside the prompt count, so not added to it).
Embedding tokens (@yuvalshkolar's comment): embedding calls that go through a Gym model server would fit as their own role in ng_captured_calls. Calls made directly to an embedding provider aren't captured. Which applies to your setup?
@GBX-Max1220, thanks for offering the test matrix on #4721/#4722: the role summary PR covers complete and partial captures, duplicate references, missing usage and capture-disabled runs. Reviews or extra cases on it are very welcome.
BIRD-Interact (#4160) is tracked separately and isn't part of these changes.
Thanks, Marc—glad to see these moving forward! I’d be happy to review the role-summary PR against the test matrix I proposed, focusing on any gaps beyond the cases you’ve already covered.
Please link it here when it’s up. I’ll keep the review focused on #4721/#4722 and share any additional cases as small, reproducible examples.
Activity
yuvalshkolar commentedon Oct 19, 2026
Thanks @anwithk !
I also noticed that embedding model tokens are not counted, and I wonder if support should be added for that as well.
marcromeyn commentedon Oct 23, 2026
Thanks @anwithk for writing these up, and @yuvalshkolar for the detailed findings and workarounds in Auto-Ontology Eval PR #44. I've taken over Gym observability, and all five children now have fixes. I'll open them a few at a time and link each PR here.
prompt_tokens(Chat Completions) already includes the cache and wins whenever present, so Anthropic-style cache fields beside it are no longer added again. A table test covers every usage shape, including both cases from the issue.agent,verifier,unattributed) in a new per-rolloutng_captured_calls.ng_perfkeeps its meaning and its docs now say it covers agent turns only. On a judge benchmark this shows the judge calls thatng_perfleft out.perf_summary.captured_callsadds the rollout summaries into run totals by role, computed by Gym from small per-rollout summaries (no payloads sent to/aggregate_metrics). Missing usage and runs without capture are reported as such, never as zero.x-ng-rollout-idheader in place of the/ng-rollout/<id>/prefix. SDKs can pass it per call (extra_headers=in the OpenAI SDK); for clients that can't,add_rollout_id_headerworks as an httpx event hook or in a requests adapter. No global patching.requirements.txt/pyproject.tomlis refused with both paths named instead of silently replaced by an installed server; only apyproject.tomldeclaringnemo-gymcounts as a Gym checkout;--agentexplains server type vs instance name. New docs section "Servers in your own repository".With these in, the Auto-Ontology integration should be able to drop its
verify()usage copying, the cache workaround and the global HTTP patching. Once they merge, it would be great if you could confirm against your setup.Two follow-ups:
input_tokens_details.cache_write_tokens/prompt_tokens_details.cache_write_tokens, which Gym didn't read. A small follow-up PR reports them ascache_creation_tokens(they are already inside the prompt count, so not added to it).ng_captured_calls. Calls made directly to an embedding provider aren't captured. Which applies to your setup?@GBX-Max1220, thanks for offering the test matrix on #4721/#4722: the role summary PR covers complete and partial captures, duplicate references, missing usage and capture-disabled runs. Reviews or extra cases on it are very welcome.
BIRD-Interact (#4160) is tracked separately and isn't part of these changes.
GBX-Max1220 commentedon Oct 23, 2026
Thanks, Marc—glad to see these moving forward! I’d be happy to review the role-summary PR against the test matrix I proposed, focusing on any gaps beyond the cases you’ve already covered.
Please link it here when it’s up. I’ll keep the review focused on #4721/#4722 and share any additional cases as small, reproducible examples.