Skip to content

Address Auto-Ontology integration gaps in NeMo Gym #4720

Open
3 of 5 issues completed
Open

Description

@anwithk

Problem

Auto-Ontology's evaluation integration exposed gaps in Gym's token accounting, external-client integration, and setup experience. Yuval Shkolar documented the accounting findings and implemented application-side workarounds in Auto-Ontology Eval PR #44. The integration README documents the setup friction.

Track the Gym changes needed to remove those workarounds and support the BIRD-Interact use case. Each child should describe a reusable Gym improvement, with Auto-Ontology as the motivating integration.

Sub-issues

Completion criteria

  • Each child has a documented resolution or explicitly agreed deferral.
  • Token counts and run summaries can be validated against captured usage without application-specific accounting code.
  • External clients can attach calls to the correct rollout without patching global HTTP client methods.
  • A documented external-repository setup covers server installation, CLI configuration, and database preparation.
  • BIRD-Interact has a validated interaction loop, or a documented disposition in Add Support for Interact Benchmark (Multi Turn) #4160.

Out of scope

RL training integration is a separate design follow-up. Capturing model calls or completing interactive evaluation does not by itself establish training readiness.

Activity

  1. yuvalshkolar commented on Oct 19, 2026

    @yuvalshkolar

    Thanks @anwithk !
    I also noticed that embedding model tokens are not counted, and I wonder if support should be added for that as well.

  2. marcromeyn commented on Oct 23, 2026

    @marcromeyn
    Contributor

    Thanks @anwithk for writing these up, and @yuvalshkolar for the detailed findings and workarounds in Auto-Ontology Eval PR #44. I've taken over Gym observability, and all five children now have fixes. I'll open them a few at a time and link each PR here.

    Issue Fix PR
    #4721 cached tokens double-counted The prompt size is decided per usage format: prompt_tokens (Chat Completions) already includes the cache and wins whenever present, so Anthropic-style cache fields beside it are no longer added again. A table test covers every usage shape, including both cases from the issue. #4968
    #4722 calls outside agent turns Every captured call is counted once by the role that owns it (agent, verifier, unattributed) in a new per-rollout ng_captured_calls. ng_perf keeps its meaning and its docs now say it covers agent turns only. On a judge benchmark this shows the judge calls that ng_perf left out. next
    #4723 run-level capture totals Same PR: perf_summary.captured_calls adds the rollout summaries into run totals by role, computed by Gym from small per-rollout summaries (no payloads sent to /aggregate_metrics). Missing usage and runs without capture are reported as such, never as zero. next
    #4724 rollout tagging for external clients Model and resources servers accept an x-ng-rollout-id header in place of the /ng-rollout/<id>/ prefix. SDKs can pass it per call (extra_headers= in the OpenAI SDK); for clients that can't, add_rollout_id_header works as an httpx event hook or in a requests adapter. No global patching. #4969
    #4725 external-repository setup A local server directory with code but no requirements.txt/pyproject.toml is refused with both paths named instead of silently replaced by an installed server; only a pyproject.toml declaring nemo-gym counts as a Gym checkout; --agent explains server type vs instance name. New docs section "Servers in your own repository". #4970

    With these in, the Auto-Ontology integration should be able to drop its verify() usage copying, the cache workaround and the global HTTP patching. Once they merge, it would be great if you could confirm against your setup.

    Two follow-ups:

    • Cache writes: OpenAI now reports cache writes in input_tokens_details.cache_write_tokens / prompt_tokens_details.cache_write_tokens, which Gym didn't read. A small follow-up PR reports them as cache_creation_tokens (they are already inside the prompt count, so not added to it).
    • Embedding tokens (@yuvalshkolar's comment): embedding calls that go through a Gym model server would fit as their own role in ng_captured_calls. Calls made directly to an embedding provider aren't captured. Which applies to your setup?

    @GBX-Max1220, thanks for offering the test matrix on #4721/#4722: the role summary PR covers complete and partial captures, duplicate references, missing usage and capture-disabled runs. Reviews or extra cases on it are very welcome.

    BIRD-Interact (#4160) is tracked separately and isn't part of these changes.

  3. GBX-Max1220 commented on Oct 23, 2026

    @GBX-Max1220

    Thanks @anwithk for writing these up, and @yuvalshkolar for the detailed findings and workarounds in Auto-Ontology Eval PR #44. I've taken over Gym observability, and all five children now have fixes. I'll open them a few at a time and link each PR here.

    Issue Fix PR
    #4721 cached tokens double-counted The prompt size is decided per usage format: prompt_tokens (Chat Completions) already includes the cache and wins whenever present, so Anthropic-style cache fields beside it are no longer added again. A table test covers every usage shape, including both cases from the issue. #4968
    #4722 calls outside agent turns Every captured call is counted once by the role that owns it (agent, verifier, unattributed) in a new per-rollout ng_captured_calls. ng_perf keeps its meaning and its docs now say it covers agent turns only. On a judge benchmark this shows the judge calls that ng_perf left out. next
    #4723 run-level capture totals Same PR: perf_summary.captured_calls adds the rollout summaries into run totals by role, computed by Gym from small per-rollout summaries (no payloads sent to /aggregate_metrics). Missing usage and runs without capture are reported as such, never as zero. next
    #4724 rollout tagging for external clients Model and resources servers accept an x-ng-rollout-id header in place of the /ng-rollout/<id>/ prefix. SDKs can pass it per call (extra_headers= in the OpenAI SDK); for clients that can't, add_rollout_id_header works as an httpx event hook or in a requests adapter. No global patching. #4969
    #4725 external-repository setup A local server directory with code but no requirements.txt/pyproject.toml is refused with both paths named instead of silently replaced by an installed server; only a pyproject.toml declaring nemo-gym counts as a Gym checkout; --agent explains server type vs instance name. New docs section "Servers in your own repository". #4970
    With these in, the Auto-Ontology integration should be able to drop its verify() usage copying, the cache workaround and the global HTTP patching. Once they merge, it would be great if you could confirm against your setup.

    Two follow-ups:

    • Cache writes: OpenAI now reports cache writes in input_tokens_details.cache_write_tokens / prompt_tokens_details.cache_write_tokens, which Gym didn't read. A small follow-up PR reports them as cache_creation_tokens (they are already inside the prompt count, so not added to it).
    • Embedding tokens (@yuvalshkolar's comment): embedding calls that go through a Gym model server would fit as their own role in ng_captured_calls. Calls made directly to an embedding provider aren't captured. Which applies to your setup?

    @GBX-Max1220, thanks for offering the test matrix on #4721/#4722: the role summary PR covers complete and partial captures, duplicate references, missing usage and capture-disabled runs. Reviews or extra cases on it are very welcome.

    BIRD-Interact (#4160) is tracked separately and isn't part of these changes.

    Thanks, Marc—glad to see these moving forward! I’d be happy to review the role-summary PR against the test matrix I proposed, focusing on any gaps beyond the cases you’ve already covered.

    Please link it here when it’s up. I’ll keep the review focused on #4721/#4722 and share any additional cases as small, reproducible examples.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions