Skip to content

perf(rollout): validate the token-capture config once per run - #4638

Merged
ananthsub merged 1 commit into
mainfrom
ananthsub/token-capture-config-once
Oct 27, 2026
Merged

ananthsub merged 1 commit into
mainfrom
ananthsub/token-capture-config-once

Conversation

@ananthsub

@ananthsub ananthsub commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

What changed and why

RolloutCollectionHelper.run_from_config (the gym eval run path) decides whether each rollout's agent participates in token capture by calling token_id_capture_enabled_for_agent. That function called token_id_capture_config, which runs TokenIdCaptureConfig.model_validate on the whole global config, an OmegaConf DictConfig. The collector made this call once per input row before dispatch, to choose which rows' stale token captures to clear, and again for every finished rollout before finalize_rollout_token_capture.

The global config does not change during a run, and #2126 describes participation as coming from static token_id_capture configuration, so every call after the first repeated the same validation on the collector's event loop. A CPU profile of the driver in a simulated production topology with 1,024 rollouts in flight attributed about 8% of the event loop's busy time to this validation and the OmegaConf access it performs.

The fix:

  • token_id_capture_enabled_for_agent accepts an optional keyword-only config, which a caller passes when it has already validated the global config. Callers that omit it behave exactly as before.
  • run_from_config passes the TokenIdCaptureConfig it already validates before dispatch, and remembers each agent's participation decision for the rest of the run. The per-row and per-rollout checks therefore read the global config only for the first row of each agent.

Participation decisions are unchanged because they depend only on the agent name and the static global config. Error behavior for an invalid config is unchanged because run_from_config already validated the config before the per-row loop, so an invalid config still fails at that point before any dispatch.

No issue is linked because the change is a self-contained performance fix with no behavior change.

Tests

test_run_from_config_validates_the_token_capture_config_a_constant_number_of_times counts calls to TokenIdCaptureConfig.model_validate during a run with one participating agent and one ordinary agent. It runs a small input and a much larger one and requires the same number of validations for both, and it requires the finalizer to see exactly the participating agent's rollouts. On main the count grows with the number of rows, so the test fails there.

Validation commands:

uv sync --extra dev --extra telemetry
PYTHONPATH=. .venv/bin/python -m pytest -q tests/unit_tests/test_rollout_collection.py -k "constant_number or nonparticipating"
PYTHONPATH=. .venv/bin/python -m pytest -n 16 -q tests/unit_tests --ignore=tests/unit_tests/test_exporters.py
pre-commit run --files nemo_gym/rollout_collection.py nemo_gym/token_id_capture/config.py tests/unit_tests/test_rollout_collection.py

The full unit suite passes apart from environment-dependent failures in test_opensandbox_compose.py, test_reward_profile.py, test_dataset_orchestrator.py, and test_sandbox_stop_retry.py, which fail the same way on main on the machine used.

Simulator comparison

The CPU capacity simulator ran the production-topology profile with 4,096 rollouts and 1,024 in flight against main (4435c6d458) and this branch (c40fb6cb01), with the driver pinned to two cores and sampled by py-spy. Within the samples taken on the driver's event loop, token_id_capture_config and the model_validate call under it accounted for 9.7% on main and for no samples on this branch. The remaining OmegaConf reads on the driver's event loop come from ServerClient.request and are not touched by this change.

Each variant ran once, so the run-level numbers are indicative only: throughput went from 99.3 to 119.8 rollouts per second, and driver CPU utilization went from 0.41 to 0.24 cores. Part of the driver utilization difference comes from module import time at startup, which differed between the two runs.

Rollout evidence

N/A. The change removes repeated validation in the rollout collector and does not change any agent, model, or verifier decision, which rows are cleared, or which rollouts are finalized.

run_from_config decides whether each rollout's agent participates in token capture by calling
token_id_capture_enabled_for_agent. That function validated the whole global config into a
TokenIdCaptureConfig on every call, and the collector made the call once per input row before
dispatch and again for every finished rollout. The global config does not change during a run,
so every call after the first repeated the same validation on the collector's event loop.

token_id_capture_enabled_for_agent now takes an optional keyword-only config, which callers
pass when they already validated the global config. run_from_config passes the config it already
validates before dispatch, and remembers each agent's decision for the rest of the run, so the
per-row and per-rollout checks no longer touch the global config after the first row of each
agent. Invalid configs still fail at the same point, because run_from_config validated the
config before the per-row loop already.

The new test runs a small and a large input with one participating and one ordinary agent. It
requires the same number of validations for both runs and requires the finalizer to see exactly
the participating agent's rollouts.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@ananthsub ananthsub added feature New capabilities, enhancements, or enablement work area:core Shared APIs, servers, telemetry, health, and registries needs-review PR is ready for code review and waiting on a reviewer complexity:low Localized change in one scope with a small, straightforward review surface labels Oct 15, 2026
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 15, 2026 16:59
@ananthsub
ananthsub requested a review from pthombre October 15, 2026 17:34
@ananthsub
ananthsub requested a review from marta-sd October 16, 2026 01:26
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Oct 16, 2026
@ananthsub
ananthsub added this pull request to the merge queue Oct 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40512902022

Merged via the queue into main with commit af9d911 Oct 27, 2026
52 of 57 checks passed
@ananthsub
ananthsub deleted the ananthsub/token-capture-config-once branch October 27, 2026 14:07
yuhezhang-ai added a commit that referenced this pull request Oct 27, 2026
#4841 adds run.stats.timeouts to the run health summary:
- Run accounting keeps #3302's split: a run with a manifest is counted from
  its history, a legacy output from its sidecar. The legacy branch now reads
  the sidecar through #4841's _read_failures.
- The timeouts section reads each never-landed rollout's failed attempt from
  the same source: the attempt the run history selects, or the legacy
  sidecar's last row.
- The built-in collector still writes the summary under its run lock, now
  with the run's model-call capture dirs, and exports the section;
  gym eval run and collect only check custom drivers, as before.
- Run timing classifies the record collection saves, after #3302's
  native episode validation.

#4638's per-run token-capture check replaces the per-row lookup, including
for recoverable judge failures.

Tests: #5081's accounting tests call #3302's _rollout_accounting signature;
#4841's collection fakes return a response, its aggregation recomputes in
place (a run is never merged onto itself), and #4638's test stubs the
server client #3302's collector fingerprints.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>

This branch was previously deployed

1 inactive deployment
public — c40fb6cb Deployed Oct 15, 2026 by copy-pr-bot[bot] via release / finalize / notify #5529
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries complexity:low Localized change in one scope with a small, straightforward review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants