Repository navigation
perf(rollout): validate the token-capture config once per run - #4638
Merged
Merged
Conversation
run_from_config decides whether each rollout's agent participates in token capture by calling token_id_capture_enabled_for_agent. That function validated the whole global config into a TokenIdCaptureConfig on every call, and the collector made the call once per input row before dispatch and again for every finished rollout. The global config does not change during a run, so every call after the first repeated the same validation on the collector's event loop. token_id_capture_enabled_for_agent now takes an optional keyword-only config, which callers pass when they already validated the global config. run_from_config passes the config it already validates before dispatch, and remembers each agent's decision for the rest of the run, so the per-row and per-rollout checks no longer touch the global config after the first row of each agent. Invalid configs still fail at the same point, because run_from_config validated the config before the per-row loop already. The new test runs a small and a large input with one participating and one ordinary agent. It requires the same number of validations for both runs and requires the finalizer to see exactly the participating agent's rollouts. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
This was referenced Oct 16, 2026
marta-sd
approved these changes
Oct 27, 2026
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40512902022 |
yuhezhang-ai
added a commit
that referenced
this pull request
Oct 27, 2026
#4841 adds run.stats.timeouts to the run health summary: - Run accounting keeps #3302's split: a run with a manifest is counted from its history, a legacy output from its sidecar. The legacy branch now reads the sidecar through #4841's _read_failures. - The timeouts section reads each never-landed rollout's failed attempt from the same source: the attempt the run history selects, or the legacy sidecar's last row. - The built-in collector still writes the summary under its run lock, now with the run's model-call capture dirs, and exports the section; gym eval run and collect only check custom drivers, as before. - Run timing classifies the record collection saves, after #3302's native episode validation. #4638's per-run token-capture check replaces the per-row lookup, including for recoverable judge failures. Tests: #5081's accounting tests call #3302's _rollout_accounting signature; #4841's collection fakes return a response, its aggregation recomputes in place (a run is never merged onto itself), and #4638's test stubs the server client #3302's collector fingerprints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed and why
RolloutCollectionHelper.run_from_config(thegym eval runpath) decides whether each rollout's agent participates in token capture by callingtoken_id_capture_enabled_for_agent. That function calledtoken_id_capture_config, which runsTokenIdCaptureConfig.model_validateon the whole global config, an OmegaConfDictConfig. The collector made this call once per input row before dispatch, to choose which rows' stale token captures to clear, and again for every finished rollout beforefinalize_rollout_token_capture.The global config does not change during a run, and #2126 describes participation as coming from static
token_id_captureconfiguration, so every call after the first repeated the same validation on the collector's event loop. A CPU profile of the driver in a simulated production topology with 1,024 rollouts in flight attributed about 8% of the event loop's busy time to this validation and the OmegaConf access it performs.The fix:
token_id_capture_enabled_for_agentaccepts an optional keyword-onlyconfig, which a caller passes when it has already validated the global config. Callers that omit it behave exactly as before.run_from_configpasses theTokenIdCaptureConfigit already validates before dispatch, and remembers each agent's participation decision for the rest of the run. The per-row and per-rollout checks therefore read the global config only for the first row of each agent.Participation decisions are unchanged because they depend only on the agent name and the static global config. Error behavior for an invalid config is unchanged because
run_from_configalready validated the config before the per-row loop, so an invalid config still fails at that point before any dispatch.No issue is linked because the change is a self-contained performance fix with no behavior change.
Tests
test_run_from_config_validates_the_token_capture_config_a_constant_number_of_timescounts calls toTokenIdCaptureConfig.model_validateduring a run with one participating agent and one ordinary agent. It runs a small input and a much larger one and requires the same number of validations for both, and it requires the finalizer to see exactly the participating agent's rollouts. Onmainthe count grows with the number of rows, so the test fails there.Validation commands:
uv sync --extra dev --extra telemetry PYTHONPATH=. .venv/bin/python -m pytest -q tests/unit_tests/test_rollout_collection.py -k "constant_number or nonparticipating" PYTHONPATH=. .venv/bin/python -m pytest -n 16 -q tests/unit_tests --ignore=tests/unit_tests/test_exporters.py pre-commit run --files nemo_gym/rollout_collection.py nemo_gym/token_id_capture/config.py tests/unit_tests/test_rollout_collection.pyThe full unit suite passes apart from environment-dependent failures in
test_opensandbox_compose.py,test_reward_profile.py,test_dataset_orchestrator.py, andtest_sandbox_stop_retry.py, which fail the same way onmainon the machine used.Simulator comparison
The CPU capacity simulator ran the production-topology profile with 4,096 rollouts and 1,024 in flight against
main(4435c6d458) and this branch (c40fb6cb01), with the driver pinned to two cores and sampled bypy-spy. Within the samples taken on the driver's event loop,token_id_capture_configand themodel_validatecall under it accounted for 9.7% onmainand for no samples on this branch. The remaining OmegaConf reads on the driver's event loop come fromServerClient.requestand are not touched by this change.Each variant ran once, so the run-level numbers are indicative only: throughput went from 99.3 to 119.8 rollouts per second, and driver CPU utilization went from 0.41 to 0.24 cores. Part of the driver utilization difference comes from module import time at startup, which differed between the two runs.
Rollout evidence
N/A. The change removes repeated validation in the rollout collector and does not change any agent, model, or verifier decision, which rows are cleared, or which rollouts are finalized.