Repository navigation
feat(nemo-gym): opt-in truncation of non-contiguous rollout episodes - #3063
Conversation
|
/ok to test c0075c4 |
c0075c4 to
ce38185
Compare
|
/ok to test ce38185 |
|
/ok to test 3bddb06 |
3bddb06 to
6b2eecc
Compare
|
/ok to test |
@Kh4L, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/ |
|
/ok to test 6b2eecc |
|
/ok to test 6b2eecc |
6b2eecc to
7dea203
Compare
|
/ok to test |
@Kh4L, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/ |
7dea203 to
678dfc2
Compare
|
/ok to test 678dfc2 |
678dfc2 to
4dac1b5
Compare
|
/ok to test 4dac1b5 |
4dac1b5 to
e448060
Compare
|
/ok to test e448060 |
e448060 to
60140cd
Compare
@Kh4L, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
|
/ok to test 60140cd |
|
/ok to test 9468b2b |
9468b2b to
60140cd
Compare
|
/ok to test 60140cd |
|
Exact-head stacked hardware smoke passed.
The small deterministic sample did not hit a non-contiguous episode, so the 16-node run in the PR body remains the branch-behavior evidence; this run validates the exact stacked integration and no-regression path. |
60140cd to
5d09ed9
Compare
Signed-off-by: Serge Panev <spanev@nvidia.com>
|
/ok to test b720a03 |
yuki-97
left a comment
There was a problem hiding this comment.
Opt-in truncation of non-contiguous Gym episodes is a welcome escape hatch: a single bad multi-turn trajectory no longer stalls async GRPO, and the default keeps today's hard failure.
Scope: full diff at b720a03: the truncation path in _postprocess_nemo_gym_to_nemo_rl_result, the new runtime-option model and its validation, and the GRPO / single-controller / distillation / prefetch call sites.
Checked:
- Default-off behaviour: with the option unset the contiguity
assertand its message are unchanged (nemo_gym.py:1287). - Truncation semantics: the
breakdrops exactly the divergent turn and everything after it, and since the first trainable turn is always contiguous, the retained prefix is never empty, so the no-generation-data error below cannot fire on this path. - Putting the default on a BaseModel rather than at the call site is the right shape for a key that lives in the untyped
env.nemo_gymdict, and the three keys are kept out of Gym's global config on both the unsharded and sharded paths. - Incompatibility set: router replay and
token_captureare the right two exclusions. The retained prefix's final route can only be repaired by the next turn's prefill (nemo_gym.py:1340-1345), and receipt mode never reaches the token-prefix check.
Remaining comments are on keeping the truncation plumbing in _build_gym_actor_config alone (dropping the distillation and prefetch copies), surfacing truncation as a per-agent metric, and removing a runtime route check that the build-time validator already covers.
Signed-off-by: Serge Panev <spanev@nvidia.com>
Signed-off-by: Serge Panev <spanev@nvidia.com>
Retain structured reasoning and answer token counts alongside the truncation flag. Exercise both truncation settings across the structured-response extraction cases. Signed-off-by: Serge Panev <spanev@nvidia.com>
yuki-97
left a comment
There was a problem hiding this comment.
This PR adds an opt-in that keeps the contiguous prefix of an episode when a later turn's token prefix diverges, while the description frames it as fixing a stall in async GRPO, which no longer happens on main: the legacy path aborts on the first batch-worker failure and the SC path re-raises once the data attempts are used up.
It would be good to update the description to say that the option tolerates this one benign case without raising the generic failure budgets, but this does not need to block the merge.
|
/ok to test bee756d |
Pull request was closed
|
🤦♂️ closed by mistake |
Signed-off-by: Serge Panev <spanev@nvidia.com>
Signed-off-by: Serge Panev <spanev@nvidia.com>
|
/ok to test aff15cc |
Signed-off-by: Serge Panev <spanev@nvidia.com>
|
/ok to test 36fd4cf |
Signed-off-by: Serge Panev <spanev@nvidia.com>
|
/ok to test 5b46766 |
Signed-off-by: Serge Panev <spanev@nvidia.com>
|
/ok to test d156070 |
Bring in the previous-release-tag fix from NVIDIA-NeMo#5144 so release validation does not generate an oversized changelog from v0.1.0. Resolve the Gym integration documentation conflict while preserving the truncation option and current main formatting. Signed-off-by: fujial <fujial@nvidia.com>
|
/ok to test 99b54fb |
yuki-97
left a comment
There was a problem hiding this comment.
Thanks for addressing the previous review comments.
Scope: Changes from 13c931e through 99b54fb, including the main merges and regression checks.
Checked:
- The description now clearly scopes this as opt-in tolerance for prefix divergence, without claiming to fix a current async-GRPO stall.
- Configuration propagation and replay/capture guards, retained-prefix tokens and logprobs, episode reward preservation, and truncation metrics remain consistent after the main merges.
- Relevant environment, rollout, and distillation regression cases passed in the current-head CI logs; all 96 checks are successful or skipped (85 successful, 11 skipped).
LGTM — no remaining findings from this re-review.
Generated by Codex
Problem
_postprocess_nemo_gym_to_nemo_rl_resultasserts that each assistant turn's accumulated token IDs form a prefix of the next turn's prompt. In long multi-turn agentic rollouts, rare tokenization or re-rendering edge cases can break contiguity and fail the rollout task.Current main already propagates these failures: the legacy path aborts on the first batch-worker failure, and the SC path re-raises after exhausting its data attempts. This PR does not fix a current async-GRPO stall. It adds an opt-in way to tolerate this one benign prefix-divergence case without increasing generic failure budgets.
Change
Add the opt-in
env.nemo_gym.truncate_noncontiguous_episodesoption. It defaults tofalse, preserving the existing assertion. When enabled, a non-contiguous turn drops the corrupted tail and returns the last contiguous prefix; that prefix remains trainable with the episode reward.NemoGymPostprocessConfigowns the defaults. The shared_build_gym_actor_configconsumes these fields and validates compatibility after each shard's overrides are merged, without forwarding NeMo-RL-only keys into Gym or mutating the caller's config. The redundant distillation check and dry-run prefetch plumbing have been removed.Router replay and token-capture receipt mode are rejected with this option during actor-config construction. Router replay needs the next contiguous prompt to repair the retained prefix's final route; receipt mode bypasses the legacy token-prefix check.
Every legacy result includes
noncontiguous_truncated(falsenormally,truewhen the tail is dropped), surfaced through the existing per-agent scalar metrics as{agent_name}/noncontiguous_truncated/mean.This PR remains independent of #3127 and the SGLang/quantization stacks.
Main refresh
Current head
5b46766b4496e7fc284c1ae3ae07004a1d2c6612merges mainefc309512601fef2a219c47240651ec130f2709b. The conflict resolution keeps the postprocessing configuration alongside main's new Environment Server task routing and episode-validation helpers. Postprocessing uses main's task-input lookup for materialized tasks; the truncation behavior is preserved.Validation
These are supplementary macOS CPU checks with project conftest/config disabled; they do not establish native serving or full current-lock validation. Fresh project CI is requested for this commit. The passing full workflow on the previous head and its passing patch coverage do not validate this new main merge.
Historical evidence — not rerun for this head
The runs below predate the current head and do not validate the latest main merge or review fixes. Their observed stall behavior describes the historical code used for those runs, not current main.
Validated at scale on a 16-node Qwen3-30B MoE SWE-bench async-GRPO run with an SGLang generation backend: 63 training steps across four clean checkpoint/resume segments, about 200 graceful truncations, zero stalls, and reward convergence from 0.10 to a 0.22 plateau. Without the option, single corrupted episodes reproducibly stalled the step until walltime on that historical code.
Stacked HSG job
6595733also completed successfully (0:0) on 2 nodes / 8 GB200 GPUs with Qwen2.5-Math-1.5B, 4-GPU Megatron training, four TP1 SGLang engines, and real stateful multi-turn NeMo-Gym async GRPO. It validated source60140cd23496d80c3ae04cd3a5fc276c38d44a60, which contains the same #3063 feature increment as the direct-main head. This is integration evidence rather than exact-current-head evidence.The strict validator passed 3/3 optimizer steps, 4/4 SGLang refits, and 24/24 trainable trajectory rows with nonzero token-loss masks.
truncate_noncontiguous_episodes=truewas resolved and forwarded correctly. This small deterministic sample did not encounter a non-contiguous episode, so the 16-node run above remains the branch-behavior evidence.Tests