Skip to content

feat(nemo-gym): opt-in truncation of non-contiguous rollout episodes - #3063

Merged
yuki-97 merged 15 commits into
NVIDIA-NeMo:mainfrom
Kh4L:nemo-gym-noncontig-truncate
Nov 17, 2026
Merged

yuki-97 merged 15 commits into
NVIDIA-NeMo:mainfrom
Kh4L:nemo-gym-noncontig-truncate

Conversation

@Kh4L

@Kh4L Kh4L commented Jul 4, 2026 •

Copy link
Copy Markdown
Contributor

Problem

_postprocess_nemo_gym_to_nemo_rl_result asserts that each assistant turn's accumulated token IDs form a prefix of the next turn's prompt. In long multi-turn agentic rollouts, rare tokenization or re-rendering edge cases can break contiguity and fail the rollout task.

Current main already propagates these failures: the legacy path aborts on the first batch-worker failure, and the SC path re-raises after exhausting its data attempts. This PR does not fix a current async-GRPO stall. It adds an opt-in way to tolerate this one benign prefix-divergence case without increasing generic failure budgets.

Change

Add the opt-in env.nemo_gym.truncate_noncontiguous_episodes option. It defaults to false, preserving the existing assertion. When enabled, a non-contiguous turn drops the corrupted tail and returns the last contiguous prefix; that prefix remains trainable with the episode reward.

NemoGymPostprocessConfig owns the defaults. The shared _build_gym_actor_config consumes these fields and validates compatibility after each shard's overrides are merged, without forwarding NeMo-RL-only keys into Gym or mutating the caller's config. The redundant distillation check and dry-run prefetch plumbing have been removed.

Router replay and token-capture receipt mode are rejected with this option during actor-config construction. Router replay needs the next contiguous prompt to repair the retained prefix's final route; receipt mode bypasses the legacy token-prefix check.

Every legacy result includes noncontiguous_truncated (false normally, true when the tail is dropped), surfaced through the existing per-agent scalar metrics as {agent_name}/noncontiguous_truncated/mean.

This PR remains independent of #3127 and the SGLang/quantization stacks.

Main refresh

Current head 5b46766b4496e7fc284c1ae3ae07004a1d2c6612 merges main efc309512601fef2a219c47240651ec130f2709b. The conflict resolution keeps the postprocessing configuration alongside main's new Environment Server task routing and episode-validation helpers. Postprocessing uses main's task-input lookup for materialized tasks; the truncation behavior is preserved.

Validation

  • 246 normal-import CPU tests pass, covering Gym configuration, task inputs and sharding. Three optional Gym-package cases are deselected because that package is unavailable in the local CPU environment.
  • The exact existing router-replay rejection test reproduces the former missing-keyword error before the test fix and reaches its expected rejection on this head, using the normal production-module import.
  • All 96 upstream Gym definitions are retained. The postprocessor differs from the preceding CI-tested head only in main's task-input lookup.
  • Pinned Ruff 0.9.9 lint/import sorting and formatting, Python syntax, and feature diff checks pass.

These are supplementary macOS CPU checks with project conftest/config disabled; they do not establish native serving or full current-lock validation. Fresh project CI is requested for this commit. The passing full workflow on the previous head and its passing patch coverage do not validate this new main merge.

Historical evidence — not rerun for this head

The runs below predate the current head and do not validate the latest main merge or review fixes. Their observed stall behavior describes the historical code used for those runs, not current main.

Validated at scale on a 16-node Qwen3-30B MoE SWE-bench async-GRPO run with an SGLang generation backend: 63 training steps across four clean checkpoint/resume segments, about 200 graceful truncations, zero stalls, and reward convergence from 0.10 to a 0.22 plateau. Without the option, single corrupted episodes reproducibly stalled the step until walltime on that historical code.

Stacked HSG job 6595733 also completed successfully (0:0) on 2 nodes / 8 GB200 GPUs with Qwen2.5-Math-1.5B, 4-GPU Megatron training, four TP1 SGLang engines, and real stateful multi-turn NeMo-Gym async GRPO. It validated source 60140cd23496d80c3ae04cd3a5fc276c38d44a60, which contains the same #3063 feature increment as the direct-main head. This is integration evidence rather than exact-current-head evidence.

The strict validator passed 3/3 optimizer steps, 4/4 SGLang refits, and 24/24 trainable trajectory rows with nonzero token-loss masks. truncate_noncontiguous_episodes=true was resolved and forwarded correctly. This small deterministic sample did not encounter a non-contiguous episode, so the 16-node run above remains the branch-behavior evidence.

Tests

  • Default-off discontinuities still assert; enabled truncation retains exactly the valid prefix and original reward.
  • Absent/false/true option resolution and nonmutation of caller/Gym configuration.
  • Router-replay and token-capture incompatibility checks after common and per-shard overrides.
  • Truncation metric on every result, both mixed-batch orders, and stale-flag reset.
  • Preservation of main's policy API-key, external-service readiness handling and structured token extraction.

@copy-pr-bot

copy-pr-bot Bot commented Jul 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Kh4L

Kh4L commented Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test c0075c4

@Kh4L

Kh4L commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test ce38185

@Kh4L

Kh4L commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 3bddb06

@Kh4L
Kh4L force-pushed the nemo-gym-noncontig-truncate branch from 3bddb06 to 6b2eecc Compare August 27, 2026 07:37
@Kh4L

Kh4L commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@copy-pr-bot

copy-pr-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown

/ok to test

@Kh4L, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

@Kh4L

Kh4L commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 6b2eecc

@Kh4L Kh4L added the CI:L1 Run doctests, unit tests, and functional tests label Aug 27, 2026
@Kh4L

Kh4L commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 6b2eecc

@Kh4L
Kh4L force-pushed the nemo-gym-noncontig-truncate branch from 6b2eecc to 7dea203 Compare August 27, 2026 11:18
@Kh4L

Kh4L commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test

@copy-pr-bot

copy-pr-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown

/ok to test

@Kh4L, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

@Kh4L
Kh4L force-pushed the nemo-gym-noncontig-truncate branch from 7dea203 to 678dfc2 Compare August 27, 2026 12:01
@Kh4L

Kh4L commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 678dfc2

@Kh4L
Kh4L force-pushed the nemo-gym-noncontig-truncate branch from 678dfc2 to 4dac1b5 Compare August 27, 2026 13:14
@Kh4L

Kh4L commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 4dac1b5

@Kh4L
Kh4L force-pushed the nemo-gym-noncontig-truncate branch from 4dac1b5 to e448060 Compare August 27, 2026 14:46
@Kh4L

Kh4L commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e448060

@Kh4L
Kh4L force-pushed the nemo-gym-noncontig-truncate branch from e448060 to 60140cd Compare August 27, 2026 14:55
@copy-pr-bot

copy-pr-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown

/ok to test 60140cd23db2ca93d3e3d98957b9370e364e35e4

@Kh4L, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

@Kh4L

Kh4L commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 60140cd

@Kh4L

Kh4L commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 9468b2b

@Kh4L
Kh4L force-pushed the nemo-gym-noncontig-truncate branch from 9468b2b to 60140cd Compare August 27, 2026 15:24
@Kh4L

Kh4L commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 60140cd

@Kh4L

Kh4L commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Exact-head stacked hardware smoke passed.

  • HSG job 6595733: COMPLETED 0:0, 2 nodes / 8 GB200 GPUs
  • This exact head: 60140cd23496d80c3ae04cd3a5fc276c38d44a60
  • 3/3 async-GRPO optimizer steps, 4/4 SGLang refits, and 24/24 trainable four-turn Gym trajectories
  • truncate_noncontiguous_episodes=true resolved and forwarded correctly

The small deterministic sample did not hit a non-contiguous episode, so the 16-node run in the PR body remains the branch-behavior evidence; this run validates the exact stacked integration and no-regression path.

@Kh4L
Kh4L force-pushed the nemo-gym-noncontig-truncate branch from 60140cd to 5d09ed9 Compare August 27, 2026 16:45
Signed-off-by: Serge Panev <spanev@nvidia.com>
@Kh4L

Kh4L commented Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b720a03

@yuki-97 yuki-97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opt-in truncation of non-contiguous Gym episodes is a welcome escape hatch: a single bad multi-turn trajectory no longer stalls async GRPO, and the default keeps today's hard failure.

Scope: full diff at b720a03: the truncation path in _postprocess_nemo_gym_to_nemo_rl_result, the new runtime-option model and its validation, and the GRPO / single-controller / distillation / prefetch call sites.

Checked:

  • Default-off behaviour: with the option unset the contiguity assert and its message are unchanged (nemo_gym.py:1287).
  • Truncation semantics: the break drops exactly the divergent turn and everything after it, and since the first trainable turn is always contiguous, the retained prefix is never empty, so the no-generation-data error below cannot fire on this path.
  • Putting the default on a BaseModel rather than at the call site is the right shape for a key that lives in the untyped env.nemo_gym dict, and the three keys are kept out of Gym's global config on both the unsharded and sharded paths.
  • Incompatibility set: router replay and token_capture are the right two exclusions. The retained prefix's final route can only be repaired by the next turn's prefill (nemo_gym.py:1340-1345), and receipt mode never reaches the token-prefix check.

Remaining comments are on keeping the truncation plumbing in _build_gym_actor_config alone (dropping the distillation and prefetch copies), surfacing truncation as a per-agent metric, and removing a runtime route check that the build-time validator already covers.

Comment thread nemo_rl/environments/nemo_gym.py Resolved
Comment thread nemo_rl/environments/nemo_gym.py Resolved
Comment thread nemo_rl/environments/nemo_gym.py Resolved
Kh4L added 3 commits October 15, 2026 18:40
Signed-off-by: Serge Panev <spanev@nvidia.com>
Signed-off-by: Serge Panev <spanev@nvidia.com>
Retain structured reasoning and answer token counts alongside the truncation flag. Exercise both truncation settings across the structured-response extraction cases.

Signed-off-by: Serge Panev <spanev@nvidia.com>
@yuki-97 yuki-97 added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:L1 Run doctests, unit tests, and functional tests labels Oct 19, 2026
yuki-97
yuki-97 previously approved these changes Oct 19, 2026

@yuki-97 yuki-97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds an opt-in that keeps the contiguous prefix of an episode when a later turn's token prefix diverges, while the description frames it as fixing a stall in async GRPO, which no longer happens on main: the legacy path aborts on the first batch-worker failure and the SC path re-raises once the data attempts are used up.
It would be good to update the description to say that the option tolerates this one benign case without raising the generic failure budgets, but this does not need to block the merge.

@yuki-97

yuki-97 commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

/ok to test bee756d

@yuki-97
yuki-97 enabled auto-merge (squash) October 19, 2026 06:46
@Kh4L Kh4L closed this Oct 19, 2026
auto-merge was automatically disabled October 19, 2026 18:59

Pull request was closed

@Kh4L Kh4L reopened this Oct 19, 2026
@Kh4L

Kh4L commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

🤦‍♂️ closed by mistake

Signed-off-by: Serge Panev <spanev@nvidia.com>
Signed-off-by: Serge Panev <spanev@nvidia.com>
@Kh4L

Kh4L commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test aff15cc

Signed-off-by: Serge Panev <spanev@nvidia.com>
@Kh4L

Kh4L commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 36fd4cf

Signed-off-by: Serge Panev <spanev@nvidia.com>
@Kh4L

Kh4L commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 5b46766

Signed-off-by: Serge Panev <spanev@nvidia.com>
@Kh4L

Kh4L commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test d156070

Bring in the previous-release-tag fix from NVIDIA-NeMo#5144 so release validation does not generate an oversized changelog from v0.1.0. Resolve the Gym integration documentation conflict while preserving the truncation option and current main formatting.

Signed-off-by: fujial <fujial@nvidia.com>
@fujial-code

Copy link
Copy Markdown
Contributor

/ok to test 99b54fb

@yuki-97 yuki-97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the previous review comments.

Scope: Changes from 13c931e through 99b54fb, including the main merges and regression checks.

Checked:

  • The description now clearly scopes this as opt-in tolerance for prefix divergence, without claiming to fix a current async-GRPO stall.
  • Configuration propagation and replay/capture guards, retained-prefix tokens and logprobs, episode reward preservation, and truncation metrics remain consistent after the main merges.
  • Relevant environment, rollout, and distillation regression cases passed in the current-head CI logs; all 96 checks are successful or skipped (85 successful, 11 skipped).

LGTM — no remaining findings from this re-review.

Generated by Codex

@yuki-97
yuki-97 merged commit 48c8843 into NVIDIA-NeMo:main Nov 17, 2026
96 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants