Skip to content

feat: bound streaming train chunk size with max_groups_for_streaming_train - #5036

Merged
terrykong merged 1 commit into
NVIDIA-NeMo:mainfrom
saumishr:feat/sc-bound-streaming-train-chunk
Nov 16, 2026
Merged

terrykong merged 1 commit into
NVIDIA-NeMo:mainfrom
saumishr:feat/sc-bound-streaming-train-chunk

Conversation

@saumishr

@saumishr saumishr commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Adds async_rl.max_groups_for_streaming_train, an optional ceiling on the prompt groups _train_pump dispatches in one chunk, so a generation-bound run cannot hand the trainer a whole step's rows at once and exhaust host memory.

min_groups_for_streaming_train is a floor, not a chunk size. _train_pump passes it as the sampler's minimum but hands target - dispatched as the maximum, so once generation has run ahead the buffer can satisfy the entire step on the first select() and the step becomes a single chunk.

The chunk is materialized on the worker as host tensors before microbatching — train_microbatches_from_meta shards the chunk's meta across the DP ranks and each worker fetches its own shard's rows, and the advantage stage fetches the chunk's input columns as well. Chunk size is therefore resident host memory, and max_tokens_per_microbatch bounds the GPU step rather than that staging buffer.

The failure this exists for

Two uncapped Nano swe-only runs died of host-RAM cgroup OOM on Nov 11, with no CUDA OOM anywhere:

Job Largest chunk Outcome
1852652 2048 rows = 128 groups 50 oom_kill events
1855756 2240 rows = 140 groups 51 oom_kill events

1855756 is the one that made the mechanism unambiguous. It streamed step 1 in eight well-behaved chunks of 512–848 rows, then took 2,240 rows in a single chunk, and the policy worker's node died. Nothing bounded how much of a full buffer one select() could swallow.

The same two recipes re-run with max_groups_for_streaming_train: 64 did not OOM, and the clamp is visible in their chunk traces rather than merely configured: 1858088 held every chunk to 512 rows, and 1862042's largest chunk was exactly 1,024 rows — 64 groups at 16 generations per prompt. 64 was picked because it was the largest chunk that had ever completed cleanly, against the 128 and 140 that died.

What the knob is not

  • It does not bound the data plane. Consumed groups are released only at step close: _cleanup_consumed_metas_unlocked and _buffer_capacity.release() both run after finish_train_step. So max_buffered_rollouts is still what sizes the rollout buffer; the cap bounds what is pulled into the trainers and the advantage stage, not what the data plane holds.
  • It does not make a step faster. A chunk boundary releases neither a buffer slot nor the lookahead gate, and each extra chunk costs a logprob pass and an advantage dispatch. Measured below.

Validators

Three guards, all of which only fire on a cap that can actually bind:

  • AsyncRLConfig._check_streaming_chunk_bounds rejects a non-positive cap, and a cap below min_groups_for_streaming_train — which would otherwise clamp the floor on every chunk and leave it as dead config.
  • resolve_ppo_chunk_alignment rejects a binding cap that is not a multiple of the policy/value group multiple. This matters for all four samplers: they all reach BaseSampler._finalize_selection, which refuses an unaligned max_prompt_groups outright when tail-aware selection is on. Uncapped, that maximum is target - dispatched, which stays aligned because the target is aligned and every chunk is — so the cap is the only way to make it unaligned, and it would surface as a ValueError on the first select() with the whole allocation already up.
  • validate_grpo_chunk_cap_alignment rejects a binding GRPO cap whose sample count is not a multiple of the policy's sample_multiple. GRPO passes no group_multiple, so the cap alone decides the chunk and the training dispatch shards it without allow_uneven_shards. sample_multiple is the rule PPO already aligns to: the DP size, times the train/logprob microbatch LCM when sequence packing and dynamic batching are both off. Forest is exempt, because its producer pads each group to the policy execution row multiple.

Issues

Follow-up filed as #5240 (unaligned GRPO floor; reinforce_plus_plus normalizing per chunk without a warning), as requested in review.

Usage

Leave it unset for current behaviour. To bound what one dispatch materializes, set it below num_prompts_per_step and at or above min_groups_for_streaming_train:

async_rl:
  min_groups_for_streaming_train: 32
  # 512 groups per chunk against a 2048-group step: the trainer stages a
  # quarter of the step at a time instead of all of it.
  max_groups_for_streaming_train: 512

The driver log reports one line per chunk and one per step close:

train_pump: step 2 chunk 1: 512 group(s), 512/2048 dispatched
...
train_pump: step 2 closing on 4 chunk(s), 2048 group(s)

Uncapped, a warm buffer produces closing on 1 chunk(s), 2048 group(s).

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

What the bound costs

Asked for in C2. One node, grpo_math_1B_megatron_single_controller.yaml, 6 steps per arm, both arms pinned to the same node (nvl72d087-T16) and run back to back so the cap is the only difference. Both pin min_groups_for_streaming_train=8 against a 32-group step; the control leaves the cap null and the capped arm sets it to 8. Means over steps 1–5, excluding the cold step 0:

control (null) capped (8) delta
chunks per step 1 4
total step time 29.95 s 37.36 s +7.41 s (+24.7%)
policy_and_reference_logprobs 13.36 s 17.97 s +4.61 s
logprob_inference_prep 0.61 s 2.34 s +1.73 s
policy_training 15.30 s 16.26 s +0.96 s
advantage_calculation 0.18 s 0.26 s +0.08 s
exposed_generation 0.02 s 0.02 s 0.00 s

Two things to read off this. The overhead is almost entirely the three extra logprob passes and their inference prep, which is the per-chunk cost the design implies — so pick the largest cap that fits the memory budget, not the smallest. And exposed_generation is identical to two decimal places in both arms, confirming that a chunk boundary releases neither a buffer slot nor the lookahead gate: the cap is purchased entirely out of step time.

Measured at 391cb0fc, the commit the review was against. The pump delta is byte-identical at the rebased head — the only runtime change since is the GRPO validator — and the available container predates the vLLM 0.33 model source that current main requires.

Tests

The behaviour that motivates the knob is covered directly. The existing _ChunkedSampler fake returns a fixed group count regardless of the bounds it is handed, so it would pass a cap test even with the cap removed. The new _OverfilledSampler instead grants min(ready, max_prompt_groups), which is the part of the sampler contract the cap depends on. Against a buffer holding 64 groups and an 8-group step:

  • uncapped, the pump takes the whole step as one dispatch (chunks == [8]);
  • capped at 2, it takes [2, 2, 2, 2], never asks for more than the cap, and still consumes exactly num_prompts_per_step groups — capping changes how much is materialized at once, not how much the step trains on;
  • with an uneven step, the full chunk carries the configured floor and only the short tail clamps it down.

The validators are covered in test_resiliency_config.py::TestStreamingChunkBounds, in the test_ppo_chunk_validation_rejects_incomplete_chunks parametrize plus test_ppo_accepts_a_chunk_cap_aligned_to_the_dp_layout and test_ppo_accepts_a_chunk_cap_that_cannot_bind, and for GRPO in test_grpo_rejects_a_chunk_cap_that_cannot_shard_across_policy_dp, test_grpo_rejects_a_cap_that_leaves_each_rank_a_partial_microbatch, test_grpo_ignores_microbatch_sizes_when_the_consumer_bins_its_own and test_grpo_setup_rejects_a_chunk_cap_that_cannot_shard_across_policy_dp.

One harness note: _train_pump_controller in test_single_controller_actor.py builds _async_cfg as a SimpleNamespace, so reading the new field raised AttributeError for every pump test. I added the field to the harness rather than reading it with a getattr default in single_controller.py, since a call-site default would duplicate the schema's default outside the config.

Review history

Round 1 (e9ad749). Addressed the automated review: the PPO alignment guard only applies to a cap that can bind (C2), GRPO gained a DP-divisibility guard (C1), and the field is documented in docs/guides/single-controller.md (C3).

Round 2 (391cb0fc). Addressed @mehraakash's review: dropped the unreachable colocated-Megatron guard and its test case, added the setup-level wiring test for validate_grpo_chunk_cap_alignment, and corrected the "full-cap chunk only" wording in the docstring and docs.

Round 3 (this push). Rebased onto main, and addressed @terrykong's review:

  • C1 — the GRPO guard now checks against sample_multiple rather than bare policy_dp, so it also covers the microbatch split inside each rank under static batching. sample_multiple moved from a closure in resolve_ppo_chunk_alignment to module level. Terry's repro (mbs 4, DP 2, G 4, cap 3) now raises at setup; covered by test_grpo_rejects_a_cap_that_leaves_each_rank_a_partial_microbatch, with a packing/dynamic-batching case confirming those configs are unchanged.
  • C2 — measured; see "What the bound costs" above. The description, the docs entry, both exemplar YAML comments and the AsyncRLConfig field comment all now state the memory bound the knob provides and the step time it costs.
  • C3 — test_the_grpo_chunk_cap_guard_leaves_ppo_to_its_own_alignment now uses 2 generations instead of 16, so it fails when the is_ppo_run skip is deleted. Verified against that mutant.
  • C4 — the docs entry is rewritten usage-first and cut from 1,600 to 1,046 characters (35%), taking your suggested text and adding only what it could not carry: the one-sentence statement of what the knob is for, the data-plane caveat, and the per-chunk cost.
  • Follow-up issue Streaming GRPO: unaligned chunks from the floor, and silent per-chunk advantage normalization for reinforce_plus_plus #5240 filed.

Both mutations Terry named were checked: reverting C1 to % policy_dp fails exactly the new microbatch test, and deleting the PPO skip fails exactly the C3 test. tests/unit/single_controller/ is 1987 passed / 19 failed, and all 19 failures (test_fullvocab_recipe_rejects_tropd*, TestOPDFullValidation::*) reproduce identically on unmodified main. ruff 0.9.9 clean on format, lint and import order.

@copy-pr-bot

copy-pr-bot Bot commented Nov 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@saumishr
saumishr force-pushed the feat/sc-bound-streaming-train-chunk branch from ec7248b to 79dd247 Compare November 3, 2026 05:02
@saumishr

saumishr commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 79dd247

@saumishr saumishr added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Nov 3, 2026
@NVIDIA-NeMo NVIDIA-NeMo deleted a comment from copy-pr-bot Bot Nov 3, 2026
@saumishr saumishr added CI:L0 Run doctests and unit tests and removed CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) labels Nov 3, 2026
@saumishr

saumishr commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 79dd247

@saumishr
saumishr requested a review from yuki-97 November 3, 2026 05:34
@saumishr
saumishr force-pushed the feat/sc-bound-streaming-train-chunk branch from 79dd247 to 4225758 Compare November 3, 2026 05:51
@saumishr

saumishr commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 4225758

@saumishr
saumishr marked this pull request as ready for review November 3, 2026 14:15

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found two correctness issues:

  1. C1: A GRPO cap can force an unshardable chunk and fail during policy dispatch (cap application).
  2. C2: PPO rejects an unaligned ceiling even when it exceeds the step size and never limits a selection (alignment guard).

C3 and C4 are low-impact documentation and explicit repository-guidance suggestions, not runtime blockers.

I inspected all seven changed files and traced the affected sampler, policy-sharding, setup and training paths. One delegated reviewer completed without a delivered report; I independently covered its PPO alignment/setup topic. This was a static review: I didn't execute tests or linters, and I haven't independently verified the results reported in the PR description.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread nemo_rl/algorithms/single_controller.py Resolved
@saumishr
saumishr force-pushed the feat/sc-bound-streaming-train-chunk branch from 4225758 to e9ad749 Compare November 3, 2026 15:57
@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Nov 3, 2026
@saumishr

saumishr commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e9ad749

@copy-pr-bot

copy-pr-bot Bot commented Nov 3, 2026

Copy link
Copy Markdown

/ok to test e9ad749

@saumishr, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

@saumishr

saumishr commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e9ad749

@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team November 3, 2026 17:04
@saumishr
saumishr force-pushed the feat/sc-bound-streaming-train-chunk branch from e9ad749 to 7bec235 Compare November 3, 2026 17:50
@saumishr

saumishr commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 7bec235

@mehraakash mehraakash left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work on this. The knob is opt-in with a null default, the pump change is three lines, and the new pump tests do guard it: deleting the min(...) line fails two of them. The PPO/GRPO alignment guards are also a sensible split, and the C1/C2/C3 fixes verified correct at this head. No blockers; the inline comments are low-severity cleanups and one missing wiring test.

Two optional items that aren't tied to a line:

  • Performance evidence. It would help to include step time or exposed_generation with and without the cap on one configuration (the Ultra run that motivated this would be ideal), plus a note on how to pick a cap, since each extra chunk adds fixed per-chunk work (logprob detour, advantage dispatch).
  • Follow-up. There is no real-backend multi-chunk test (cap below the step, DP >= 2, loss/grad-norm parity against a single-chunk run). That needs GPU infrastructure, so I'd track it as a follow-up.

Linters were not run in this review (the pinned Python was unavailable locally); CI covers them.

Generated by Claude Code

Comment thread nemo_rl/algorithms/single_controller.py Resolved
Comment thread nemo_rl/algorithms/single_controller.py Resolved
@saumishr
saumishr force-pushed the feat/sc-bound-streaming-train-chunk branch from 7bec235 to 391cb0f Compare November 3, 2026 18:23
@saumishr

saumishr commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 391cb0f

@terrykong terrykong added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:L0 Run doctests and unit tests labels Nov 5, 2026

@terrykong terrykong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround on the earlier rounds. Every fix from the bot review and Akash's review holds at 391cb0f (checked against the code), and the decline on the is_ppo_run double check has a good reason. The PR's tests pass locally, and deleting each guard or the cap line makes at least one test fail, except for the PPO skip covered in C3.

  1. (C1) The GRPO cap guard misses the micro-batch size under static batching, so a passing config crashes mid-step. Please fix in this PR.
  2. (C2) Please share capped vs uncapped timing before merge; the code doesn't show how a cap frees generation.
  3. (C3) One test can't fail; one-line fix.
  4. (C4) Shorter docs entry.

Not tied to a line:

  • The PR description is out of date: it still lists four validators including the colocated-Megatron guard, the megatron_colocated_chunk_cap test case, and reviewer note 2, all removed in 391cb0f. Please update it to match the squashed commit.
  • Follow-up, not this PR: please file the issue you mentioned for the unaligned GRPO floor. The real fix is to give GRPO a group_multiple the way PPO has, so every chunk is aligned, not just the full-cap one. In the same issue: reinforce_plus_plus normalizes advantages per chunk, so streaming changes its advantages with no warning (PPO already warns for the same effect).

Linters (ruff 0.9.9, pinned) pass on all changed files.

Generated by Claude Code

Comment thread docs/guides/single-controller.md Resolved
@saumishr

Copy link
Copy Markdown
Contributor Author

/ok to test ece43df

@saumishr

Copy link
Copy Markdown
Contributor Author

/ok to test 28023c0

…train

min_groups_for_streaming_train is a floor, not a chunk size: _train_pump
passes it as the sampler's minimum but hands target - dispatched as the
maximum, so once generation has run ahead the buffer satisfies the whole
step on the first select and the step is dispatched as a single chunk.

On a generation-bound workload that kills the run. The chunk is
materialized on the worker as host tensors before microbatching --
train_microbatches_from_meta shards the chunk's meta across the DP ranks
and each worker fetches its own shard's rows, and the advantage stage
fetches the chunk's input columns too -- so chunk size is resident host
memory, and max_tokens_per_microbatch bounds the GPU step rather than
that staging buffer. Two uncapped Nano swe-only runs died of host-RAM
cgroup OOM on Nov 11 with no CUDA OOM anywhere:

  1852652  2048 rows (128 groups) per chunk, 50 oom_kill events
  1855756  step 1 streamed in eight chunks of 512-848 rows, then one
           chunk of 2240 rows (140 groups), 51 oom_kill events

Add an optional ceiling on the chunk so that peak is a function of the
ceiling rather than of num_prompts_per_step. The cap is applied before
the floor is clamped against it, so a capped chunk still carries the
configured floor and an uneven tail still closes the step. Defaults to
None, which preserves today's unbounded behaviour.

The same two recipes re-run with the cap at 64 did not OOM, and the
clamp is visible in their chunk traces rather than merely configured:
1858088 held every chunk to 512 rows, and 1862042's largest chunk was
exactly 1024 rows, which is 64 groups at 16 generations per prompt.

Two things the cap deliberately does not do. It does not bound the data
plane: consumed groups are released only at step close, where
_cleanup_consumed_metas_unlocked and _buffer_capacity.release() run after
finish_train_step, so max_buffered_rollouts still sizes the rollout
buffer. And it does not make a step faster: a chunk boundary releases
neither a buffer slot nor the lookahead gate, and each extra chunk costs
one logprob pass and one advantage dispatch. This is a memory bound, not
a throughput knob.

A validator rejects a cap below the floor, which would otherwise clamp
the floor on every chunk and leave it as dead config. Colocated Megatron
needs no cap rule of its own: it already requires the floor to equal
num_prompts_per_step, which with that bound leaves no binding cap
expressible.

Covered by unit tests on the behaviour that motivates the knob: a
sampler modelling a buffer generation has already raced ahead of grants
whatever the pump asks for, up to what is ready, so an uncapped pump
takes the whole step as one dispatch while a capped one never exceeds
the cap and still consumes exactly num_prompts_per_step groups.

Every sampler honours the cap -- WindowedSampler, ReadyFirstSampler,
WeightFifoSampler and InOrderSampler all select through
BaseSampler._finalize_selection, which caps at max_prompt_groups. That
helper also aligns the selection to group_multiple, which streaming PPO
derives from the policy and value data-parallel layouts, and refuses an
unaligned max once tail-aware selection is on. The uncapped max is
target - dispatched and stays aligned, so the cap is the only way to
break the invariant; resolve_ppo_chunk_alignment rejects an unaligned cap
next to the existing num_prompts_per_step check, before the pump claims
any rows. Only a cap below num_prompts_per_step is checked: the target is
that step size less any dropped groups, so a larger cap can never be the
minimum of the two and never limits a selection.

GRPO needs its own guard. It passes no group_multiple, so
_finalize_selection returns whatever the bounds allow and the cap alone
decides the chunk, while the training dispatch shards without
allow_uneven_shards. A cap of 3 at 2 generations per prompt is 6 samples,
which cannot split across 4 DP ranks. validate_grpo_chunk_cap_alignment
rejects a binding cap whose sample count is not a multiple of the policy
sample_multiple, called where the trainer mesh is first available. That
is the same rule PPO already aligns to, so it also covers the second
split: with sequence packing and dynamic batching both off the Megatron
worker divides each rank's rows by the train and logprob microbatch
sizes, and a cap of 3 at 4 generations on a DP of 2 clears the DP split
but leaves each rank 6 rows against a microbatch of 4. Forest is exempt:
its producer pads each group to the policy execution row multiple, so the
logical group count is not the physical row count.

Signed-off-by: Saurabh Mishra <sauramishra@nvidia.com>
@saumishr
saumishr force-pushed the feat/sc-bound-streaming-train-chunk branch from 28023c0 to d2f0d49 Compare November 16, 2026 03:50
@saumishr

Copy link
Copy Markdown
Contributor Author

/ok to test d2f0d49

@saumishr

Copy link
Copy Markdown
Contributor Author

/ok to test d2f0d49

@NVIDIA-NeMo NVIDIA-NeMo deleted a comment from copy-pr-bot Bot Nov 16, 2026
@terrykong
terrykong enabled auto-merge (squash) November 16, 2026 04:44
@terrykong
terrykong merged commit 6efd4b5 into NVIDIA-NeMo:main Nov 16, 2026
98 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants