Repository navigation
feat: bound streaming train chunk size with max_groups_for_streaming_train - #5036
Conversation
ec7248b to
79dd247
Compare
|
/ok to test 79dd247 |
|
/ok to test 79dd247 |
79dd247 to
4225758
Compare
|
/ok to test 4225758 |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
I found two correctness issues:
- C1: A GRPO cap can force an unshardable chunk and fail during policy dispatch (cap application).
- C2: PPO rejects an unaligned ceiling even when it exceeds the step size and never limits a selection (alignment guard).
C3 and C4 are low-impact documentation and explicit repository-guidance suggestions, not runtime blockers.
I inspected all seven changed files and traced the affected sampler, policy-sharding, setup and training paths. One delegated reviewer completed without a delivered report; I independently covered its PPO alignment/setup topic. This was a static review: I didn't execute tests or linters, and I haven't independently verified the results reported in the PR description.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
4225758 to
e9ad749
Compare
|
/ok to test e9ad749 |
@saumishr, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
|
/ok to test e9ad749 |
e9ad749 to
7bec235
Compare
|
/ok to test 7bec235 |
mehraakash
left a comment
There was a problem hiding this comment.
Thanks for the work on this. The knob is opt-in with a null default, the pump change is three lines, and the new pump tests do guard it: deleting the min(...) line fails two of them. The PPO/GRPO alignment guards are also a sensible split, and the C1/C2/C3 fixes verified correct at this head. No blockers; the inline comments are low-severity cleanups and one missing wiring test.
Two optional items that aren't tied to a line:
- Performance evidence. It would help to include step time or
exposed_generationwith and without the cap on one configuration (the Ultra run that motivated this would be ideal), plus a note on how to pick a cap, since each extra chunk adds fixed per-chunk work (logprob detour, advantage dispatch). - Follow-up. There is no real-backend multi-chunk test (cap below the step, DP >= 2, loss/grad-norm parity against a single-chunk run). That needs GPU infrastructure, so I'd track it as a follow-up.
Linters were not run in this review (the pinned Python was unavailable locally); CI covers them.
Generated by Claude Code
7bec235 to
391cb0f
Compare
|
/ok to test 391cb0f |
terrykong
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround on the earlier rounds. Every fix from the bot review and Akash's review holds at 391cb0f (checked against the code), and the decline on the is_ppo_run double check has a good reason. The PR's tests pass locally, and deleting each guard or the cap line makes at least one test fail, except for the PPO skip covered in C3.
- (C1) The GRPO cap guard misses the micro-batch size under static batching, so a passing config crashes mid-step. Please fix in this PR.
- (C2) Please share capped vs uncapped timing before merge; the code doesn't show how a cap frees generation.
- (C3) One test can't fail; one-line fix.
- (C4) Shorter docs entry.
Not tied to a line:
- The PR description is out of date: it still lists four validators including the colocated-Megatron guard, the
megatron_colocated_chunk_captest case, and reviewer note 2, all removed in 391cb0f. Please update it to match the squashed commit. - Follow-up, not this PR: please file the issue you mentioned for the unaligned GRPO floor. The real fix is to give GRPO a
group_multiplethe way PPO has, so every chunk is aligned, not just the full-cap one. In the same issue:reinforce_plus_plusnormalizes advantages per chunk, so streaming changes its advantages with no warning (PPO already warns for the same effect).
Linters (ruff 0.9.9, pinned) pass on all changed files.
Generated by Claude Code
391cb0f to
ece43df
Compare
|
/ok to test ece43df |
ece43df to
28023c0
Compare
|
/ok to test 28023c0 |
…train
min_groups_for_streaming_train is a floor, not a chunk size: _train_pump
passes it as the sampler's minimum but hands target - dispatched as the
maximum, so once generation has run ahead the buffer satisfies the whole
step on the first select and the step is dispatched as a single chunk.
On a generation-bound workload that kills the run. The chunk is
materialized on the worker as host tensors before microbatching --
train_microbatches_from_meta shards the chunk's meta across the DP ranks
and each worker fetches its own shard's rows, and the advantage stage
fetches the chunk's input columns too -- so chunk size is resident host
memory, and max_tokens_per_microbatch bounds the GPU step rather than
that staging buffer. Two uncapped Nano swe-only runs died of host-RAM
cgroup OOM on Nov 11 with no CUDA OOM anywhere:
1852652 2048 rows (128 groups) per chunk, 50 oom_kill events
1855756 step 1 streamed in eight chunks of 512-848 rows, then one
chunk of 2240 rows (140 groups), 51 oom_kill events
Add an optional ceiling on the chunk so that peak is a function of the
ceiling rather than of num_prompts_per_step. The cap is applied before
the floor is clamped against it, so a capped chunk still carries the
configured floor and an uneven tail still closes the step. Defaults to
None, which preserves today's unbounded behaviour.
The same two recipes re-run with the cap at 64 did not OOM, and the
clamp is visible in their chunk traces rather than merely configured:
1858088 held every chunk to 512 rows, and 1862042's largest chunk was
exactly 1024 rows, which is 64 groups at 16 generations per prompt.
Two things the cap deliberately does not do. It does not bound the data
plane: consumed groups are released only at step close, where
_cleanup_consumed_metas_unlocked and _buffer_capacity.release() run after
finish_train_step, so max_buffered_rollouts still sizes the rollout
buffer. And it does not make a step faster: a chunk boundary releases
neither a buffer slot nor the lookahead gate, and each extra chunk costs
one logprob pass and one advantage dispatch. This is a memory bound, not
a throughput knob.
A validator rejects a cap below the floor, which would otherwise clamp
the floor on every chunk and leave it as dead config. Colocated Megatron
needs no cap rule of its own: it already requires the floor to equal
num_prompts_per_step, which with that bound leaves no binding cap
expressible.
Covered by unit tests on the behaviour that motivates the knob: a
sampler modelling a buffer generation has already raced ahead of grants
whatever the pump asks for, up to what is ready, so an uncapped pump
takes the whole step as one dispatch while a capped one never exceeds
the cap and still consumes exactly num_prompts_per_step groups.
Every sampler honours the cap -- WindowedSampler, ReadyFirstSampler,
WeightFifoSampler and InOrderSampler all select through
BaseSampler._finalize_selection, which caps at max_prompt_groups. That
helper also aligns the selection to group_multiple, which streaming PPO
derives from the policy and value data-parallel layouts, and refuses an
unaligned max once tail-aware selection is on. The uncapped max is
target - dispatched and stays aligned, so the cap is the only way to
break the invariant; resolve_ppo_chunk_alignment rejects an unaligned cap
next to the existing num_prompts_per_step check, before the pump claims
any rows. Only a cap below num_prompts_per_step is checked: the target is
that step size less any dropped groups, so a larger cap can never be the
minimum of the two and never limits a selection.
GRPO needs its own guard. It passes no group_multiple, so
_finalize_selection returns whatever the bounds allow and the cap alone
decides the chunk, while the training dispatch shards without
allow_uneven_shards. A cap of 3 at 2 generations per prompt is 6 samples,
which cannot split across 4 DP ranks. validate_grpo_chunk_cap_alignment
rejects a binding cap whose sample count is not a multiple of the policy
sample_multiple, called where the trainer mesh is first available. That
is the same rule PPO already aligns to, so it also covers the second
split: with sequence packing and dynamic batching both off the Megatron
worker divides each rank's rows by the train and logprob microbatch
sizes, and a cap of 3 at 4 generations on a DP of 2 clears the DP split
but leaves each rank 6 rows against a microbatch of 4. Forest is exempt:
its producer pads each group to the policy execution row multiple, so the
logical group count is not the physical row count.
Signed-off-by: Saurabh Mishra <sauramishra@nvidia.com>
28023c0 to
d2f0d49
Compare
|
/ok to test d2f0d49 |
|
/ok to test d2f0d49 |
What does this PR do ?
Adds
async_rl.max_groups_for_streaming_train, an optional ceiling on the prompt groups_train_pumpdispatches in one chunk, so a generation-bound run cannot hand the trainer a whole step's rows at once and exhaust host memory.min_groups_for_streaming_trainis a floor, not a chunk size._train_pumppasses it as the sampler's minimum but handstarget - dispatchedas the maximum, so once generation has run ahead the buffer can satisfy the entire step on the firstselect()and the step becomes a single chunk.The chunk is materialized on the worker as host tensors before microbatching —
train_microbatches_from_metashards the chunk's meta across the DP ranks and each worker fetches its own shard's rows, and the advantage stage fetches the chunk's input columns as well. Chunk size is therefore resident host memory, andmax_tokens_per_microbatchbounds the GPU step rather than that staging buffer.The failure this exists for
Two uncapped Nano swe-only runs died of host-RAM cgroup OOM on Nov 11, with no CUDA OOM anywhere:
1852652oom_killevents1855756oom_killevents1855756is the one that made the mechanism unambiguous. It streamed step 1 in eight well-behaved chunks of 512–848 rows, then took 2,240 rows in a single chunk, and the policy worker's node died. Nothing bounded how much of a full buffer oneselect()could swallow.The same two recipes re-run with
max_groups_for_streaming_train: 64did not OOM, and the clamp is visible in their chunk traces rather than merely configured:1858088held every chunk to 512 rows, and1862042's largest chunk was exactly 1,024 rows — 64 groups at 16 generations per prompt. 64 was picked because it was the largest chunk that had ever completed cleanly, against the 128 and 140 that died.What the knob is not
_cleanup_consumed_metas_unlockedand_buffer_capacity.release()both run afterfinish_train_step. Somax_buffered_rolloutsis still what sizes the rollout buffer; the cap bounds what is pulled into the trainers and the advantage stage, not what the data plane holds.Validators
Three guards, all of which only fire on a cap that can actually bind:
AsyncRLConfig._check_streaming_chunk_boundsrejects a non-positive cap, and a cap belowmin_groups_for_streaming_train— which would otherwise clamp the floor on every chunk and leave it as dead config.resolve_ppo_chunk_alignmentrejects a binding cap that is not a multiple of the policy/value group multiple. This matters for all four samplers: they all reachBaseSampler._finalize_selection, which refuses an unalignedmax_prompt_groupsoutright when tail-aware selection is on. Uncapped, that maximum istarget - dispatched, which stays aligned because the target is aligned and every chunk is — so the cap is the only way to make it unaligned, and it would surface as aValueErroron the firstselect()with the whole allocation already up.validate_grpo_chunk_cap_alignmentrejects a binding GRPO cap whose sample count is not a multiple of the policy'ssample_multiple. GRPO passes nogroup_multiple, so the cap alone decides the chunk and the training dispatch shards it withoutallow_uneven_shards.sample_multipleis the rule PPO already aligns to: the DP size, times the train/logprob microbatch LCM when sequence packing and dynamic batching are both off. Forest is exempt, because its producer pads each group to the policy execution row multiple.Issues
Follow-up filed as #5240 (unaligned GRPO floor;
reinforce_plus_plusnormalizing per chunk without a warning), as requested in review.Usage
Leave it unset for current behaviour. To bound what one dispatch materializes, set it below
num_prompts_per_stepand at or abovemin_groups_for_streaming_train:The driver log reports one line per chunk and one per step close:
Uncapped, a warm buffer produces
closing on 1 chunk(s), 2048 group(s).Before your PR is "Ready for review"
Pre checks:
Additional Information
What the bound costs
Asked for in C2. One node,
grpo_math_1B_megatron_single_controller.yaml, 6 steps per arm, both arms pinned to the same node (nvl72d087-T16) and run back to back so the cap is the only difference. Both pinmin_groups_for_streaming_train=8against a 32-group step; the control leaves the capnulland the capped arm sets it to8. Means over steps 1–5, excluding the cold step 0:null)8)policy_and_reference_logprobslogprob_inference_preppolicy_trainingadvantage_calculationexposed_generationTwo things to read off this. The overhead is almost entirely the three extra logprob passes and their inference prep, which is the per-chunk cost the design implies — so pick the largest cap that fits the memory budget, not the smallest. And
exposed_generationis identical to two decimal places in both arms, confirming that a chunk boundary releases neither a buffer slot nor the lookahead gate: the cap is purchased entirely out of step time.Measured at
391cb0fc, the commit the review was against. The pump delta is byte-identical at the rebased head — the only runtime change since is the GRPO validator — and the available container predates the vLLM 0.33 model source that currentmainrequires.Tests
The behaviour that motivates the knob is covered directly. The existing
_ChunkedSamplerfake returns a fixed group count regardless of the bounds it is handed, so it would pass a cap test even with the cap removed. The new_OverfilledSamplerinstead grantsmin(ready, max_prompt_groups), which is the part of the sampler contract the cap depends on. Against a buffer holding 64 groups and an 8-group step:chunks == [8]);[2, 2, 2, 2], never asks for more than the cap, and still consumes exactlynum_prompts_per_stepgroups — capping changes how much is materialized at once, not how much the step trains on;The validators are covered in
test_resiliency_config.py::TestStreamingChunkBounds, in thetest_ppo_chunk_validation_rejects_incomplete_chunksparametrize plustest_ppo_accepts_a_chunk_cap_aligned_to_the_dp_layoutandtest_ppo_accepts_a_chunk_cap_that_cannot_bind, and for GRPO intest_grpo_rejects_a_chunk_cap_that_cannot_shard_across_policy_dp,test_grpo_rejects_a_cap_that_leaves_each_rank_a_partial_microbatch,test_grpo_ignores_microbatch_sizes_when_the_consumer_bins_its_ownandtest_grpo_setup_rejects_a_chunk_cap_that_cannot_shard_across_policy_dp.One harness note:
_train_pump_controllerintest_single_controller_actor.pybuilds_async_cfgas aSimpleNamespace, so reading the new field raisedAttributeErrorfor every pump test. I added the field to the harness rather than reading it with agetattrdefault insingle_controller.py, since a call-site default would duplicate the schema's default outside the config.Review history
Round 1 (
e9ad749). Addressed the automated review: the PPO alignment guard only applies to a cap that can bind (C2), GRPO gained a DP-divisibility guard (C1), and the field is documented indocs/guides/single-controller.md(C3).Round 2 (
391cb0fc). Addressed @mehraakash's review: dropped the unreachable colocated-Megatron guard and its test case, added the setup-level wiring test forvalidate_grpo_chunk_cap_alignment, and corrected the "full-cap chunk only" wording in the docstring and docs.Round 3 (this push). Rebased onto
main, and addressed @terrykong's review:sample_multiplerather than barepolicy_dp, so it also covers the microbatch split inside each rank under static batching.sample_multiplemoved from a closure inresolve_ppo_chunk_alignmentto module level. Terry's repro (mbs 4, DP 2, G 4, cap 3) now raises at setup; covered bytest_grpo_rejects_a_cap_that_leaves_each_rank_a_partial_microbatch, with a packing/dynamic-batching case confirming those configs are unchanged.AsyncRLConfigfield comment all now state the memory bound the knob provides and the step time it costs.test_the_grpo_chunk_cap_guard_leaves_ppo_to_its_own_alignmentnow uses 2 generations instead of 16, so it fails when theis_ppo_runskip is deleted. Verified against that mutant.Both mutations Terry named were checked: reverting C1 to
% policy_dpfails exactly the new microbatch test, and deleting the PPO skip fails exactly the C3 test.tests/unit/single_controller/is 1987 passed / 19 failed, and all 19 failures (test_fullvocab_recipe_rejects_tropd*,TestOPDFullValidation::*) reproduce identically on unmodifiedmain.ruff0.9.9 clean on format, lint and import order.