Repository navigation
feat(single-controller): support dynamic sampling for streaming sampler - #4784
Conversation
|
/ok to test 4f9bbf6 |
73e2434 to
de629ae
Compare
|
/ok to test 4fae850 |
|
/ok to test db5f343 |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
The optional sampler protocol is a useful separation between gated and continuously producing policies. I inspected all 16 changed files and traced admission, shortfall accounting, training claims, checkpoint restoration, and reward tallies.
Critical (1)
One correctness issue remains: losses arriving during a nonfinal training chunk can over-credit the current step, leaving the following step short without accounting for it.
Suggestions (1)
Quantitative training evidence would help validate the newly supported sampling policies.
Validation was static only: no tests or linters were executed, and no delegated independent review was available.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
Two review findings on #4588, coupled closely enough that fixing one alone leaves the other half-answered. The spare pool could run dry while data and budget remained. It was refilled only when the rollout pump reached the top of its loop with a new dataloader batch, and the pump spends most of a step blocked in admit(). Filtering is routine, so tasks drained the pool during that window and the next one was credited short even though the dataloader had more to give. A filtered group now asks the pump for a refill and waits; the pump services that request while it waits to admit. admit() is a polling wait followed by a synchronous commit, so it is waited on rather than cancelled and can never be left half-admitted. On the recovery path the refill takes its own prompt_reservations cut, keeping the property that the pool and the dataloader advance together. Normal data exhaustion could also fail the run. The drain counts raw spare prompts, not groups that survive the filter, so a full batch of spares can come up short once some score uniformly, with no data left to replace them. Such a step is exempt from min_step_batch_fraction now: the floor exists to catch a sick fleet or a mis-tuned filter, not the dataset ending. Per the review, dynamic sampling no longer borrows fault tolerance's configuration either. It has its own pool, its own low-water mark (grpo.dynamic_sampling_reserve_prompts, null meaning one dataloader batch), and grpo.dynamic_sampling_max_gen_batches is honored as the per-step generation allowance rather than warned about as inert. The per-slot key added in the previous round is gone: a step-level allowance is what v1 bounds, and two budgets for one thing was the redundancy the review pointed at. The checkpoint sidecar now holds both pools. A checkpoint written before the split is a plain list and still restores, as the fault-tolerance pool. Dynamic sampling is narrowed to async_rl.sampler.name='in_order'. Replacement needs a stamped target step and on-demand refill needs an interruptible admission wait; neither is knowable at setup for a custom class. #4784 reopens this through its capability protocol, where a custom sampler can declare both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: ruit <ruit@nvidia.com>
b8cfc24 to
5615433
Compare
|
/ok to test 5615433 |
…ager init, unconditional drop-accounting check - Rename StepAssemblyAccountant -> StepAssembler and self._accountant -> self._step_assembler; build it eagerly in __init__ instead of a lazy Optional-sentinel property, matching the rest of the controller's members. Test controllers built via object.__new__ construct the assembler explicitly. - Rename the config-validation local sampler_shrinks_steps -> sampler_charges_drops_to_steps and clarify why windowed is excluded (it charges no drop to a step; it backfills from a later dispatch). - Hoist require_drop_accounting_members out of the use_dynamic_sampling gate: charge_step_for_lost_group / requires_shortfall_reconciliation (credit_shortfall), closes_short_at_exhaustion (EOF), and uses_pooled_replacements (fault-tolerance replacement) are read off the sampler on the async SC path regardless of dynamic sampling, so a custom sampler missing one must fail at setup, not mid-run. - Fix 'windoweds' typo in create_sampler docstring. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: ruit <ruit@nvidia.com>
|
/ok to test 99adccd |
…der dynamic sampling Add test_windowed_tail_under_dynamic_sampling_trains_or_skips_by_the_floor, parametrized trainable vs below-floor: with dynamic sampling on, an exhausted windowed step is marked allowed-short, so the final floor check trains it (50% floor) or skips it without an optimizer update (100% floor) instead of raising. The existing dynamic-sampling-off tail tests cover the raise path; this pins the skip path the review asked for. Also shorten the _validate_failure_settings floor-warning comment back to one line, per the comment-length rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: ruit <ruit@nvidia.com>
|
/ok to test 5f00e46 |
…__new__ fixtures 99adccd made _step_assembler an eager __init__ attribute, but controllers built via object.__new__ in test_advantage_stage, test_checkpoint_borrow_restore, test_checkpoint_dispatch_races and test_finalizer_lifecycle still relied on the removed lazy property and raised AttributeError in L0_Unit_Tests_Other. Construct the assembler explicitly in those fixtures. Also cut the two DAPO single-controller nightly drivers' NUM_MINUTES from 120 to 90 to match the observed ~72-82 min / 100-step runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: ruit <ruit@nvidia.com>
|
/ok to test 09d295f |
yuki-97
left a comment
There was a problem hiding this comment.
Thanks for addressing the previous round's accounting and sampler-interface comments.
Scope: fix commits b69dd6d through 5f00e46, checked against the remaining review requests and the current base integrations.
Checked:
- Selected-group accounting includes the active training chunk; concurrent shortfall credits are reapplied using the settled selection and sampler policy.
- Snapshot exclusion covers provisional credits and rechecks under the exclusive cut; accounting starts from the restored trainer version, including step 0.
- Admission callbacks remain inside their TaskGroup owner, and sampler-required-member validation now runs independently of dynamic sampling.
- The posted three-sampler training comparisons provide useful native and token-capture validation; the recorded native runs reach 100 steps at equal trained-group counts.
Remaining comments are on:
- Windowed generation-budget enforcement across successful selections and EOF.
- Capture-finalizer propagation of exhausted-filter permission through reconciliation.
- The setup-test assertion that contradicts unconditional validation, and the unused duplicate sampler protocol.
- Completing the requested capability-gate accounting and snapshot coverage.
nit
- Direct capability reads, explicit assembler dependencies, and the remaining declaration-placement request.
- Required-interface descriptions, long function comments, and stale test descriptions.
Generated by Codex
Eager __init__ construction (plus a hand-written StepAssembler assignment in every object.__new__ test fixture) seeded the assembling cursor before the fixture set _trainer_version, so a dispatch-race test's rollout pump never terminated and L0_Unit_Tests_Other hung. A cached_property builds the assembler on first use, seeding the cursor from the then-current _trainer_version exactly as the previous lazy accessor did, so the object.__new__ fixtures need no manual construction. Keeps the StepAssembler name and drops the Optional sentinel. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: ruit <ruit@nvidia.com>
|
/ok to test a737532 |
…filtered EOF credit, remove dead protocol - Move require_drop_accounting_members from setup into validate_single_controller_config (called unconditionally), so a custom sampler missing a drop-accounting member fails config validation regardless of dynamic sampling; migrate the regression test to the config-validation entry point. - Pass filtered=True when a zero-variance capture finalizer closes its step short, so the EOF short-step permission is recorded (allow_short) instead of failing an allowance it never spent once reconciliation moves the loss. - Remove the dead DropAccountingSampler Protocol: PromptGroupSampler already declares all four members. Clean up its imports, comments, docstrings, docs and the protocol-only isinstance tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: ruit <ruit@nvidia.com>
|
/ok to test 71d6ff8 |
|
/ok to test 8fd360a |
|
/ok to test c105158 |
… exhausted; shorten in-body comments Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: ruit <ruit@nvidia.com>
…ion, direct capability read, reconciliation coverage - Enforce the windowed filtered budget after every selection against the groups actually selected (selected + drawn vs the whole allowance), replacing the starvation-only poll check so a successful selection no longer bypasses it and an EOF tail keeps its full resample allowance instead of being overcounted. - Read the pooled-replacement capability directly now that required-member validation runs before the reader. - Move _DROP_ACCOUNTING_MEMBERS below its validator; rename the last-spare test and fix a stale helper name in a docstring; shorten over-long in-body comments. - Add capability-gated selection-reconciliation coverage for requires_shortfall_reconciliation False and True. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: ruit <ruit@nvidia.com>
…per-step and runtime instead of the whole controller Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: ruit <ruit@nvidia.com>
…pooled-replacement capability its path now reads directly 098fd05 made _divert_batch_to_reserve / _take_replacement read sampler.uses_pooled_replacements directly, but _reserve_controller's stub sampler was a bare SimpleNamespace. With on_dropped_prompt='replace' the divert gate reaches that read and raises AttributeError inside the refill task, so mutation_applied is never set and the waiter hangs forever. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: ruit <ruit@nvidia.com>
c105158 to
c40c116
Compare
|
/ok to test c40c116 |
…or the unconditional interface check require_drop_accounting_members now runs for every config, so a custom sampler is imported and checked even when no drop is tolerated. The old test pointed at a non-existent my_pkg.samplers:MySampler and raised ModuleNotFoundError. Replace it with one test that a compliant custom sampler is accepted at zero budget and one that a sampler missing the members is rejected. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: ruit <ruit@nvidia.com>
|
/ok to test 388d190 |
What does this PR do?
Extends single-controller dynamic sampling from
in_ordertoready_firstandwindowed. The base ismain; the prerequisite implementation from #4588 is already merged.For gated streaming, a lost group now fills an unselected slot rather than excusing work already selected for training. For example, with four prompts per step, two losses during a three-group chunk are charged one to the current step and one to the next, allowing two valid three-group updates instead of failing the next step at 3/4 groups.
Issues
#1365
Supported policies
async_rl.sampler.namein_orderready_firstwindowedmin_step_batch_fraction.weight_fifocustomTransactionalAdmissionSampler. OptionalDropAccountingSamplerdefines pool use, final-step behavior, and loss destinations; missing members warn and use gated defaults. Its charge policy must compute a destination without mutating state because selection races can require reevaluation.Dynamic sampling remains GRPO-only. It also runs on the token-capture path (first enabled by #5202), where this PR corrects the streaming-sampler (
windowed/ready_first) accounting. Filtering usesgrpo.dynamic_sampling_max_gen_batches, independently ofasync_rl.rollout_failure.max_replacement_attemptsand the fault-tolerance spare pool.on_dropped_prompt: replaceis not required for filtering. Data-exhausted steps below the batch floor are skipped without an optimizer update.Accounting and checkpoint behavior
The optional sampler protocol separates stamped, gated streaming, and continuously producing policies. The controller tracks selected groups, including the chunk currently training, so another loss cannot be charged to a step whose available slots are already filled.
Losses remain recorded immediately while
select()awaits its training claim. After selection returns, only those concurrent credits are reapplied in arrival order using the actual selected count and the sampler's policy. Stamped and fixed custom destinations remain unchanged; exhausted-filter permission follows each reassigned loss. Rollout snapshots skip the short interval with unreconciled credits and recheck under the exclusive checkpoint cut. This avoids persisting a provisional allocation that would violate the batch floor after restore, without changing the checkpoint format.Per-step unfiltered reward metrics also work for unstamped samplers. Attribution uses the step being assembled when the group completes, so it is approximate for streaming policies.
Usage
The filtering budget counts the initial generation batch and replacement prompts. Fault-tolerance replacement settings remain separate and are only needed when the run should replace infrastructure failures.
GPU training runs (dynamic sampling on
windowedandready_first, vs thein_orderbaseline)100-step DAPO runs of the two new nightlies against the established
in_orderbaseline, all on the same setup (Llama-3.1-8B-Instruct, 2n8g, native math env, no NeMo-Gym;grpo.use_dynamic_sampling: truewith reward shaping/scaling), one continuous job each:in_orderbaseline (wandb)windowed(wandb)ready_first(wandb)reward@ step 10 / 50 / 100unfiltered_rewardmedian / @100zero_variance_fractionmedian (dynamic sampling)token_mult_prob_errormedian / @100grad_norm(last 2 steps)Both new streaming samplers match the
in_orderbaseline: all three reach 100 steps and converge —rewardtrends up on every sampler,ready_firstcrossing positive by step 100 — with the same healthy generation↔training logprob consistency, while dynamic sampling filters ~0.22–0.26 (median) zero-variance groups across all three. These back the new nightliestests/test_suites/llm/dapo-llama3.1-8b-instruct-2n8g-{windowed,ready_first}-single-controller.sh, which gate onmedian(token_mult_prob_error)(oneready_firststep spikes the raw ratio, so the median rather than a max is gated), anunfiltered_rewardfloor, azero_variance_groups_totalfloor proving the filter ran, and no skipped/dropped steps.Nightly Result
GPU training runs on the token-capture path
Dynamic sampling on the token-capture path was first enabled by #5202. These 100-step DAPO runs put the same three samplers through the token-capture finalizer to confirm
windowedandready_firstaccount correctly there, against thein_orderbaseline — all on the same setup (Llama-3.1-8B-Instruct, 2n8g, native math env, token capture on;grpo.use_dynamic_sampling: true), one continuous job each.rewardhere is the raw math reward (unshaped), so compare samplers against each other rather than against the shaped-reward table above:in_orderbaseline (wandb)windowed(wandb)ready_first(wandb)reward@ step 10 / 50 / 100unfiltered_rewardmedian / @100zero_variance_fractionmedian (dynamic sampling)token_mult_prob_errormedian / @100grad_norm(last 2 steps)On the token-capture path the two streaming samplers track the
in_orderbaseline: all three reach 100 steps with no skipped or dropped steps, hold generation↔training logprob consistency (token_mult_prob_error~1.007), and keep dynamic sampling filtering ~0.45–0.50 (median) zero-variance groups.rewardstays in the same band across all three samplers, so these runs establish mechanism parity on the token-capture path rather than a convergence change.