Skip to content

feat(single-controller): support dynamic sampling for streaming sampler - #4784

Merged
yuki-97 merged 25 commits into
mainfrom
ruit/dapo_dynamic_sampling_sc_streamingsampler
Nov 16, 2026
Merged

yuki-97 merged 25 commits into
mainfrom
ruit/dapo_dynamic_sampling_sc_streamingsampler

Conversation

@RayenTian

@RayenTian RayenTian commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Extends single-controller dynamic sampling from in_order to ready_first and windowed. The base is main; the prerequisite implementation from #4588 is already merged.

For gated streaming, a lost group now fills an unselected slot rather than excusing work already selected for training. For example, with four prompts per step, two losses during a three-group chunk are charged one to the current step and one to the next, allowing two valid three-group updates instead of failing the next step at 3/4 groups.

Issues

#1365

Supported policies

async_rl.sampler.name Dynamic sampling and loss accounting
in_order Supported. Losses stay on their stamped target step.
ready_first Supported. Filtering uses its own replacement pool; permanent losses are charged to the step being assembled, advancing as selected groups plus losses fill each step.
windowed Supported. Later ordinary dispatches replace filtered groups without a spare pool. The final remainder trains or skips according to min_step_batch_fraction.
weight_fifo Rejected with dynamic sampling. A replacement can carry a newer weight version and cannot reliably refill the oldest bucket. Nonzero fault-tolerance drop budgets are also rejected.
custom Dynamic sampling requires TransactionalAdmissionSampler. Optional DropAccountingSampler defines pool use, final-step behavior, and loss destinations; missing members warn and use gated defaults. Its charge policy must compute a destination without mutating state because selection races can require reevaluation.

Dynamic sampling remains GRPO-only. It also runs on the token-capture path (first enabled by #5202), where this PR corrects the streaming-sampler (windowed/ready_first) accounting. Filtering uses grpo.dynamic_sampling_max_gen_batches, independently of async_rl.rollout_failure.max_replacement_attempts and the fault-tolerance spare pool. on_dropped_prompt: replace is not required for filtering. Data-exhausted steps below the batch floor are skipped without an optimizer update.

Accounting and checkpoint behavior

The optional sampler protocol separates stamped, gated streaming, and continuously producing policies. The controller tracks selected groups, including the chunk currently training, so another loss cannot be charged to a step whose available slots are already filled.

Losses remain recorded immediately while select() awaits its training claim. After selection returns, only those concurrent credits are reapplied in arrival order using the actual selected count and the sampler's policy. Stamped and fixed custom destinations remain unchanged; exhausted-filter permission follows each reassigned loss. Rollout snapshots skip the short interval with unreconciled credits and recheck under the exclusive checkpoint cut. This avoids persisting a provisional allocation that would violate the batch floor after restore, without changing the checkpoint format.

Per-step unfiltered reward metrics also work for unstamped samplers. Attribution uses the step being assembled when the group completes, so it is approximate for streaming policies.

Usage

grpo:
  use_dynamic_sampling: true
  num_generations_per_prompt: 16
  num_prompts_per_step: 8
  dynamic_sampling_max_gen_batches: 10

async_rl:
  sampler:
    name: ready_first  # or windowed; in_order uses max_lookahead_versions
    max_staleness_versions: 1
  rollout_failure:
    min_step_batch_fraction: 1.0

loss_fn:
  use_importance_sampling_correction: true

The filtering budget counts the initial generation batch and replacement prompts. Fault-tolerance replacement settings remain separate and are only needed when the run should replace infrastructure failures.

GPU training runs (dynamic sampling on windowed and ready_first, vs the in_order baseline)

100-step DAPO runs of the two new nightlies against the established in_order baseline, all on the same setup (Llama-3.1-8B-Instruct, 2n8g, native math env, no NeMo-Gym; grpo.use_dynamic_sampling: true with reward shaping/scaling), one continuous job each:

Metric in_order baseline (wandb) windowed (wandb) ready_first (wandb)
reward @ step 10 / 50 / 100 −0.36 / −0.15 / −0.08 −0.21 / −0.17 / −0.11 −0.20 / −0.17 / +0.05
unfiltered_reward median / @100 0.44 / 0.49 0.44 / 0.49 0.45 / 0.56
zero_variance_fraction median (dynamic sampling) 0.23 0.22 0.26
token_mult_prob_error median / @100 1.011 / 1.011 1.010 / 1.010 1.011 / 1.007
grad_norm (last 2 steps) 0.26 0.25 0.42
skipped steps / dropped prompt groups 0 / 0 0 / 0 0 / 0

Both new streaming samplers match the in_order baseline: all three reach 100 steps and converge — reward trends up on every sampler, ready_first crossing positive by step 100 — with the same healthy generation↔training logprob consistency, while dynamic sampling filters ~0.22–0.26 (median) zero-variance groups across all three. These back the new nightlies tests/test_suites/llm/dapo-llama3.1-8b-instruct-2n8g-{windowed,ready_first}-single-controller.sh, which gate on median(token_mult_prob_error) (one ready_first step spikes the raw ratio, so the median rather than a max is gated), an unfiltered_reward floor, a zero_variance_groups_total floor proving the filter ran, and no skipped/dropped steps.

Nightly Result

image

GPU training runs on the token-capture path

Dynamic sampling on the token-capture path was first enabled by #5202. These 100-step DAPO runs put the same three samplers through the token-capture finalizer to confirm windowed and ready_first account correctly there, against the in_order baseline — all on the same setup (Llama-3.1-8B-Instruct, 2n8g, native math env, token capture on; grpo.use_dynamic_sampling: true), one continuous job each. reward here is the raw math reward (unshaped), so compare samplers against each other rather than against the shaped-reward table above:

Metric in_order baseline (wandb) windowed (wandb) ready_first (wandb)
reward @ step 10 / 50 / 100 0.73 / 0.63 / 0.66 0.70 / 0.72 / 0.64 0.72 / 0.58 / 0.67
unfiltered_reward median / @100 0.80 / 0.86 0.81 / 0.80 0.80 / 0.81
zero_variance_fraction median (dynamic sampling) 0.50 0.48 0.45
token_mult_prob_error median / @100 1.007 / 1.006 1.007 / 1.006 1.007 / 1.007
grad_norm (last 2 steps) 0.69 / 0.73 0.80 / 0.78 0.95 / 0.80
skipped steps / dropped prompt groups 0 / 0 0 / 0 0 / 0

On the token-capture path the two streaming samplers track the in_order baseline: all three reach 100 steps with no skipped or dropped steps, hold generation↔training logprob consistency (token_mult_prob_error ~1.007), and keep dynamic sampling filtering ~0.45–0.50 (median) zero-variance groups. reward stays in the same band across all three samplers, so these runs establish mechanism parity on the token-capture path rather than a convergence change.

image

@copy-pr-bot

copy-pr-bot Bot commented Oct 19, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Oct 19, 2026
@RayenTian RayenTian added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Oct 19, 2026
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 4f9bbf6

@RayenTian
RayenTian force-pushed the ruit/dapo_dynamic_sampling_sc_streamingsampler branch 2 times, most recently from 73e2434 to de629ae Compare October 20, 2026 06:08
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 4fae850

@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test db5f343

@RayenTian
RayenTian marked this pull request as ready for review October 22, 2026 05:31

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The optional sampler protocol is a useful separation between gated and continuously producing policies. I inspected all 16 changed files and traced admission, shortfall accounting, training claims, checkpoint restoration, and reward tallies.

Critical (1)

One correctness issue remains: losses arriving during a nonfinal training chunk can over-credit the current step, leaving the following step short without accounting for it.

Suggestions (1)

Quantitative training evidence would help validate the newly supported sampling policies.

Validation was static only: no tests or linters were executed, and no delegated independent review was available.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread nemo_rl/algorithms/single_controller.py Resolved
Comment thread docs/guides/single-controller.md Resolved
RayenTian added a commit that referenced this pull request Nov 3, 2026
Two review findings on #4588, coupled closely enough that fixing one alone
leaves the other half-answered.

The spare pool could run dry while data and budget remained. It was refilled
only when the rollout pump reached the top of its loop with a new dataloader
batch, and the pump spends most of a step blocked in admit(). Filtering is
routine, so tasks drained the pool during that window and the next one was
credited short even though the dataloader had more to give. A filtered group
now asks the pump for a refill and waits; the pump services that request while
it waits to admit. admit() is a polling wait followed by a synchronous commit,
so it is waited on rather than cancelled and can never be left half-admitted.
On the recovery path the refill takes its own prompt_reservations cut, keeping
the property that the pool and the dataloader advance together.

Normal data exhaustion could also fail the run. The drain counts raw spare
prompts, not groups that survive the filter, so a full batch of spares can come
up short once some score uniformly, with no data left to replace them. Such a
step is exempt from min_step_batch_fraction now: the floor exists to catch a
sick fleet or a mis-tuned filter, not the dataset ending.

Per the review, dynamic sampling no longer borrows fault tolerance's
configuration either. It has its own pool, its own low-water mark
(grpo.dynamic_sampling_reserve_prompts, null meaning one dataloader batch), and
grpo.dynamic_sampling_max_gen_batches is honored as the per-step generation
allowance rather than warned about as inert. The per-slot key added in the
previous round is gone: a step-level allowance is what v1 bounds, and two
budgets for one thing was the redundancy the review pointed at.

The checkpoint sidecar now holds both pools. A checkpoint written before the
split is a plain list and still restores, as the fault-tolerance pool.

Dynamic sampling is narrowed to async_rl.sampler.name='in_order'. Replacement
needs a stamped target step and on-demand refill needs an interruptible
admission wait; neither is knowable at setup for a custom class. #4784 reopens
this through its capability protocol, where a custom sampler can declare both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>
@cmunley1
cmunley1 added this pull request to stack #5054 November 3, 2026 22:27
Base automatically changed from ruit/dapo_dynamic_sampling_sc to main November 10, 2026 13:41
@RayenTian
RayenTian force-pushed the ruit/dapo_dynamic_sampling_sc_streamingsampler branch from b8cfc24 to 5615433 Compare November 15, 2026 05:34
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 5615433

…ager init, unconditional drop-accounting check

- Rename StepAssemblyAccountant -> StepAssembler and self._accountant ->
  self._step_assembler; build it eagerly in __init__ instead of a lazy
  Optional-sentinel property, matching the rest of the controller's members.
  Test controllers built via object.__new__ construct the assembler explicitly.
- Rename the config-validation local sampler_shrinks_steps ->
  sampler_charges_drops_to_steps and clarify why windowed is excluded (it
  charges no drop to a step; it backfills from a later dispatch).
- Hoist require_drop_accounting_members out of the use_dynamic_sampling gate:
  charge_step_for_lost_group / requires_shortfall_reconciliation (credit_shortfall),
  closes_short_at_exhaustion (EOF), and uses_pooled_replacements (fault-tolerance
  replacement) are read off the sampler on the async SC path regardless of
  dynamic sampling, so a custom sampler missing one must fail at setup, not mid-run.
- Fix 'windoweds' typo in create_sampler docstring.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 99adccd

@RayenTian
RayenTian requested a review from yuki-97 November 15, 2026 08:35
…der dynamic sampling

Add test_windowed_tail_under_dynamic_sampling_trains_or_skips_by_the_floor,
parametrized trainable vs below-floor: with dynamic sampling on, an exhausted
windowed step is marked allowed-short, so the final floor check trains it (50%
floor) or skips it without an optimizer update (100% floor) instead of raising.
The existing dynamic-sampling-off tail tests cover the raise path; this pins the
skip path the review asked for.

Also shorten the _validate_failure_settings floor-warning comment back to one
line, per the comment-length rule.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 5f00e46

…__new__ fixtures

99adccd made _step_assembler an eager __init__ attribute, but controllers
built via object.__new__ in test_advantage_stage, test_checkpoint_borrow_restore,
test_checkpoint_dispatch_races and test_finalizer_lifecycle still relied on the
removed lazy property and raised AttributeError in L0_Unit_Tests_Other. Construct
the assembler explicitly in those fixtures.

Also cut the two DAPO single-controller nightly drivers' NUM_MINUTES from 120 to
90 to match the observed ~72-82 min / 100-step runtime.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 09d295f

@yuki-97 yuki-97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the previous round's accounting and sampler-interface comments.

Scope: fix commits b69dd6d through 5f00e46, checked against the remaining review requests and the current base integrations.

Checked:

  • Selected-group accounting includes the active training chunk; concurrent shortfall credits are reapplied using the settled selection and sampler policy.
  • Snapshot exclusion covers provisional credits and rechecks under the exclusive cut; accounting starts from the restored trainer version, including step 0.
  • Admission callbacks remain inside their TaskGroup owner, and sampler-required-member validation now runs independently of dynamic sampling.
  • The posted three-sampler training comparisons provide useful native and token-capture validation; the recorded native runs reach 100 steps at equal trained-group counts.

Remaining comments are on:

  • Windowed generation-budget enforcement across successful selections and EOF.
  • Capture-finalizer propagation of exhausted-filter permission through reconciliation.
  • The setup-test assertion that contradicts unconditional validation, and the unused duplicate sampler protocol.
  • Completing the requested capability-gate accounting and snapshot coverage.

nit

  • Direct capability reads, explicit assembler dependencies, and the remaining declaration-placement request.
  • Required-interface descriptions, long function comments, and stale test descriptions.

Generated by Codex

Comment thread nemo_rl/algorithms/single_controller.py Resolved
Comment thread nemo_rl/algorithms/single_controller.py Resolved
Eager __init__ construction (plus a hand-written StepAssembler assignment in
every object.__new__ test fixture) seeded the assembling cursor before the
fixture set _trainer_version, so a dispatch-race test's rollout pump never
terminated and L0_Unit_Tests_Other hung. A cached_property builds the assembler
on first use, seeding the cursor from the then-current _trainer_version exactly
as the previous lazy accessor did, so the object.__new__ fixtures need no manual
construction. Keeps the StepAssembler name and drops the Optional sentinel.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test a737532

…filtered EOF credit, remove dead protocol

- Move require_drop_accounting_members from setup into
  validate_single_controller_config (called unconditionally), so a custom sampler
  missing a drop-accounting member fails config validation regardless of dynamic
  sampling; migrate the regression test to the config-validation entry point.
- Pass filtered=True when a zero-variance capture finalizer closes its step short,
  so the EOF short-step permission is recorded (allow_short) instead of failing an
  allowance it never spent once reconciliation moves the loss.
- Remove the dead DropAccountingSampler Protocol: PromptGroupSampler already
  declares all four members. Clean up its imports, comments, docstrings, docs and
  the protocol-only isinstance tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 71d6ff8

@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 8fd360a

@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test c105158

RayenTian and others added 4 commits November 15, 2026 21:22
… exhausted; shorten in-body comments

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>
…ion, direct capability read, reconciliation coverage

- Enforce the windowed filtered budget after every selection against the groups
  actually selected (selected + drawn vs the whole allowance), replacing the
  starvation-only poll check so a successful selection no longer bypasses it and
  an EOF tail keeps its full resample allowance instead of being overcounted.
- Read the pooled-replacement capability directly now that required-member
  validation runs before the reader.
- Move _DROP_ACCOUNTING_MEMBERS below its validator; rename the last-spare test
  and fix a stale helper name in a docstring; shorten over-long in-body comments.
- Add capability-gated selection-reconciliation coverage for
  requires_shortfall_reconciliation False and True.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>
…per-step and runtime instead of the whole controller

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>
…pooled-replacement capability its path now reads directly

098fd05 made _divert_batch_to_reserve / _take_replacement read
sampler.uses_pooled_replacements directly, but _reserve_controller's stub
sampler was a bare SimpleNamespace. With on_dropped_prompt='replace' the
divert gate reaches that read and raises AttributeError inside the refill
task, so mutation_applied is never set and the waiter hangs forever.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>
@RayenTian
RayenTian force-pushed the ruit/dapo_dynamic_sampling_sc_streamingsampler branch from c105158 to c40c116 Compare November 16, 2026 05:23
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test c40c116

@RayenTian
RayenTian requested a review from yuki-97 November 16, 2026 05:36
…or the unconditional interface check

require_drop_accounting_members now runs for every config, so a custom sampler
is imported and checked even when no drop is tolerated. The old test pointed at
a non-existent my_pkg.samplers:MySampler and raised ModuleNotFoundError. Replace
it with one test that a compliant custom sampler is accepted at zero budget and
one that a sampler missing the members is rejected.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 388d190

@yuki-97
yuki-97 merged commit 99ae93a into main Nov 16, 2026
107 checks passed
@yuki-97
yuki-97 deleted the ruit/dapo_dynamic_sampling_sc_streamingsampler branch November 16, 2026 09:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants