Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
c9eb585
feat(single-controller): support dynamic sampling with windowed, read…
RayenTian Oct 19, 2026
6cc9e47
fix(single-controller): reject dynamic sampling with the weight_fifo …
RayenTian Oct 20, 2026
0ed1fbd
feat(single-controller): warn when a custom sampler lacks DropAccount…
RayenTian Oct 20, 2026
9965ef6
fix claude self review comment
RayenTian Oct 21, 2026
099ade0
fix(sc): keep streaming shortfalls aligned with selected groups
RayenTian Nov 11, 2026
8e83b3f
fix(sc): validate resolved sampler selection requirements
RayenTian Nov 12, 2026
2cbaf34
refactor(sc): clarify pooled replacement strategy naming
RayenTian Nov 12, 2026
b69dd6d
test(single-controller): add dynamic-sampling nightlies for windowed …
RayenTian Nov 14, 2026
bbbc067
refactor(single-controller): address review nits on sampler docs, com…
RayenTian Nov 14, 2026
5f23afb
fix(single-controller): close a windowed step short at EOF instead of…
RayenTian Nov 14, 2026
6a25f6c
test(single-controller): list shortfall_reconciliation_pending in the…
RayenTian Nov 14, 2026
ced995d
test(single-controller): run the DAPO SC nightlies for 100 steps with…
RayenTian Nov 14, 2026
4020138
refactor(single-controller): extract step-assembly accounting into St…
RayenTian Nov 14, 2026
098fd05
refactor(single-controller): require sampler drop-accounting capabili…
RayenTian Nov 14, 2026
5615433
fix(single-controller): unblock non-pooled samplers under token-captu…
RayenTian Nov 15, 2026
99adccd
refactor(single-controller): address review — StepAssembler rename, e…
RayenTian Nov 15, 2026
5f00e46
test(single-controller): cover the windowed EOF tail train-or-skip un…
RayenTian Nov 15, 2026
09d295f
test(single-controller): build StepAssembler in the remaining object.…
RayenTian Nov 15, 2026
a737532
refactor(single-controller): provide _step_assembler via cached_property
RayenTian Nov 16, 2026
71d6ff8
fix(single-controller): address review — drop-accounting validation, …
RayenTian Nov 16, 2026
5016b7a
fix(single-controller): skip windowed budget check once dataloader is…
RayenTian Nov 16, 2026
4e42d63
fix(single-controller): address review — windowed budget after select…
RayenTian Nov 16, 2026
a3ad1c1
refactor(single-controller): inject StepAssembler's sampler, prompts-…
RayenTian Nov 16, 2026
c40c116
fix(single-controller): give the reserve-controller test sampler the …
RayenTian Nov 16, 2026
388d190
fix(single-controller): update the zero-budget custom-sampler tests f…
RayenTian Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions docs/guides/single-controller.md
Original file line number Diff line number Diff line change
Expand Up @@ -327,25 +327,27 @@ Field definitions:

## Dynamic sampling

Set `grpo.use_dynamic_sampling: true` to drop prompt groups whose generations all received the same reward (zero within-group variance, so no advantage signal) and resample a substitute, as in [DAPO](dapo.md). The filter reads the environment's reward as the environment returned it, before `env.nemo_gym.effort_levels` shaping and `reward_penalties`, on both rollout paths. A group whose rewards differ only because of that shaping or a penalty is therefore dropped, even though the trained rewards differ. A dropped group is always replaced, and the step it was stamped for is filled immediately from a later step's finished group where one exists, so the trainer does not wait out the resample. Setup requires:
Set `grpo.use_dynamic_sampling: true` to drop prompt groups whose participating rewards carry no group-relative advantage signal, as in [DAPO](dapo.md). The filter reads the environment's reward as the environment returned it, before `env.nemo_gym.effort_levels` shaping and `reward_penalties`, on both rollout paths. A group whose rewards differ only because of that shaping or a penalty is therefore dropped, even though the trained rewards differ. `in_order` and `ready_first` draw substitutes from a filtering pool. `in_order` can fill a stamped step immediately by borrowing a later step's finished group; `ready_first` does not stamp steps. `windowed` uses later ordinary dispatches instead of a pool. Setup requires:

- `grpo.dynamic_sampling_max_gen_batches >= 1` (default `10`). A step may consume this many generation batches in total, the batch it was admitted with included, counted in prompts. Past that the run raises, as v1 does: a step that cannot be filled within its allowance means the filter is rejecting faster than the data can refill it. This is a separate budget from `async_rl.rollout_failure.max_replacement_attempts`, which covers rollout failures only; neither drop spends the other's allowance.

Size it from the drop rate the run reports. At drop rate `p` a step needs about `1 / (1 - p)` prompts per committed group, so `grpo.dynamic_sampling_max_gen_batches` has to exceed `1 / (1 - p)` with room for the variance around it. Read `p` off `zero_variance_fraction`.
- `async_rl.sampler.name: in_order`. A filtered group is charged to the target step the sampler stamps, and the spare pool is refilled while the pump waits to admit; no other built-in supports both, and a `custom` class cannot declare at setup that it does.
- `async_rl.sampler.name`: `in_order`, `ready_first`, or `windowed`. `in_order` charges filtered replacements to the stamped step; `ready_first` charges the step currently being assembled. `windowed` uses later ordinary dispatches instead of a filtering spare pool and closes its final step on the groups left, subject to the train-or-skip batch floor. A `custom` sampler must declare the boolean class attribute `requires_single_weight_batch = False` and implement `TransactionalAdmissionSampler` so the pump can refill prompts during admission waits. It must declare the drop-accounting members (`uses_pooled_replacements`, `closes_short_at_exhaustion`, `requires_shortfall_reconciliation`, `charge_step_for_lost_group`); setup rejects a custom sampler that omits any. Setup rejects policies requiring a complete single-weight batch, including `weight_fifo` loaded through a custom class path or subclass: a replacement may land in a newer weight-version bucket and leave the oldest bucket permanently short. Nonzero rollout-failure drop budgets use the same selection requirement check.
- `grpo.num_generations_per_prompt >= 2`.

With `token_capture.enabled: true` the rewards are final only after the finalizer has verified the group's rows, so the filter runs there: a group whose voting rewards carry no signal is rejected after its canonical rows are written but before it becomes selectable, and those rows are cleared. The replacement, the borrow and the per-step allowance work as above. Which rows vote follows `grpo.masked_reward_policy`, `grpo.overlong_filtering` and the environment's `mask_sample` flags, as on the non-capture path; a row that failed capture verification never votes. A filtered group has already paid for row assembly, so a high drop rate costs more here than without capture. `token_capture.min_valid_fraction_per_group` is checked first, and a group that fails it is replaced from the fault-tolerance pool (`async_rl.rollout_failure`), not the filtering pool.

SC pre-warms its filtering pool with one dataloader batch (`grpo.num_prompts_per_step` raw prompts). Below that low-water mark it adds a whole batch, so the pool can hold more than PPS prompts. An empty pool is refilled on demand, including while admission or dispatch is blocked. This is an internal policy, independent of the fault-tolerance pool controlled by `async_rl.rollout_failure.replacement_reserve_prompts`; only `grpo.dynamic_sampling_max_gen_batches` limits the generation attempts for a training step.
For samplers that use a pool, SC pre-warms its filtering pool with one dataloader batch (`grpo.num_prompts_per_step` raw prompts). Below that low-water mark it adds a whole batch, so the pool can hold more than PPS prompts. An empty pool is refilled on demand, including while admission or dispatch is blocked. This is an internal policy, independent of the fault-tolerance pool controlled by `async_rl.rollout_failure.replacement_reserve_prompts`; only `grpo.dynamic_sampling_max_gen_batches` limits the generation attempts for a training step.

`async_rl.rollout_failure.on_dropped_prompt` does not gate filtering. Training on a uniform group is never right, whatever the run does about infrastructure failures, so a filtered group is replaced under `shrink` too, and the spare pool is filled either way. A filtered group shrinks its step only once the dataloader is exhausted and the pool cannot be refilled: the step trains only if its remaining valid groups meet `ceil(num_prompts_per_step * min_step_batch_fraction)`. Otherwise it is skipped, accumulated gradients are discarded, and no optimizer update is performed (`train/skipped_step=1`). Skipped steps advance the scheduling step/version so later admitted steps can finish; they do not advance the optimizer or count toward trained samples. While data remains the pump refills on demand, so a drop waits for a prompt instead of being credited short.
The sampler's `uses_pooled_replacements` flag declares whether it uses pool-based replacements. Filtering and rollout failures have separate prompt pools and budgets; this flag selects the shared replacement strategy, while each feature enables its own pool independently. `in_order` and `ready_first` enable the strategy; `windowed` uses ordinary future dispatches instead.

Expect fewer training steps than `grpo.max_num_steps`. Replacements are ordinary dataloader prompts, drawn from batches diverted into a pool instead of admitted as steps, whether the pump pre-warmed that pool or refilled it on demand, and `max_num_steps` is clamped to `grpo.max_num_epochs * len(dataloader)`. Each committed group costs `1 / (1 - p)` prompts at drop rate `p`, so a single epoch yields about `(1 - p)` of the steps it otherwise would. The run then ends cleanly either way: a step left short because the data ran out is trained or skipped according to `min_step_batch_fraction`. Raise `grpo.max_num_epochs` to absorb the shorter epoch if the run needs its full step budget, or set it to `null` for no epoch limit at all, in which case nothing clamps `max_num_steps` and the loader wraps.
`async_rl.rollout_failure.on_dropped_prompt` does not gate filtering. `in_order` and `ready_first` refill their filtering pool under `shrink` too; `windowed` uses no pool and relies on later ordinary dispatches. A data-exhausted step allowed to close short trains only if its valid groups meet `ceil(num_prompts_per_step * min_step_batch_fraction)`. Otherwise it is skipped, accumulated gradients are discarded, and no optimizer update is performed (`train/skipped_step=1`). Skipped steps advance the scheduling step/version so later admitted steps can finish; they do not advance the optimizer or count toward trained samples.

Expect fewer training steps than `grpo.max_num_steps`. `in_order` and `ready_first` draw replacements from dataloader batches diverted into a filtering pool; `windowed` consumes later ordinary dispatches. `max_num_steps` is clamped to `grpo.max_num_epochs * len(dataloader)`. Each committed group costs `1 / (1 - p)` prompts at drop rate `p`, so a single epoch yields about `(1 - p)` of the steps it otherwise would. The run then ends cleanly either way: a step left short because the data ran out is trained or skipped according to `min_step_batch_fraction`. Raise `grpo.max_num_epochs` to absorb the shorter epoch if the run needs its full step budget, or set it to `null` for no epoch limit at all, in which case nothing clamps `max_num_steps` and the loader wraps.

`grpo.batch_multiplier` is rejected outright, filter or no filter, because the SC dataloader batches `num_prompts_per_step` directly and a run sized around an over-generating multiplier would quietly train on a fraction of the batch it asked for. Filtered groups are counted separately from rollout failures and do not use up `max_skipped_prompts` or `max_consecutive_dropped_prompts`. This section covers GRPO only: PPO needs full-size steps (see [Known Missing Features](#known-missing-features)).

`reward` stays the mean over the trained rows, which only hold groups with reward spread, so it can stay flat while the policy still improves. Watch `unfiltered_reward` (or `first_attempt_reward`) for the pre-filter progress signal instead. These per-step metrics are also reported, and all of them need a sampler that stamps target steps:
`reward` stays the mean over the trained rows, which only hold groups with reward spread, so it can stay flat while the policy still improves. Watch `unfiltered_reward` (or `first_attempt_reward`) for the pre-filter progress signal instead. These per-step metrics are also reported. A group from a sampler that stamps no step is counted under the step being assembled when it finishes:

| Metric | Meaning |
| --- | --- |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
defaults: ./grpo-llama3.1-8b-instruct-2n8g-ready-first-single-controller.yaml
grpo:
use_dynamic_sampling: true
overlong_filtering: true
reward_scaling:
enabled: true
target_min: -1.0
reward_shaping:
enabled: true
overlong_buffer_length: 512
max_response_length: ${policy.generation.max_new_tokens}
logger:
log_dir: logs/dapo-llama3.1-8b-instruct-2n8g-ready-first-single-controller
wandb:
name: dapo-llama3.1-8b-instruct-2n8g-ready-first-single-controller
checkpointing:
checkpoint_dir: results/dapo-llama3.1-8b-instruct-2n8g-ready-first-single-controller
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
defaults: ./grpo-llama3.1-8b-instruct-2n8g-async-1off-single-controller-streaming2.yaml
grpo:
use_dynamic_sampling: true
overlong_filtering: true
reward_scaling:
enabled: true
target_min: -1.0
reward_shaping:
enabled: true
overlong_buffer_length: 512
max_response_length: ${policy.generation.max_new_tokens}
logger:
log_dir: logs/dapo-llama3.1-8b-instruct-2n8g-windowed-single-controller
wandb:
name: dapo-llama3.1-8b-instruct-2n8g-windowed-single-controller
checkpointing:
checkpoint_dir: results/dapo-llama3.1-8b-instruct-2n8g-windowed-single-controller
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ data_plane:
token_capture:
enabled: false
async_rl:
# Dynamic sampling requires the in_order sampler to stamp each target step.
# Match each admitted batch to a training step.
sampler:
name: in_order
max_lookahead_versions: 1
Expand Down
Loading
Loading