Skip to content
5 changes: 3 additions & 2 deletions docs/guides/single-controller.md
Original file line number Diff line number Diff line change
Expand Up @@ -327,14 +327,15 @@ Field definitions:

## Dynamic sampling

Set `grpo.use_dynamic_sampling: true` to drop prompt groups whose generations all received the same reward (zero within-group variance, so no advantage signal) and resample a substitute, as in [DAPO](dapo.md). A dropped group is always replaced, and the step it was stamped for is filled immediately from a later step's finished group where one exists, so the trainer does not wait out the resample. Setup requires:
Set `grpo.use_dynamic_sampling: true` to drop prompt groups whose generations all received the same reward (zero within-group variance, so no advantage signal) and resample a substitute, as in [DAPO](dapo.md). The filter reads the environment's reward as the environment returned it, before `env.nemo_gym.effort_levels` shaping and `reward_penalties`, on both rollout paths. A group whose rewards differ only because of that shaping or a penalty is therefore dropped, even though the trained rewards differ. A dropped group is always replaced, and the step it was stamped for is filled immediately from a later step's finished group where one exists, so the trainer does not wait out the resample. Setup requires:

- `grpo.dynamic_sampling_max_gen_batches >= 1` (default `10`). A step may consume this many generation batches in total, the batch it was admitted with included, counted in prompts. Past that the run raises, as v1 does: a step that cannot be filled within its allowance means the filter is rejecting faster than the data can refill it. This is a separate budget from `async_rl.rollout_failure.max_replacement_attempts`, which covers rollout failures only; neither drop spends the other's allowance.

Size it from the drop rate the run reports. At drop rate `p` a step needs about `1 / (1 - p)` prompts per committed group, so `grpo.dynamic_sampling_max_gen_batches` has to exceed `1 / (1 - p)` with room for the variance around it. Read `p` off `zero_variance_fraction`.
- `async_rl.sampler.name: in_order`. A filtered group is charged to the target step the sampler stamps, and the spare pool is refilled while the pump waits to admit; no other built-in supports both, and a `custom` class cannot declare at setup that it does.
- `grpo.num_generations_per_prompt >= 2`.
- `token_capture.enabled: false`. The capture path cannot borrow a finished group from a later step, so each drop would stall its step for a full fresh rollout; setup rejects the combination rather than paying that on every step.

With `token_capture.enabled: true` the rewards are final only after the finalizer has verified the group's rows, so the filter runs there: a group whose voting rewards carry no signal is rejected after its canonical rows are written but before it becomes selectable, and those rows are cleared. The replacement, the borrow and the per-step allowance work as above. Which rows vote follows `grpo.masked_reward_policy`, `grpo.overlong_filtering` and the environment's `mask_sample` flags, as on the non-capture path; a row that failed capture verification never votes. A filtered group has already paid for row assembly, so a high drop rate costs more here than without capture. `token_capture.min_valid_fraction_per_group` is checked first, and a group that fails it is replaced from the fault-tolerance pool (`async_rl.rollout_failure`), not the filtering pool.

SC pre-warms its filtering pool with one dataloader batch (`grpo.num_prompts_per_step` raw prompts). Below that low-water mark it adds a whole batch, so the pool can hold more than PPS prompts. An empty pool is refilled on demand, including while admission or dispatch is blocked. This is an internal policy, independent of the fault-tolerance pool controlled by `async_rl.rollout_failure.replacement_reserve_prompts`; only `grpo.dynamic_sampling_max_gen_batches` limits the generation attempts for a training step.

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# DAPO SingleController token-capture Gym nightly: the capture sibling of the
# #4588 dynamic-sampling nightly it inherits. Same Lightning-30B model, Megatron
# training, vLLM generation, DAPO reward shaping/scaling, dynamic sampling and 4n8g
# topology. The only difference: the math task runs through NeMo-Gym (math_with_judge
# / OpenMathReasoning) instead of #4588's built-in math env, because token capture is
# only wired up on the Gym rollout path. Math keeps the reward variance the
# zero-variance filter needs so it does not saturate.
# Launch with:
# uv run examples/run_grpo_single_controller.py \
# --config examples/configs/recipes/llm/dapo-nanov3.5-30BA3B-4n8g-megatron-vllm-noncolocated-async-gym-single-controller-token_capture.yaml
defaults: ./dapo-nanov3.5-30BA3B-4n8g-megatron-vllm-noncolocated-single-controller.yaml
token_capture:
enabled: true
policy:
generation:
vllm_cfg:
expose_http_server: true
# Route the DAPO math task through NeMo-Gym for capture (OpenMathReasoning via
# math_with_judge, HF-downloadable so CI can ng_prepare_data it). Judge off:
# math-verify alone scores the rows, keeping the reward binary.
env:
_override_: true
math:
num_workers: 16
math_verify_impl: dapo_math_verify
should_use_nemo_gym: true
nemo_gym:
config_paths:
- responses_api_models/vllm_model/configs/vllm_model_for_training.yaml
- resources_servers/math_with_judge/configs/math_with_judge.yaml
math_with_judge:
resources_servers:
math_with_judge:
judge_model_server:
name: policy_model
should_use_judge: false
policy_model:
responses_api_models:
vllm_model:
uses_reasoning_parser: false
data:
_override_: true
max_input_seq_length: 2048
shuffle: true
num_workers: 0
use_multiple_dataloader: false
default:
dataset_name: NemoGymDataset
env_name: nemo_gym
processor: nemo_gym_data_processor
prompt_file: null
system_prompt_file: null
train:
env_name: nemo_gym
data_path: 3rdparty/Gym-workspace/Gym/data/math_with_judge/train.jsonl
processor: nemo_gym_data_processor
dataset_name: NemoGymDataset
prompt_file: null
system_prompt_file: null
split_validation_size: 0
# SC has no validation (val_period=0), so this is never read; point it at the
# train rows to avoid depending on a second materialized file.
validation:
env_name: nemo_gym
data_path: 3rdparty/Gym-workspace/Gym/data/math_with_judge/train.jsonl
processor: nemo_gym_data_processor
dataset_name: NemoGymDataset
prompt_file: null
system_prompt_file: null
split_validation_size: 0
checkpointing:
checkpoint_dir: results/dapo-nanov3.5-30BA3B-4n8g-megatron-vllm-noncolocated-async-gym-single-controller-token_capture
logger:
log_dir: logs/dapo-nanov3.5-30BA3B-4n8g-megatron-vllm-noncolocated-async-gym-single-controller-token_capture
wandb:
name: dapo-nanov3.5-30BA3B-4n8g-megatron-vllm-noncolocated-async-gym-single-controller-token_capture
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,6 @@
# No NeMo-Gym: the filter is a property of the rollout pump, not of the
# environment, so the cheaper built-in math env exercises it just as well and
# needs no dataset preparation step in CI.
#
# Non-capture on purpose. Setup rejects dynamic sampling together with
# token_capture.enabled, because the capture path cannot borrow a finished
# group from a later step and every drop would stall its step for a fresh
# rollout.
# Launch with:
# uv run examples/run_grpo_single_controller.py \
# --config examples/configs/recipes/llm/dapo-nanov3.5-30BA3B-4n8g-megatron-vllm-noncolocated-single-controller.yaml
Expand Down
Loading
Loading