Repository navigation
feat(sc): support dynamic sampling on the token-capture path - #5202
Conversation
b1d6260 to
644f188
Compare
f6a7f55 to
5e782ea
Compare
|
/ok to test 5e782ea |
ecca789 to
d5a8e8b
Compare
|
/ok to test d5a8e8b |
d5a8e8b to
922104e
Compare
|
/ok to test 922104e |
yuki-97
left a comment
There was a problem hiding this comment.
Thanks for adding dynamic sampling to the token-capture path.
Scope: Static review of filtering, replacement ownership and coverage at 922104e. Tests were not run here.
Checked:
- Rejected groups never become selectable; cleanup and replacement reservation share the checkpoint mutation boundary.
- Filtering on original rewards is intentional, as discussed. Training keeps the processed rewards.
Remaining comments
- Add coverage for the pump's spare claim and non-default finalizer voting fields.
nit
- Move SC reward checks into the rollout manager and reuse each group's zero-variance result.
- Restore two test names and correct the EOF and finalization descriptions.
Generated by Codex (#4999 version)
|
/ok to test 0f28ba8 |
0f28ba8 to
3fa616b
Compare
|
/ok to test 3fa616b |
3fa616b to
9931939
Compare
|
/ok to test 9931939 |
|
/ok to test b8183dd |
b8183dd to
05a3f83
Compare
|
/ok to test 05a3f83 |
…e commit Dropping the slot after the row clears (previous commit) still left a window: while the clear RPCs are awaited the slot is ready, so a starved train pump can claim it and train on rows being cleared, and the later drop then fails with "unknown group_ids". Decide token_capture.min_valid_fraction_per_group inside the finalize cut, before commit_finalized. A rejected group is cleaned up like a structural finalizer drop (rows cleared, unready slot aborted), so it is never selectable and never enriched; drop_committed_group is no longer needed. Its replacement is taken in the same cut, and with rollout recovery the spare's ledger reservation is too (or the step is credited short), so no rollout snapshot sees the group gone without its replacement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jonas Yang <joyang@nvidia.com>
Review injected three regressions the suite missed; each now fails: - the pump dropping target_step / replacements_used when it calls _finalize_with_actor (replacement and shortfall silently disabled), - the pump crediting a rejected step short a second time, - reserving the spare's lineage with rollout recovery disabled, which the pump would then never discard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Jonas Yang <joyang@nvidia.com>
…ace test _finalize_with_actor now requires target_step and replacements_used so that a forgotten argument raises instead of silently disabling the replacement and shortfall handling. test_rollout_trace still called it with neither. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: ruit <ruit@nvidia.com>
On the capture path the rewards are final only after the finalizer has verified a group's rows, so the zero-variance filter runs in the finalize cut, before commit_finalized, next to the min_valid_fraction rejection introduced for the below-threshold drop. A rejected group is cleaned up like a structural finalizer drop and never becomes selectable; its replacement, the borrow from a later step's finished group, and the spare's ledger reservation share that cut, so no snapshot sees the group gone without its replacement. The filtered pool cannot be waited on inside the cut, so a finalizing group first claims a spare (_claim_filtered_spare). A claim is only a count against the pool; the spare is popped inside the cut, on rejection. Both rollout paths filter on the reward the environment returned, before effort shaping and reward penalties: Completion.reward_original on the ordinary path and FinalizedGroup.reward_original on the capture path. Which rows vote follows the masking policy on both: the finalizer hands back the raw mask_sample / truncated flags and loss_multiplier, and dynamic_sampling.voting_rewards_from_record and voting_rewards_from_finalized both delegate to voting_rewards. Training and metrics keep the trained reward. Setup no longer rejects token_capture.enabled together with grpo.use_dynamic_sampling. The controller-side finalize outcomes (RejectedGroup, BelowValidFractionGroup, ZeroVarianceGroup, and the FinalizeOutcome alias) live in single_controller_utils/utils.py next to the other controller result types. The three places that put a spare prompt in a dropped group's place (the below-valid and zero-variance rejections, and the pump's replacement of a skipped or filtered group) share _substitute_dropped_group. A below-valid capture group now borrows a finished group from a later step like the others; the borrow does not depend on the rollout path. A nightly runs the capture path with dynamic sampling, reward shaping and reward scaling on (dapo-nanov3-30BA3B-2n8g-megatron_generation-noncolocated- async-gym-single-controller-token_capture). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: ruit <ruit@nvidia.com>
Docstrings/descriptions: - single_controller.py: _claim_filtered_spare docstring — dataloader exhaustion stops waiting for a spare, it does not stop claiming an available one - dapo capture recipe + nightly.txt: zero-variance groups are rejected before they become selectable (the group is finalized, then its replay commit is blocked), not 'before finalization' - test_rollout_manager.py: restore the nemo_gym_* reward-penalty test names Dedup the capture zero-variance classification (own it in RolloutManager): - move the finalized-group classification to RolloutManager._is_zero_variance_finalized, beside _is_zero_variance_group, so the manager that owns the voting settings owns both paths; delete the SC _voting_rewards helper - classify once: the finalize cut sets the verdict on FinalizedGroup.zero_variance and the dispatch loop reads it instead of reclassifying; the ordinary path reuses one voting/zero-variance result for its filter check, log and record Coverage: - test_rollout_pump: a pump regression test that drives a real finalize so a zero-variance group claims and pops its filtered spare (not a mock) - test_selected_forest_boundaries: finalizer carries non-default voting fields (mixed mask_sample/truncated, non-unit loss weight, pre-shaping rewards) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: ruit <ruit@nvidia.com>
…tory test The review-nits commit added a controller-set zero_variance verdict field to FinalizedGroup but did not update test_rpc_dataclass_fields_are_classified, which pins the exact field inventory, so L0 unit tests failed. Register the new field (a light bool that is safe to cross the wire). Signed-off-by: ruit <ruit@nvidia.com>
…recipe Replace the Megatron-generation 2n8g DAPO capture nightly with one that inherits the #4588 dynamic-sampling recipe (Lightning-30B, Megatron training, vLLM generation, 4n8g) and routes the math task through NeMo-Gym (OpenMathReasoning via math_with_judge) so token capture is exercised on the faster vLLM path. MAX_STEPS=5 and the check_metrics thresholds are calibrated on a completed Gym run (unfiltered reward ~0.25, token_mult_prob_error median 1.018/max 1.029, ~1040s/step). Nightly GPU-hour budget stays under 5310 (5283). Signed-off-by: ruit <ruit@nvidia.com>
05a3f83 to
ae65f4a
Compare
|
/ok to test ae65f4a |
yuki-97
left a comment
There was a problem hiding this comment.
Thanks for addressing the voting-setting ownership, classify-once behavior, and EOF spare handling from the previous round.
Scope: static re-review of fix commits 0a1324c through ae65f4a, checked against the previous requests and the changed files.
Checked:
- Capture and ordinary classification use manager-owned voting settings and reuse each completed attempt's result at
rollout_manager.py:2520–2527andsingle_controller.py:1875–1879. - EOF stops waiting for another spare while retaining claims on available prompts at
single_controller.py:1368–1385. - Rejected groups remain unselectable; canonical cleanup and replacement reservation share the mutation cut at
single_controller.py:1505–1551.
Remaining comments are on:
- Asserting that the pump holds a spare claim during finalization, alongside the existing final-zero assertion.
- Fixing the voting-field test's receipt identity mismatch and adding explicit invalid-row coverage.
- Removing the ordinary classification wrapper that now has only test callers.
nit
- Correct the guide's canonical-publication order.
- Shorten the two new in-function test comment blocks.
Generated by Codex
- Fix the voting-field finalizer test: give the healthy receipt its matching capture_key and add an explicit invalid row between the two valid rows, so it no longer trips the identity check (was failing CI with assert 1 == 2) and now proves the voting lists skip the invalid row. - Observe the filtered-spare claim count during capture finalization in the pump composition test and assert it is held (1) across the cut, so removing the pump's claim wrapper now fails the test. - Delete the dead _is_zero_variance_group wrapper (no production callers after the classify-once refactor) and point its reward tests at the live voter/predicate path. - Correct the single-controller doc: a filtered capture group is rejected after its canonical rows are written, not before publication. - Shorten two over-long in-function comment blocks. Signed-off-by: ruit <ruit@nvidia.com>
|
/ok to test d77a1b5 |
…re dynamic sampling and fix allow-short reconciliation Two dynamic-sampling bugs on the streaming-sampler path, both reachable only once token capture and dynamic sampling combine (first supported after #5202): 1. windowed (and any uses_pooled_replacements=False sampler) deadlocked under token capture: the finalize path waited on _claim_filtered_spare() for a spare pool that maybe_refill() never fills for a non-pooled sampler, so every token-captured rollout blocked before commit and no training step ran. _claim_filtered_spare() now yields immediately for a non-pooled sampler, and _finalize_with_actor records the filtered prompt and leaves the slot for a later ordinary dispatch instead of popping the never-refilled pool, mirroring the non-capture path. 2. reconcile_selection_credits rolled back a racing loss's batch_shortfall but left its steps_allowed_short permission in place. The set is append-only, so a racing allow-short credit whose charged step moved on reconciliation stranded a stale permission, silently skipping a later legitimate shortfall instead of tripping the batch floor. Allow-short charges are now reference-counted so reconciliation undoes exactly its racing contribution, never a permission another path (drained pooled step, exhausted close, restore) owns. Finalizer and rollout-pump test helpers now provide the live sampler the replacement, filtered-spare and shortfall-crediting paths read. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: ruit <ruit@nvidia.com>
Summary
Lets
grpo.use_dynamic_samplingrun together withtoken_capture.enabledon the single-controller path. Setup used to reject the combination.The branch also carries the three below-threshold commits that this feature builds on (
f1116eeca,f8bc9f748,612fb38e3):token_capture.min_valid_fraction_per_groupis decided inside the finalize cut, beforecommit_finalized, so a rejected group is never selectable and its replacement is taken in the same cut. They were prepared for a separate PR that will not land, so they are part of this one.How it works
commit_finalized, right after themin_valid_fraction_per_groupcheck, which is decided first. A rejected group is cleaned up like a structural finalizer drop and never becomes selectable. Its replacement prompt, the borrow from a later step's finished group, and the spare's ledger reservation all share that cut, so no rollout snapshot sees the group gone without its replacement.target_step, andpromote_ready_groupdoes not distinguish paths, so the "capture cannot borrow" premise behind the old setup rejection did not hold. The three places that put a spare prompt in a dropped group's place (the below-valid and zero-variance rejections, and the pump's replacement of a skipped or filtered group) share_substitute_dropped_group, which borrows a finished group from a later step when one exists. A below-valid capture group used to skip the borrow; it now borrows like the others._claim_filtered_spare, which is only a count against the pool. The spare itself is popped inside the cut, and only on rejection.Completion.reward_originalon the ordinary path (stamped before shaping rewrites the result) andFinalizedGroup.reward_originalon the capture path (the finalizer's input reward). Training and the reward metrics keep the trained reward. A group whose rewards differ only because of shaping or a penalty is therefore dropped, and a row zeroed by a penalty still votes with its original reward.mask_sample/truncatedflags,reward_originalandloss_multiplieronFinalizedGroup.dynamic_sampling.voting_rewards_from_record(ordinary path) andvoting_rewards_from_finalized(capture path) both delegate tovoting_rewards, so both paths pick the same voters undermasked_reward_policyandoverlong_filtering.FinalizedRewardis untouched: it has a strict checkpoint schema.RejectedGroupis the base shared byBelowValidFractionGroupandZeroVarianceGroup, andFinalizeOutcomenames what_finalize_with_actorreturns. They live insingle_controller_utils/utils.pynext to the other controller result types. The pump handles both rejections in one place and differs only in budget and log line. A group that fails both checks is replaced from the fault-tolerance pool, not the filtering pool.Scope
GRPO only, as for dynamic sampling in general. PPO is still rejected at setup. A PPO run with token capture only tallies the
zero_variancemetric, as before.Behavior to know about
nemo_gym_rollout_hit_max_tokens; the capture finalizer counts a row as truncated when it fillsmax_seq_lenexactly. It only changes which rows vote undermasked_reward_policy: excludewithoverlong_filtering, and it is the definition capture training already uses for the same mask.config.py, and the capture rejection here assumes a pooled sampler.Nightly (still being tested)
dapo-nanov3-30BA3B-2n8g-megatron_generation-noncolocated-async-gym-single-controller-token_captureruns the existing Gym capture nightly (Nano V3 30B-A3B,workplace_assistant, 2 nodes) with dynamic sampling, reward shaping and reward scaling on. It is listed innightly.txtand adds 32 GPU-hours, which keeps the nightly total under its 5310 limit (5246 in a dry run).It has not been run. The time budget (120 minutes) and every metric threshold other than the filter gates are those of the capture nightly and are uncalibrated for dynamic sampling. I do not know how many
workplace_assistantgroups have no reward spread, so the filter gate (zero_variance_groups_total >= 1) anddynamic_sampling_max_gen_batchesmay need adjusting, and reward scaling assumes a 0/1 task reward that I have not checked. Expect to tune it, or drop it fromnightly.txt, once a real run exists.Local end-to-end verification (SingleController token-capture dynamic sampling)
Smoke-scale local runs confirming the capture-path dynamic sampling executes end to end and matches the non-capture path.
Setup (common to all arms): Qwen2.5-1.5B-Instruct · NeMo-Gym math (
math_with_judge, GSM8K, 6000 materialized prompts) · SingleController + vLLM non-colocated (4 generation + 4 training, TP=1) · 100 steps · 8 prompts × 8 generations ·dynamic_sampling_max_gen_batches=30· on-policy (async_rl.sampler.max_lookahead_versions=0) · 1×H100 node. Smoke-scale: validates plumbing/behavior, not convergence.Arms (identical except the two toggles):
d5a8e8be(latest)644f188a644f188aResults (100 steps each, 0 errors):
W&B (project
nvidia/nemo-rl-pr5202): A65bpzp1a, Bvpjjtehw, Catwkv0ba.