Skip to content

feat(sc): support dynamic sampling on the token-capture path - #5202

Merged
yuki-97 merged 8 commits into
mainfrom
ruit/sc-capture-dynamic-sampling
Nov 14, 2026
Merged

yuki-97 merged 8 commits into
mainfrom
ruit/sc-capture-dynamic-sampling

Conversation

@RayenTian

@RayenTian RayenTian commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Lets grpo.use_dynamic_sampling run together with token_capture.enabled on the single-controller path. Setup used to reject the combination.

The branch also carries the three below-threshold commits that this feature builds on (f1116eeca, f8bc9f748, 612fb38e3): token_capture.min_valid_fraction_per_group is decided inside the finalize cut, before commit_finalized, so a rejected group is never selectable and its replacement is taken in the same cut. They were prepared for a separate PR that will not land, so they are part of this one.

How it works

  • Where the filter runs. On the capture path the rewards are final only after the finalizer has verified a group's rows. The zero-variance check runs in the finalize cut, before commit_finalized, right after the min_valid_fraction_per_group check, which is decided first. A rejected group is cleaned up like a structural finalizer drop and never becomes selectable. Its replacement prompt, the borrow from a later step's finished group, and the spare's ledger reservation all share that cut, so no rollout snapshot sees the group gone without its replacement.
  • Borrowing works on this path. The buffer slot of a capture group is reserved with its target_step, and promote_ready_group does not distinguish paths, so the "capture cannot borrow" premise behind the old setup rejection did not hold. The three places that put a spare prompt in a dropped group's place (the below-valid and zero-variance rejections, and the pump's replacement of a skipped or filtered group) share _substitute_dropped_group, which borrows a finished group from a later step when one exists. A below-valid capture group used to skip the borrow; it now borrows like the others.
  • Claiming a spare. The filtered pool cannot be waited on inside the cut. A finalizing group first claims a spare with _claim_filtered_spare, which is only a count against the pool. The spare itself is popped inside the cut, and only on rejection.
  • What is filtered on. Both rollout paths filter on the reward the environment returned, before effort shaping and reward penalties: Completion.reward_original on the ordinary path (stamped before shaping rewrites the result) and FinalizedGroup.reward_original on the capture path (the finalizer's input reward). Training and the reward metrics keep the trained reward. A group whose rewards differ only because of shaping or a penalty is therefore dropped, and a row zeroed by a penalty still votes with its original reward.
  • Who votes. The finalizer returns the raw mask_sample / truncated flags, reward_original and loss_multiplier on FinalizedGroup. dynamic_sampling.voting_rewards_from_record (ordinary path) and voting_rewards_from_finalized (capture path) both delegate to voting_rewards, so both paths pick the same voters under masked_reward_policy and overlong_filtering. FinalizedReward is untouched: it has a strict checkpoint schema.
  • Rejected-group types. RejectedGroup is the base shared by BelowValidFractionGroup and ZeroVarianceGroup, and FinalizeOutcome names what _finalize_with_actor returns. They live in single_controller_utils/utils.py next to the other controller result types. The pump handles both rejections in one place and differs only in budget and log line. A group that fails both checks is replaced from the fault-tolerance pool, not the filtering pool.

Scope

GRPO only, as for dynamic sampling in general. PPO is still rejected at setup. A PPO run with token capture only tallies the zero_variance metric, as before.

Behavior to know about

  • A finalizing group waits for a spare no other finalizing group has claimed. This only limits finalize concurrency when the filtering pool holds fewer spares than there are groups finalizing at once.
  • A filtered capture group has already paid for row assembly and the TQ write and clear, so a high drop rate costs more here than on the non-capture path.
  • Truncation is defined differently on the two paths. The ordinary path uses nemo_gym_rollout_hit_max_tokens; the capture finalizer counts a row as truncated when it fills max_seq_len exactly. It only changes which rows vote under masked_reward_policy: exclude with overlong_filtering, and it is the definition capture training already uses for the same mask.
  • feat(single-controller): support dynamic sampling for streaming sampler #4784 (open) keeps rejecting dynamic sampling with token capture and adds samplers that use no spare pool. Whichever PR lands second has to reconcile config.py, and the capture rejection here assumes a pooled sampler.

Nightly (still being tested)

dapo-nanov3-30BA3B-2n8g-megatron_generation-noncolocated-async-gym-single-controller-token_capture runs the existing Gym capture nightly (Nano V3 30B-A3B, workplace_assistant, 2 nodes) with dynamic sampling, reward shaping and reward scaling on. It is listed in nightly.txt and adds 32 GPU-hours, which keeps the nightly total under its 5310 limit (5246 in a dry run).

It has not been run. The time budget (120 minutes) and every metric threshold other than the filter gates are those of the capture nightly and are uncalibrated for dynamic sampling. I do not know how many workplace_assistant groups have no reward spread, so the filter gate (zero_variance_groups_total >= 1) and dynamic_sampling_max_gen_batches may need adjusting, and reward scaling assumes a 0/1 task reward that I have not checked. Expect to tune it, or drop it from nightly.txt, once a real run exists.

Local end-to-end verification (SingleController token-capture dynamic sampling)

Smoke-scale local runs confirming the capture-path dynamic sampling executes end to end and matches the non-capture path.

Setup (common to all arms): Qwen2.5-1.5B-Instruct · NeMo-Gym math (math_with_judge, GSM8K, 6000 materialized prompts) · SingleController + vLLM non-colocated (4 generation + 4 training, TP=1) · 100 steps · 8 prompts × 8 generations · dynamic_sampling_max_gen_batches=30 · on-policy (async_rl.sampler.max_lookahead_versions=0) · 1×H100 node. Smoke-scale: validates plumbing/behavior, not convergence.

Arms (identical except the two toggles):

arm code token_capture dynamic_sampling
A d5a8e8be (latest) ✅ ✅
B 644f188a ❌ ✅
C 644f188a ✅ ❌

Results (100 steps each, 0 errors):

metric A capture+DS B noncapture+DS C capture+noDS
reward (mean) 0.663 0.659 0.794
zero_variance_fraction (mean) 0.462 0.437 0.480
finalize/total_ms (mean) 66.8 — (no finalizer) 67.2
promoted_prompt_groups 0 0 0
image

W&B (project nvidia/nemo-rl-pr5202): A 65bpzp1a, B vpjjtehw, C atwkv0ba.

@RayenTian RayenTian added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Nov 13, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Nov 13, 2026
@RayenTian
RayenTian force-pushed the ruit/sc-capture-dynamic-sampling branch from b1d6260 to 644f188 Compare November 13, 2026 06:36
@RayenTian
RayenTian changed the base branch from joyang/sc-below-threshold-drop to main November 13, 2026 08:51
@RayenTian
RayenTian force-pushed the ruit/sc-capture-dynamic-sampling branch 2 times, most recently from f6a7f55 to 5e782ea Compare November 13, 2026 09:45
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 5e782ea

@RayenTian
RayenTian marked this pull request as ready for review November 13, 2026 09:52
@RayenTian
RayenTian force-pushed the ruit/sc-capture-dynamic-sampling branch 3 times, most recently from ecca789 to d5a8e8b Compare November 13, 2026 10:06
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test d5a8e8b

@RayenTian
RayenTian force-pushed the ruit/sc-capture-dynamic-sampling branch from d5a8e8b to 922104e Compare November 13, 2026 10:46
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 922104e

@RayenTian
RayenTian requested a review from yuki-97 November 13, 2026 12:55

@yuki-97 yuki-97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding dynamic sampling to the token-capture path.

Scope: Static review of filtering, replacement ownership and coverage at 922104e. Tests were not run here.

Checked:

  • Rejected groups never become selectable; cleanup and replacement reservation share the checkpoint mutation boundary.
  • Filtering on original rewards is intentional, as discussed. Training keeps the processed rewards.

Remaining comments

  • Add coverage for the pump's spare claim and non-default finalizer voting fields.

nit

  • Move SC reward checks into the rollout manager and reuse each group's zero-variance result.
  • Restore two test names and correct the EOF and finalization descriptions.

Generated by Codex (#4999 version)

Comment thread nemo_rl/experience/dynamic_sampling.py Resolved
Comment thread nemo_rl/algorithms/single_controller.py Resolved
Comment thread nemo_rl/algorithms/single_controller.py Resolved
Comment thread nemo_rl/algorithms/single_controller.py Resolved
Comment thread nemo_rl/experience/dynamic_sampling.py Resolved
Comment thread nemo_rl/algorithms/single_controller.py Resolved
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 0f28ba8

@RayenTian
RayenTian force-pushed the ruit/sc-capture-dynamic-sampling branch from 0f28ba8 to 3fa616b Compare November 14, 2026 05:04
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 3fa616b

@RayenTian
RayenTian force-pushed the ruit/sc-capture-dynamic-sampling branch from 3fa616b to 9931939 Compare November 14, 2026 05:23
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 9931939

@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test b8183dd

@RayenTian
RayenTian requested a review from yuki-97 November 14, 2026 07:14
@RayenTian
RayenTian force-pushed the ruit/sc-capture-dynamic-sampling branch from b8183dd to 05a3f83 Compare November 14, 2026 07:18
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 05a3f83

joyang-nv and others added 4 commits November 13, 2026 23:39
…e commit

Dropping the slot after the row clears (previous commit) still left a
window: while the clear RPCs are awaited the slot is ready, so a starved
train pump can claim it and train on rows being cleared, and the later
drop then fails with "unknown group_ids".

Decide token_capture.min_valid_fraction_per_group inside the finalize cut,
before commit_finalized. A rejected group is cleaned up like a structural
finalizer drop (rows cleared, unready slot aborted), so it is never
selectable and never enriched; drop_committed_group is no longer needed.
Its replacement is taken in the same cut, and with rollout recovery the
spare's ledger reservation is too (or the step is credited short), so no
rollout snapshot sees the group gone without its replacement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jonas Yang <joyang@nvidia.com>
Review injected three regressions the suite missed; each now fails:
- the pump dropping target_step / replacements_used when it calls
  _finalize_with_actor (replacement and shortfall silently disabled),
- the pump crediting a rejected step short a second time,
- reserving the spare's lineage with rollout recovery disabled, which
  the pump would then never discard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jonas Yang <joyang@nvidia.com>
…ace test

_finalize_with_actor now requires target_step and replacements_used so that
a forgotten argument raises instead of silently disabling the replacement and
shortfall handling. test_rollout_trace still called it with neither.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>
On the capture path the rewards are final only after the finalizer has
verified a group's rows, so the zero-variance filter runs in the finalize
cut, before commit_finalized, next to the min_valid_fraction rejection
introduced for the below-threshold drop. A rejected group is cleaned up
like a structural finalizer drop and never becomes selectable; its
replacement, the borrow from a later step's finished group, and the
spare's ledger reservation share that cut, so no snapshot sees the group
gone without its replacement.

The filtered pool cannot be waited on inside the cut, so a finalizing
group first claims a spare (_claim_filtered_spare). A claim is only a
count against the pool; the spare is popped inside the cut, on rejection.

Both rollout paths filter on the reward the environment returned, before
effort shaping and reward penalties: Completion.reward_original on the
ordinary path and FinalizedGroup.reward_original on the capture path.
Which rows vote follows the masking policy on both: the finalizer hands
back the raw mask_sample / truncated flags and loss_multiplier, and
dynamic_sampling.voting_rewards_from_record and voting_rewards_from_finalized
both delegate to voting_rewards. Training and metrics keep the trained
reward. Setup no longer rejects token_capture.enabled together with
grpo.use_dynamic_sampling.

The controller-side finalize outcomes (RejectedGroup, BelowValidFractionGroup,
ZeroVarianceGroup, and the FinalizeOutcome alias) live in
single_controller_utils/utils.py next to the other controller result types.

The three places that put a spare prompt in a dropped group's place (the
below-valid and zero-variance rejections, and the pump's replacement of a
skipped or filtered group) share _substitute_dropped_group. A below-valid
capture group now borrows a finished group from a later step like the others;
the borrow does not depend on the rollout path.

A nightly runs the capture path with dynamic sampling, reward shaping and
reward scaling on (dapo-nanov3-30BA3B-2n8g-megatron_generation-noncolocated-
async-gym-single-controller-token_capture).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>
RayenTian and others added 3 commits November 13, 2026 23:39
Docstrings/descriptions:
- single_controller.py: _claim_filtered_spare docstring — dataloader exhaustion
  stops waiting for a spare, it does not stop claiming an available one
- dapo capture recipe + nightly.txt: zero-variance groups are rejected before
  they become selectable (the group is finalized, then its replay commit is
  blocked), not 'before finalization'
- test_rollout_manager.py: restore the nemo_gym_* reward-penalty test names

Dedup the capture zero-variance classification (own it in RolloutManager):
- move the finalized-group classification to RolloutManager._is_zero_variance_finalized,
  beside _is_zero_variance_group, so the manager that owns the voting settings
  owns both paths; delete the SC _voting_rewards helper
- classify once: the finalize cut sets the verdict on FinalizedGroup.zero_variance
  and the dispatch loop reads it instead of reclassifying; the ordinary path reuses
  one voting/zero-variance result for its filter check, log and record

Coverage:
- test_rollout_pump: a pump regression test that drives a real finalize so a
  zero-variance group claims and pops its filtered spare (not a mock)
- test_selected_forest_boundaries: finalizer carries non-default voting fields
  (mixed mask_sample/truncated, non-unit loss weight, pre-shaping rewards)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>
…tory test

The review-nits commit added a controller-set zero_variance verdict field to
FinalizedGroup but did not update test_rpc_dataclass_fields_are_classified, which
pins the exact field inventory, so L0 unit tests failed. Register the new field
(a light bool that is safe to cross the wire).

Signed-off-by: ruit <ruit@nvidia.com>
…recipe

Replace the Megatron-generation 2n8g DAPO capture nightly with one that inherits
the #4588 dynamic-sampling recipe (Lightning-30B, Megatron training, vLLM
generation, 4n8g) and routes the math task through NeMo-Gym (OpenMathReasoning via
math_with_judge) so token capture is exercised on the faster vLLM path. MAX_STEPS=5
and the check_metrics thresholds are calibrated on a completed Gym run (unfiltered
reward ~0.25, token_mult_prob_error median 1.018/max 1.029, ~1040s/step). Nightly
GPU-hour budget stays under 5310 (5283).

Signed-off-by: ruit <ruit@nvidia.com>
@RayenTian
RayenTian force-pushed the ruit/sc-capture-dynamic-sampling branch from 05a3f83 to ae65f4a Compare November 14, 2026 07:42
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test ae65f4a

@yuki-97 yuki-97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the voting-setting ownership, classify-once behavior, and EOF spare handling from the previous round.

Scope: static re-review of fix commits 0a1324c through ae65f4a, checked against the previous requests and the changed files.

Checked:

Remaining comments are on:

  • Asserting that the pump holds a spare claim during finalization, alongside the existing final-zero assertion.
  • Fixing the voting-field test's receipt identity mismatch and adding explicit invalid-row coverage.
  • Removing the ordinary classification wrapper that now has only test callers.

nit

  • Correct the guide's canonical-publication order.
  • Shorten the two new in-function test comment blocks.

Generated by Codex

Comment thread nemo_rl/experience/rollout_manager.py Resolved
Comment thread docs/guides/single-controller.md Resolved
Comment thread nemo_rl/algorithms/single_controller.py Resolved
- Fix the voting-field finalizer test: give the healthy receipt its matching
  capture_key and add an explicit invalid row between the two valid rows, so it
  no longer trips the identity check (was failing CI with assert 1 == 2) and now
  proves the voting lists skip the invalid row.
- Observe the filtered-spare claim count during capture finalization in the pump
  composition test and assert it is held (1) across the cut, so removing the
  pump's claim wrapper now fails the test.
- Delete the dead _is_zero_variance_group wrapper (no production callers after
  the classify-once refactor) and point its reward tests at the live
  voter/predicate path.
- Correct the single-controller doc: a filtered capture group is rejected after
  its canonical rows are written, not before publication.
- Shorten two over-long in-function comment blocks.

Signed-off-by: ruit <ruit@nvidia.com>
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test d77a1b5

@RayenTian
RayenTian requested a review from yuki-97 November 14, 2026 10:20
@yuki-97
yuki-97 enabled auto-merge (squash) November 14, 2026 10:34
@yuki-97
yuki-97 merged commit f79dd15 into main Nov 14, 2026
105 checks passed
@yuki-97
yuki-97 deleted the ruit/sc-capture-dynamic-sampling branch November 14, 2026 12:25
RayenTian added a commit that referenced this pull request Nov 15, 2026
…re dynamic sampling and fix allow-short reconciliation

Two dynamic-sampling bugs on the streaming-sampler path, both reachable only
once token capture and dynamic sampling combine (first supported after #5202):

1. windowed (and any uses_pooled_replacements=False sampler) deadlocked under
   token capture: the finalize path waited on _claim_filtered_spare() for a spare
   pool that maybe_refill() never fills for a non-pooled sampler, so every
   token-captured rollout blocked before commit and no training step ran.
   _claim_filtered_spare() now yields immediately for a non-pooled sampler, and
   _finalize_with_actor records the filtered prompt and leaves the slot for a
   later ordinary dispatch instead of popping the never-refilled pool, mirroring
   the non-capture path.

2. reconcile_selection_credits rolled back a racing loss's batch_shortfall but
   left its steps_allowed_short permission in place. The set is append-only, so a
   racing allow-short credit whose charged step moved on reconciliation stranded a
   stale permission, silently skipping a later legitimate shortfall instead of
   tripping the batch floor. Allow-short charges are now reference-counted so
   reconciliation undoes exactly its racing contribution, never a permission
   another path (drained pooled step, exhausted close, restore) owns.

Finalizer and rollout-pump test helpers now provide the live sampler the
replacement, filtered-spare and shortfall-crediting paths read.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: ruit <ruit@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants