Skip to content

feat(grpo): per-prompt-group length penalties for NeMo-Gym GRPO rollouts - #4553

Merged
yfw merged 6 commits into
mainfrom
adithyare/main-pr-length-penalty
Oct 9, 2026
Merged

yfw merged 6 commits into
mainfrom
adithyare/main-pr-length-penalty

Conversation

@arendu

@arendu arendu commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Branch: adithyare/main-pr-length-penalty (6 commits on top of origin/main @ 8bf6bd4ba; DCO signed-off)

Summary

Ports the Super 3.5 length-penalty feature set (NeMo-RL super-v3.5-posttraining, #3852 + #3897) to main.
It adds an optional grpo.length_penalty config block that applies per-prompt-group length
bonuses/penalties to NeMo-Gym rollout rewards in place during rollout postprocessing. Runs
without the block are untouched (GRPOConfig.length_penalty defaults to None).

Penalty families (all resolved per prompt group; binary 0/1 env rewards only, rewards clamped at 0):

  • Shortest-correct bonus (reasoning_bonus / answer_bonus / total_bonus, gated by top_percentile).
  • Longest-correct flat penalty (longest_{reasoning,answer,total}_penalty).
  • Zero-centered group length penalty (group_{reasoning,answer,total}_length_penalty_coeff),
    optionally gated by a per-prompt profile field (group_length_penalty_profile_gate*).
  • zMAD (Iglewicz-Hoaglin modified-Z) high-side outlier penalty per channel.
  • Profiled-length penalty vs. a per-prompt profiled distribution (profiled_length_penalty, _n_std, _min_samples).
  • Pass-rate-scaled length penalty (pass_rate_length_penalty_weight).
  • Profile-band multipliers (profile_band_{total,reasoning,answer}) using per-prompt {a, b, f} bands with
    global defaults under grpo.length_penalty.profile_band; multiplier clamps at f past b.
  • Per-agent overrides (agent_overrides.<agent_name>) and length_type: tokens|chars.
  • The block is a pydantic model (LengthPenaltyConfig, extra="forbid"): unknown keys, bad Literal
    values and malformed profile bands are rejected at config load.

Config block (minimal example):

grpo:
  length_penalty:
    default:
      total_bonus: 0.1
      top_percentile: 0.5
    agent_overrides:
      math_agent:
        enabled: false

Profile-band dataset tool: scripts/build_profile_band_dataset.py attaches a per-prompt profile_band
block to a profiled JSONL (rows with profiled_rewards, profiled_output_lengths, profiled_reasoning_lengths,
profiled_answer_lengths, pass_rate).

Docs: docs/guides/length-penalty.md (registered in the Guides toctree).

Hook locations on main

File Change
nemo_rl/experience/rollouts.py run_async_nemo_gym_rollout, run_nemo_gym_rollout_sync, _postprocess_single_nemo_gym_group take length_penalty_config; after apply_reward_penalties, copy agent_ref / profiled_* / profile_band from the Gym rows onto results and call apply_group_length_penalties(results, {"grpo": grpo_config}, tokenizer)
nemo_rl/algorithms/grpo.py GRPOConfig.length_penalty: Optional[LengthPenaltyConfig] = None; _grpo_train_impl passes master_config.grpo.length_penalty; validate() intentionally does not (val metrics stay on the raw env reward; grouping stride would also mismatch val_num_generations_per_prompt); baseline_reward/pct_* read the env_reward column when present
nemo_rl/algorithms/async_utils/trajectory_collector.py same forwarding on the async GRPO path
nemo_rl/experience/sync_rollout_actor.py, nemo_rl/algorithms/grpo_sync.py rollout_to_tq(apply_length_penalty=...); grpo_train_sync passes True for training rollouts only
tests/unit/reference_configs/grpo_math_1B.yaml length_penalty: null so test_config_v2_same_as_v1 still passes

Per-source-commit disposition

Source (super-v3.5-posttraining) Disposition
0ca223057 feat: super v3.5 posttraining with flexible length penalties (#3852) Ported via git cherry-pick -x -n + one import-order conflict resolved in rollouts.py. Split into 3 commits (core+hooks, docs, script). Doc moved from nemo_rl/utils/length-penalty.md to docs/guides/. Added typed GRPOConfig.length_penalty field (super-v3.5 relied on extra="allow"). Dropped examples/nemo_gym/nemotron-3-super/super_launch.sh hunk: unrelated Slurm/launcher plumbing (RESULTS_DIR layout, QOS/segment/comment flags).
20ae0bb86 fix(grpo): apply effort shaping to async Gym rollouts (#3897) effort_config forwarding is already on main; ported only the remaining length_penalty_config line (in commit 1) and extended the existing retry test to assert the forwarded block (commit 4).
dd1f771d2 fp32 LM head + fix max-length handling (#3936) Not included: touches vLLM/MCore fp32 head and context-overflow handling, none of the length-penalty files.
origin/adithyare/gdpo-lenpen-no-truncation-main (older port) Used only as a placement guide; 35251ee0d (HSG recipe) and all other commits not included. The super-v3.5 versions are newer (renamed length_bonus -> length_penalty, pass-rate-scaled penalty, binary-reward restriction, clamp-at-0, think-close fix, profile_band defaults/clamp).
Out of scope by request pjin's max_output_tokens = None (#4333), joint-MOPD/vision, skip_verification, Gym bumps.

How tested

  • tests/unit/utils/test_length_penalty.py: 49 passed, 2 skipped (python -m pytest tests/unit/utils/test_length_penalty.py -q --noconftest; the 2 skipped are TestNemoGymPostprocessHook, which imports torch and runs on a compute node / CI).
  • tests/unit/algorithms/test_grpo.py::test_grpo_train_sync_applies_length_penalty_to_training_rollouts: not executed locally (torch); compute node / CI.
  • tests/unit/algorithms/test_async_utils.py::TestAsyncTrajectoryCollector::test_nemo_gym_batch_retry_forwards_effort_config_without_duplicates: not executed locally. No available environment imports main's nemo.lens dependency (worktree venv could not be synced offline; 3rdparty submodules uninitialized). File passes py_compile, ruff check, ruff format --check. Please rely on CI for this one.
  • tests/unit/test_config_v2.py: not executed locally (same import issue). Reference yaml updated the same way dd1f771d2 did for new GRPOConfig fields.
  • ruff check / ruff format --check (ruff 0.16.5): clean on all touched Python files.
  • No GPU jobs run.

Review follow-ups (commit 5)

Addresses the inline review on #4553. Per comment:

Comment Change
length_penalty.py:294-316 (binary gate) enabled is resolved before the binary check, so agent_overrides.<agent>.enabled: false skips silently; groups with reward_components are skipped like non-binary ones (warn once per agent). Tests test_disabled_graded_agent_is_skipped_without_warning, test_multi_reward_group_skipped.
length_penalty.py:134-135 (answer length) _extract_reasoning_and_answer_text counts only message items with role assistant/None; env observations appended as user messages are ignored. Test test_env_observation_messages_not_counted.
grpo.py:403 (typed config) LengthPenaltyConfig / LengthPenaltyParams / ProfileBandConfig / ProfileBandChannel pydantic models (extra="forbid", Literal for length_type and gate channel/field, f in [0, 1], b > a). agent_overrides merge via model_dump(exclude_unset=True); global-band auto-enable from default.model_fields_set. GRPOConfig.length_penalty: Optional[LengthPenaltyConfig] = None; call sites pass the block itself; _postprocess_single_nemo_gym_group(group_size=...) filled from identity_num_generations; apply_group_length_penalties(results, block, group_size, tokenizer).
grpo.py:3292 (sync path) rollout_to_tq(apply_length_penalty: bool = False) forwards cfg.grpo.length_penalty to run_nemo_gym_rollout_sync; grpo_train_sync sets it True for training only. Test test_grpo_train_sync_applies_length_penalty_to_training_rollouts (torch; compute node).
rollouts.py:3030 (metrics) apply_group_length_penalties returns length_penalty/{env_reward_mean,reward_delta_mean,wiped_correct_frac,adjusted_frac,skipped_non_binary_frac}, merged into rollout_metrics next to the sibling shapers; pre-hook reward kept as batch column env_reward (also carried through driver_carry on the TQ path); baseline_reward/pct_* computed from an env_reward baseline in both drivers.
docs/guides/length-penalty.md:139-175 Single top-level JSONL example (reviewer's text); guide also updated for the typed config, chars, model-only lengths, the metrics / env_reward, and the sync-path support.
tests/unit/utils/test_length_penalty.py:595 TestNemoGymPostprocessHook added (adapted to the typed block + group_size; also asserts env_reward and the metrics). Needs torch, so it skips locally and runs on the compute node / CI.
docs/guides/length-penalty.md:6-7 (SingleController) Deferred to a follow-up PR. Guide now states plainly that the SC path does not apply length penalties yet and that support is planned as a follow-up.

Second round (commit 6):

Comment Change
docs/guides/length-penalty.md:7 reply (SC/native guards) SC: ("length_penalty", getattr(algo_cfg, "length_penalty", None) is not None) added to the unsupported list in _validate_algo_settings, so an SC run with the block set raises NotImplementedError at setup; None and PPO configs pass. Native: new _raise_if_length_penalty_enabled_without_nemo_gym in grpo.py, called next to the reward-penalties guard in setup(), raises ValueError when the block is set and env.should_use_nemo_gym is false. Tests: test_ppo_setup.py::test_rejects_length_penalty_until_sc_applies_it / test_accepts_absent_length_penalty[grpo,ppo]; test_grpo.py::test_raise_if_length_penalty_enabled_without_nemo_gym_{noops_when_unset,raises,allows_nemo_gym}. Guide updated.
rollouts.py:3030 reply (per-group metric bias) length_penalty/wiped_correct_frac is now a per-row rate (wiped correct rollouts / rollouts), so the async path's per-group averaging matches the batched paths; the other four metrics were already per-row. Test test_metrics_match_batched_and_per_group (reviewer's all-wrong + one-wiped example: 1/8 either way). Guide definition changed to "fraction of rollouts".

Follow-up: SingleController path

nemo_rl/experience/rollout_manager.py (SC Gym path) applies _apply_effort_shaping and
apply_reward_penalties per streamed row so that recovery checkpoints never hold a raw reward
that finalization later trains on. Length penalties are group-level (they need every sibling's
reward and length), so they cannot be applied at the same point without either (a) applying them at
group finalization and accepting that streamed recovery rows carry the pre-penalty reward, or (b)
re-running the penalty on group reassembly after recovery. This needs a design decision, so SC is
not hooked in this PR; grpo.length_penalty is a no-op there (documented in the guide).

Risks

  • GRPOConfig.length_penalty becomes a typed field (LengthPenaltyConfig, extra="forbid"); configs that
    already carried a well-formed grpo.length_penalty block keep working, configs with unknown keys or bad
    values now fail at load instead of at the first rollout.
  • Penalties only run when the block is present and default.enabled / agent_overrides / profile_band
    activate them; otherwise the hook is a dict lookup.
  • Validation deliberately reports raw env reward; training-time reward metrics include the adjustments.
  • Non-binary-reward envs are skipped with a one-time warning per agent rather than erroring.

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Oct 7, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Oct 7, 2026
@arendu
arendu force-pushed the adithyare/main-pr-length-penalty branch 2 times, most recently from 4ab0df8 to b8d743f Compare October 7, 2026 08:00
@arendu
arendu marked this pull request as ready for review October 7, 2026 16:51
@copy-pr-bot

copy-pr-bot Bot commented Oct 7, 2026

Copy link
Copy Markdown

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

arendu and others added 4 commits October 7, 2026 10:24
Port the Super 3.5 length-penalty feature set to main. Adds
nemo_rl/utils/length_penalty.py, which applies per-prompt-group length
bonuses/penalties to NeMo-Gym rollout rewards in place, configured by a new
optional `grpo.length_penalty` block (`default`, `agent_overrides`,
`profile_band`, `verbose`). Families: shortest-correct bonus, longest-correct
penalty, zero-centered group length penalty (reasoning/answer/total channels),
zMAD high-side outlier penalty, profiled-length penalty, pass-rate-scaled
length penalty, and per-prompt/global profile-band multipliers (clamped at `f`
past `b`). Adjustments are restricted to binary-reward envs (warn once per
agent, then skip) and rewards are clamped at 0.

Hooks (training only, validation keeps the raw env reward):
- rollouts.py `_postprocess_single_nemo_gym_group`: after
  `apply_reward_penalties`, copy agent_ref/profiled_*/profile_band from the
  Gym rows onto the results and call `apply_group_length_penalties`.
  `run_async_nemo_gym_rollout` / `run_nemo_gym_rollout_sync` accept and
  forward `length_penalty_config`.
- grpo.py `_grpo_train_impl`: pass `master_config.grpo.model_dump()` as
  `length_penalty_config`; `validate()` deliberately does not.
- trajectory_collector.py (async GRPO): same forwarding.
- GRPOConfig gains a typed `length_penalty: dict | None = None` field (the
  module rejects unknown keys itself); reference config updated so the
  v1/v2 config parity test still passes. Existing configs are unaffected.

The SingleController Gym path (rollout_manager.py) is not hooked here: it
shapes rewards per streamed row before the group completes, while these
penalties are group-level. Tracked as a follow-up.

(cherry picked from commit 0ca2230, minus
the unrelated examples/nemo_gym/nemotron-3-super/super_launch.sh changes)

Co-authored-by: Yi-Fu Wu <yifu.wu@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: adithyare <adithya.r@gmail.com>
Move the Super 3.5 `nemo_rl/utils/length-penalty.md` into docs/guides and
register it in the Guides toctree. Documents the binary-reward requirement,
the `grpo.length_penalty` config block, flag reference, per-prompt data
format (profiled_* fields and profile_band), and each implemented algorithm.
Generic examples only.

(cherry picked from commit 0ca2230)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: adithyare <adithya.r@gmail.com>
Offline tool that attaches a per-prompt `profile_band` block to a profiled
training JSONL (rows carrying profiled_rewards / profiled_output_lengths /
profiled_reasoning_lengths / profiled_answer_lengths / pass_rate), for use
with `grpo.length_penalty.default.profile_band_*`.

(cherry picked from commit 0ca2230)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: adithyare <adithya.r@gmail.com>
Extend the existing NeMo-Gym batch-retry test so it also checks that
AsyncTrajectoryCollector forwards the grpo block (with `length_penalty`) to
run_async_nemo_gym_rollout. The effort_config forwarding from the original
Super 3.5 fix (#3897) was already on main; only the length-penalty part is
new here.

(ported from commit 20ae0bb)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: adithyare <adithya.r@gmail.com>
@arendu
arendu force-pushed the adithyare/main-pr-length-penalty branch from b8d743f to 9b61c40 Compare October 7, 2026 17:25
@arendu
arendu requested review from aroshanghias-nvd and yfw October 7, 2026 21:45

@yfw yfw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for porting this. Keeping the length shaping in a pure-Python module behind an opt-in hook, with Ray-free unit tests, is a good shape for it.

The main item is parity (grpo.py L3292 and guide L6). grpo_train_sync and the SingleController path accept grpo.length_penalty but never apply it, with no error. On token-capture SC it should be applied when each group is finalized, using the rewards saved in the recovery ledger.

Test status at this head: pre-commit run --all-files passes, and tests/unit/utils/test_length_penalty.py passes (41). The two runs the description lists as not executed both pass:

  • tests/unit/test_config_v2.py: 15 passed. The new reference-config line is required; without it test_config_v2_same_as_v1 fails.
  • The async retry-forwarding test: passes, and goes red if the new kwarg at trajectory_collector.py L1551 is dropped.

Generated by Claude Code

Comment thread nemo_rl/utils/length_penalty.py Resolved
Comment thread nemo_rl/algorithms/grpo.py Resolved
Comment thread docs/guides/length-penalty.md Resolved
Comment thread nemo_rl/experience/rollouts.py Resolved
Comment thread tests/unit/utils/test_length_penalty.py Resolved
Comment thread nemo_rl/utils/length_penalty.py Resolved
Comment thread nemo_rl/algorithms/grpo.py Resolved
Comment thread docs/guides/length-penalty.md Resolved
…h, metrics, binary gate)

Addresses the inline review on #4553:

- length_penalty.py: resolve `enabled` before the binary check so a
  disabled agent is skipped silently; skip groups carrying
  `reward_components` like non-binary ones (adjusting only the scalar
  would break reward == sum(reward_components)); warn once per agent.
- length_penalty.py: count only model messages (role assistant/None)
  when measuring answer length; env observations appended as user-role
  messages no longer rank rollouts.
- Type `grpo.length_penalty` as pydantic models (LengthPenaltyConfig,
  LengthPenaltyParams, ProfileBandConfig, ProfileBandChannel) with
  extra="forbid", Literal length_type / gate channel / gate field,
  f in [0, 1] and b > a. Merge agent_overrides via
  model_dump(exclude_unset=True); decide the global-band auto-enable
  from default.model_fields_set. GRPOConfig.length_penalty is
  Optional[LengthPenaltyConfig] = None. Call sites pass the block itself;
  _postprocess_single_nemo_gym_group takes group_size from
  identity_num_generations; apply_group_length_penalties takes
  (results, block, group_size, tokenizer).
- Forward the penalty on the grpo_train_sync path: new
  rollout_to_tq(apply_length_penalty=False) kwarg, True for training
  rollouts only (validation untouched).
- apply_group_length_penalties returns length_penalty/* metrics
  (env_reward_mean, reward_delta_mean, wiped_correct_frac,
  adjusted_frac, skipped_non_binary_frac) merged into rollout_metrics;
  the pre-hook reward is kept as batch column `env_reward` and
  baseline_reward/pct_* are computed from its per-prompt baseline in
  both drivers.
- Guide: single top-level JSONL example, typed config, metrics,
  env_reward, sync-path support; SingleController support stays a
  documented follow-up.
- Tests: reviewer regression tests, typed-config/metrics/role-filter
  tests, TestNemoGymPostprocessHook (torch), grpo_train_sync
  apply_length_penalty assertion; async collector test updated for the
  typed block.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: adithyare <adithya.r@gmail.com>
@arendu

arendu commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Pushed ce0e01460 addressing all review items except SingleController parity (deferred to a stacked follow-up PR; see the thread on the guide). Summary of the follow-up commit: enabled resolved before the binary gate + multi-reward groups skipped; answer length counts model messages only; grpo.length_penalty is a typed pydantic block (extra="forbid", literals, bounds) passed as the block itself; grpo_train_sync applies it to training rollouts; length_penalty/* metrics + env_reward column with baseline_reward/pct_* computed from the env reward; guide fixed (top-level row fields) and extended; hook tests added.

Verification on an HSG GB200 node (super-v3.5 NeMo-RL container, nemo-lens at main's pin): test_length_penalty.py + test_async_utils.py + test_config_v2.py -> 183 passed; test_grpo.py sync tests -> 2 passed. Ruff check/format clean.

@arendu
arendu requested a review from yfw October 8, 2026 16:13
…penalty metrics

Second review round on #4553:

- SingleController: add length_penalty to the `unsupported` list in
  _validate_algo_settings so an SC run with grpo.length_penalty set
  fails at setup with NotImplementedError until the SC hook lands
  (getattr: PPO's config has no such field; None and PPO configs pass).
- Native path: new _raise_if_length_penalty_enabled_without_nemo_gym,
  called in setup() next to the reward-penalties guard, rejects
  grpo.length_penalty when env.should_use_nemo_gym is false instead of
  training on unadjusted rewards.
- length_penalty/wiped_correct_frac is now a per-row rate (wiped
  correct rollouts / rollouts) so the async path, which applies the
  hook per prompt group and averages the per-group values, reports the
  same number as the batched drivers; the other metrics were already
  per-row. Guide definition updated accordingly.
- Tests: SC reject/accept (grpo None, ppo) in test_ppo_setup.py; native
  guard noop/raise/allow in test_grpo.py; batched-vs-per-group metric
  parity in test_length_penalty.py.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: adithyare <adithya.r@gmail.com>
@yfw yfw added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Oct 8, 2026
@yfw

yfw commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

/ok to test 2e6ddab

@yfw
yfw enabled auto-merge (squash) October 8, 2026 22:17
@yfw
yfw merged commit 39eeee7 into main Oct 9, 2026
96 checks passed
@yfw
yfw deleted the adithyare/main-pr-length-penalty branch October 9, 2026 03:24
yuekaizhang pushed a commit to yuekaizhang/RL that referenced this pull request Oct 9, 2026
NVIDIA-NeMo#3882 switched calculate_baseline_and_std_per_prompt to group by
prompt_group_ids and removed input_ids from _grpo_train_impl; NVIDIA-NeMo#4553,
merged right after, added the env-reward baseline call still passing
input_ids. On main that is an undefined name (ruff F821, failing Lint
check) and a NameError whenever grpo.length_penalty sets env_reward.

Use repeated_batch[PROMPT_GROUP_IDS_KEY] there, and prompt_group_ids
(instead of prompt token ids) for the same baseline in grpo_sync, so the
env-reward baseline groups rows exactly like the main baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>
(cherry picked from commit 5328e0c)
linj-glitch added a commit to linj-glitch/RL that referenced this pull request Oct 13, 2026
main grew a fourth all-ones valid-mask site after NVIDIA-NeMo#4553/NVIDIA-NeMo#4620: the
env_baseline that grpo.length_penalty keeps for the pct_* diagnostics.
Pass the same participation mask there, so a masked row's env reward
does not vote in its siblings' env baseline either.

State the two rules the mask follows in the helper docstring and at
both call sites. Participation is binary: a nonzero fractional
loss_multiplier still counts as one vote, and the training loss
weights are never changed. Truncated rollouts keep voting under
overlong_filtering because the mask is taken before that rewrite,
which is what docs/guides/grpo.md documents for these loops; the
SingleController advantage_stage folds truncated rows into its
valid_mask instead, and reconciling the two is left to the
configurable policy work rather than decided here.

Tests: fractional-nonzero versus zero loss weight, the truncated-row
call-order rule, one driver test asserting the trivial check, the
baseline/std and the env baseline all receive the mask (and that the
env baseline of the real rows ignores the masked reward), a fractional
case for the sync/async forwarding test plus an assertion that the
forwarded sample_mask keeps the original weight, and a sync/async test
that a truncated row votes in the baseline while overlong filtering
removes it from the loss.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
linj-glitch added a commit to linj-glitch/RL that referenced this pull request Oct 13, 2026
Reconciles the straggler cutoff with what main gained on the NeMo-Gym
rollout path since the branch was cut: reward penalties and effort shaping
(NVIDIA-NeMo#4099), per-prompt-group length penalties (NVIDIA-NeMo#4553) and per-agent token
metrics (NVIDIA-NeMo#4379).

- A cut placeholder is not a model output. _postprocess_single_nemo_gym_group
  now runs effort shaping and the reward-penalty checks over the scored rows
  only, and reports the penalty rates over that population, so an empty
  placeholder response is not counted as an "empty final answer". Cut rows
  also stay out of main's new per-agent token/truncation metrics, alongside
  the existing exclusion from the per-agent full_result means. Length
  penalties need no change: they only adjust rows with a positive reward.
- Token-capture (receipt) mode is unsupported: a cancelled rollout has no
  receipt for the finalizer to rebuild. The actor now logs once and runs the
  batch uncut, the same fallback as a Gym without aclose(); the warning text
  names the reason.

Tests: receipt-mode fallback; penalty rate and rewards with a cut row under
penalize_empty_final_answer; per-agent gen-token mean excludes the cut row.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
linj-glitch added a commit to linj-glitch/RL that referenced this pull request Oct 23, 2026
main grew a fourth all-ones valid-mask site after NVIDIA-NeMo#4553/NVIDIA-NeMo#4620: the
env_baseline that grpo.length_penalty keeps for the pct_* diagnostics.
Pass the same participation mask there, so a masked row's env reward
does not vote in its siblings' env baseline either.

State the two rules the mask follows in the helper docstring and at
both call sites. Participation is binary: a nonzero fractional
loss_multiplier still counts as one vote, and the training loss
weights are never changed. Truncated rollouts keep voting under
overlong_filtering because the mask is taken before that rewrite,
which is what docs/guides/grpo.md documents for these loops; the
SingleController advantage_stage folds truncated rows into its
valid_mask instead, and reconciling the two is left to the
configurable policy work rather than decided here.

Tests: fractional-nonzero versus zero loss weight, the truncated-row
call-order rule, one driver test asserting the trivial check, the
baseline/std and the env baseline all receive the mask (and that the
env baseline of the real rows ignores the masked reward), a fractional
case for the sync/async forwarding test plus an assertion that the
forwarded sample_mask keeps the original weight, and a sync/async test
that a truncated row votes in the baseline while overlong filtering
removes it from the loss.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants