Repository navigation
feat(grpo): per-prompt-group length penalties for NeMo-Gym GRPO rollouts - #4553
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
4ab0df8 to
b8d743f
Compare
|
Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Port the Super 3.5 length-penalty feature set to main. Adds nemo_rl/utils/length_penalty.py, which applies per-prompt-group length bonuses/penalties to NeMo-Gym rollout rewards in place, configured by a new optional `grpo.length_penalty` block (`default`, `agent_overrides`, `profile_band`, `verbose`). Families: shortest-correct bonus, longest-correct penalty, zero-centered group length penalty (reasoning/answer/total channels), zMAD high-side outlier penalty, profiled-length penalty, pass-rate-scaled length penalty, and per-prompt/global profile-band multipliers (clamped at `f` past `b`). Adjustments are restricted to binary-reward envs (warn once per agent, then skip) and rewards are clamped at 0. Hooks (training only, validation keeps the raw env reward): - rollouts.py `_postprocess_single_nemo_gym_group`: after `apply_reward_penalties`, copy agent_ref/profiled_*/profile_band from the Gym rows onto the results and call `apply_group_length_penalties`. `run_async_nemo_gym_rollout` / `run_nemo_gym_rollout_sync` accept and forward `length_penalty_config`. - grpo.py `_grpo_train_impl`: pass `master_config.grpo.model_dump()` as `length_penalty_config`; `validate()` deliberately does not. - trajectory_collector.py (async GRPO): same forwarding. - GRPOConfig gains a typed `length_penalty: dict | None = None` field (the module rejects unknown keys itself); reference config updated so the v1/v2 config parity test still passes. Existing configs are unaffected. The SingleController Gym path (rollout_manager.py) is not hooked here: it shapes rewards per streamed row before the group completes, while these penalties are group-level. Tracked as a follow-up. (cherry picked from commit 0ca2230, minus the unrelated examples/nemo_gym/nemotron-3-super/super_launch.sh changes) Co-authored-by: Yi-Fu Wu <yifu.wu@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: adithyare <adithya.r@gmail.com>
Move the Super 3.5 `nemo_rl/utils/length-penalty.md` into docs/guides and register it in the Guides toctree. Documents the binary-reward requirement, the `grpo.length_penalty` config block, flag reference, per-prompt data format (profiled_* fields and profile_band), and each implemented algorithm. Generic examples only. (cherry picked from commit 0ca2230) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: adithyare <adithya.r@gmail.com>
Offline tool that attaches a per-prompt `profile_band` block to a profiled training JSONL (rows carrying profiled_rewards / profiled_output_lengths / profiled_reasoning_lengths / profiled_answer_lengths / pass_rate), for use with `grpo.length_penalty.default.profile_band_*`. (cherry picked from commit 0ca2230) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: adithyare <adithya.r@gmail.com>
Extend the existing NeMo-Gym batch-retry test so it also checks that AsyncTrajectoryCollector forwards the grpo block (with `length_penalty`) to run_async_nemo_gym_rollout. The effort_config forwarding from the original Super 3.5 fix (#3897) was already on main; only the length-penalty part is new here. (ported from commit 20ae0bb) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: adithyare <adithya.r@gmail.com>
b8d743f to
9b61c40
Compare
yfw
left a comment
There was a problem hiding this comment.
Thanks for porting this. Keeping the length shaping in a pure-Python module behind an opt-in hook, with Ray-free unit tests, is a good shape for it.
The main item is parity (grpo.py L3292 and guide L6). grpo_train_sync and the SingleController path accept grpo.length_penalty but never apply it, with no error. On token-capture SC it should be applied when each group is finalized, using the rewards saved in the recovery ledger.
Test status at this head: pre-commit run --all-files passes, and tests/unit/utils/test_length_penalty.py passes (41). The two runs the description lists as not executed both pass:
tests/unit/test_config_v2.py: 15 passed. The new reference-config line is required; without ittest_config_v2_same_as_v1fails.- The async retry-forwarding test: passes, and goes red if the new kwarg at
trajectory_collector.pyL1551 is dropped.
Generated by Claude Code
…h, metrics, binary gate) Addresses the inline review on #4553: - length_penalty.py: resolve `enabled` before the binary check so a disabled agent is skipped silently; skip groups carrying `reward_components` like non-binary ones (adjusting only the scalar would break reward == sum(reward_components)); warn once per agent. - length_penalty.py: count only model messages (role assistant/None) when measuring answer length; env observations appended as user-role messages no longer rank rollouts. - Type `grpo.length_penalty` as pydantic models (LengthPenaltyConfig, LengthPenaltyParams, ProfileBandConfig, ProfileBandChannel) with extra="forbid", Literal length_type / gate channel / gate field, f in [0, 1] and b > a. Merge agent_overrides via model_dump(exclude_unset=True); decide the global-band auto-enable from default.model_fields_set. GRPOConfig.length_penalty is Optional[LengthPenaltyConfig] = None. Call sites pass the block itself; _postprocess_single_nemo_gym_group takes group_size from identity_num_generations; apply_group_length_penalties takes (results, block, group_size, tokenizer). - Forward the penalty on the grpo_train_sync path: new rollout_to_tq(apply_length_penalty=False) kwarg, True for training rollouts only (validation untouched). - apply_group_length_penalties returns length_penalty/* metrics (env_reward_mean, reward_delta_mean, wiped_correct_frac, adjusted_frac, skipped_non_binary_frac) merged into rollout_metrics; the pre-hook reward is kept as batch column `env_reward` and baseline_reward/pct_* are computed from its per-prompt baseline in both drivers. - Guide: single top-level JSONL example, typed config, metrics, env_reward, sync-path support; SingleController support stays a documented follow-up. - Tests: reviewer regression tests, typed-config/metrics/role-filter tests, TestNemoGymPostprocessHook (torch), grpo_train_sync apply_length_penalty assertion; async collector test updated for the typed block. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: adithyare <adithya.r@gmail.com>
|
Pushed Verification on an HSG GB200 node (super-v3.5 NeMo-RL container, |
…penalty metrics Second review round on #4553: - SingleController: add length_penalty to the `unsupported` list in _validate_algo_settings so an SC run with grpo.length_penalty set fails at setup with NotImplementedError until the SC hook lands (getattr: PPO's config has no such field; None and PPO configs pass). - Native path: new _raise_if_length_penalty_enabled_without_nemo_gym, called in setup() next to the reward-penalties guard, rejects grpo.length_penalty when env.should_use_nemo_gym is false instead of training on unadjusted rewards. - length_penalty/wiped_correct_frac is now a per-row rate (wiped correct rollouts / rollouts) so the async path, which applies the hook per prompt group and averages the per-group values, reports the same number as the batched drivers; the other metrics were already per-row. Guide definition updated accordingly. - Tests: SC reject/accept (grpo None, ppo) in test_ppo_setup.py; native guard noop/raise/allow in test_grpo.py; batched-vs-per-group metric parity in test_length_penalty.py. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: adithyare <adithya.r@gmail.com>
|
/ok to test 2e6ddab |
NVIDIA-NeMo#3882 switched calculate_baseline_and_std_per_prompt to group by prompt_group_ids and removed input_ids from _grpo_train_impl; NVIDIA-NeMo#4553, merged right after, added the env-reward baseline call still passing input_ids. On main that is an undefined name (ruff F821, failing Lint check) and a NameError whenever grpo.length_penalty sets env_reward. Use repeated_batch[PROMPT_GROUP_IDS_KEY] there, and prompt_group_ids (instead of prompt token ids) for the same baseline in grpo_sync, so the env-reward baseline groups rows exactly like the main baseline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com> (cherry picked from commit 5328e0c)
main grew a fourth all-ones valid-mask site after NVIDIA-NeMo#4553/NVIDIA-NeMo#4620: the env_baseline that grpo.length_penalty keeps for the pct_* diagnostics. Pass the same participation mask there, so a masked row's env reward does not vote in its siblings' env baseline either. State the two rules the mask follows in the helper docstring and at both call sites. Participation is binary: a nonzero fractional loss_multiplier still counts as one vote, and the training loss weights are never changed. Truncated rollouts keep voting under overlong_filtering because the mask is taken before that rewrite, which is what docs/guides/grpo.md documents for these loops; the SingleController advantage_stage folds truncated rows into its valid_mask instead, and reconciling the two is left to the configurable policy work rather than decided here. Tests: fractional-nonzero versus zero loss weight, the truncated-row call-order rule, one driver test asserting the trivial check, the baseline/std and the env baseline all receive the mask (and that the env baseline of the real rows ignores the masked reward), a fractional case for the sync/async forwarding test plus an assertion that the forwarded sample_mask keeps the original weight, and a sync/async test that a truncated row votes in the baseline while overlong filtering removes it from the loss. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
Reconciles the straggler cutoff with what main gained on the NeMo-Gym rollout path since the branch was cut: reward penalties and effort shaping (NVIDIA-NeMo#4099), per-prompt-group length penalties (NVIDIA-NeMo#4553) and per-agent token metrics (NVIDIA-NeMo#4379). - A cut placeholder is not a model output. _postprocess_single_nemo_gym_group now runs effort shaping and the reward-penalty checks over the scored rows only, and reports the penalty rates over that population, so an empty placeholder response is not counted as an "empty final answer". Cut rows also stay out of main's new per-agent token/truncation metrics, alongside the existing exclusion from the per-agent full_result means. Length penalties need no change: they only adjust rows with a positive reward. - Token-capture (receipt) mode is unsupported: a cancelled rollout has no receipt for the finalizer to rebuild. The actor now logs once and runs the batch uncut, the same fallback as a Gym without aclose(); the warning text names the reason. Tests: receipt-mode fallback; penalty rate and rewards with a cut row under penalize_empty_final_answer; per-agent gen-token mean excludes the cut row. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
main grew a fourth all-ones valid-mask site after NVIDIA-NeMo#4553/NVIDIA-NeMo#4620: the env_baseline that grpo.length_penalty keeps for the pct_* diagnostics. Pass the same participation mask there, so a masked row's env reward does not vote in its siblings' env baseline either. State the two rules the mask follows in the helper docstring and at both call sites. Participation is binary: a nonzero fractional loss_multiplier still counts as one vote, and the training loss weights are never changed. Truncated rollouts keep voting under overlong_filtering because the mask is taken before that rewrite, which is what docs/guides/grpo.md documents for these loops; the SingleController advantage_stage folds truncated rows into its valid_mask instead, and reconciling the two is left to the configurable policy work rather than decided here. Tests: fractional-nonzero versus zero loss weight, the truncated-row call-order rule, one driver test asserting the trivial check, the baseline/std and the env baseline all receive the mask (and that the env baseline of the real rows ignores the masked reward), a fractional case for the sync/async forwarding test plus an assertion that the forwarded sample_mask keeps the original weight, and a sync/async test that a truncated row votes in the baseline while overlong filtering removes it from the loss. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
Branch:
adithyare/main-pr-length-penalty(6 commits on top oforigin/main@8bf6bd4ba; DCO signed-off)Summary
Ports the Super 3.5 length-penalty feature set (NeMo-RL
super-v3.5-posttraining, #3852 + #3897) tomain.It adds an optional
grpo.length_penaltyconfig block that applies per-prompt-group lengthbonuses/penalties to NeMo-Gym rollout rewards in place during rollout postprocessing. Runs
without the block are untouched (
GRPOConfig.length_penaltydefaults toNone).Penalty families (all resolved per prompt group; binary 0/1 env rewards only, rewards clamped at 0):
reasoning_bonus/answer_bonus/total_bonus, gated bytop_percentile).longest_{reasoning,answer,total}_penalty).group_{reasoning,answer,total}_length_penalty_coeff),optionally gated by a per-prompt profile field (
group_length_penalty_profile_gate*).profiled_length_penalty,_n_std,_min_samples).pass_rate_length_penalty_weight).profile_band_{total,reasoning,answer}) using per-prompt{a, b, f}bands withglobal defaults under
grpo.length_penalty.profile_band; multiplier clamps atfpastb.agent_overrides.<agent_name>) andlength_type: tokens|chars.LengthPenaltyConfig,extra="forbid"): unknown keys, badLiteralvalues and malformed profile bands are rejected at config load.
Config block (minimal example):
Profile-band dataset tool:
scripts/build_profile_band_dataset.pyattaches a per-promptprofile_bandblock to a profiled JSONL (rows with
profiled_rewards,profiled_output_lengths,profiled_reasoning_lengths,profiled_answer_lengths,pass_rate).Docs:
docs/guides/length-penalty.md(registered in the Guides toctree).Hook locations on main
nemo_rl/experience/rollouts.pyrun_async_nemo_gym_rollout,run_nemo_gym_rollout_sync,_postprocess_single_nemo_gym_grouptakelength_penalty_config; afterapply_reward_penalties, copyagent_ref/profiled_*/profile_bandfrom the Gym rows onto results and callapply_group_length_penalties(results, {"grpo": grpo_config}, tokenizer)nemo_rl/algorithms/grpo.pyGRPOConfig.length_penalty: Optional[LengthPenaltyConfig] = None;_grpo_train_implpassesmaster_config.grpo.length_penalty;validate()intentionally does not (val metrics stay on the raw env reward; grouping stride would also mismatchval_num_generations_per_prompt);baseline_reward/pct_*read theenv_rewardcolumn when presentnemo_rl/algorithms/async_utils/trajectory_collector.pynemo_rl/experience/sync_rollout_actor.py,nemo_rl/algorithms/grpo_sync.pyrollout_to_tq(apply_length_penalty=...);grpo_train_syncpassesTruefor training rollouts onlytests/unit/reference_configs/grpo_math_1B.yamllength_penalty: nullsotest_config_v2_same_as_v1still passesPer-source-commit disposition
0ca223057feat: super v3.5 posttraining with flexible length penalties (#3852)git cherry-pick -x -n+ one import-order conflict resolved inrollouts.py. Split into 3 commits (core+hooks, docs, script). Doc moved fromnemo_rl/utils/length-penalty.mdtodocs/guides/. Added typedGRPOConfig.length_penaltyfield (super-v3.5 relied onextra="allow"). Droppedexamples/nemo_gym/nemotron-3-super/super_launch.shhunk: unrelated Slurm/launcher plumbing (RESULTS_DIR layout, QOS/segment/comment flags).20ae0bb86fix(grpo): apply effort shaping to async Gym rollouts (#3897)effort_configforwarding is already on main; ported only the remaininglength_penalty_configline (in commit 1) and extended the existing retry test to assert the forwarded block (commit 4).dd1f771d2fp32 LM head + fix max-length handling (#3936)origin/adithyare/gdpo-lenpen-no-truncation-main(older port)35251ee0d(HSG recipe) and all other commits not included. The super-v3.5 versions are newer (renamedlength_bonus->length_penalty, pass-rate-scaled penalty, binary-reward restriction, clamp-at-0, think-close fix, profile_band defaults/clamp).max_output_tokens = None(#4333), joint-MOPD/vision, skip_verification, Gym bumps.How tested
tests/unit/utils/test_length_penalty.py: 49 passed, 2 skipped (python -m pytest tests/unit/utils/test_length_penalty.py -q --noconftest; the 2 skipped areTestNemoGymPostprocessHook, which imports torch and runs on a compute node / CI).tests/unit/algorithms/test_grpo.py::test_grpo_train_sync_applies_length_penalty_to_training_rollouts: not executed locally (torch); compute node / CI.tests/unit/algorithms/test_async_utils.py::TestAsyncTrajectoryCollector::test_nemo_gym_batch_retry_forwards_effort_config_without_duplicates: not executed locally. No available environment imports main'snemo.lensdependency (worktree venv could not be synced offline; 3rdparty submodules uninitialized). File passespy_compile,ruff check,ruff format --check. Please rely on CI for this one.tests/unit/test_config_v2.py: not executed locally (same import issue). Reference yaml updated the same waydd1f771d2did for newGRPOConfigfields.ruff check/ruff format --check(ruff 0.16.5): clean on all touched Python files.Review follow-ups (commit 5)
Addresses the inline review on #4553. Per comment:
length_penalty.py:294-316(binary gate)enabledis resolved before the binary check, soagent_overrides.<agent>.enabled: falseskips silently; groups withreward_componentsare skipped like non-binary ones (warn once per agent). Teststest_disabled_graded_agent_is_skipped_without_warning,test_multi_reward_group_skipped.length_penalty.py:134-135(answer length)_extract_reasoning_and_answer_textcounts onlymessageitems with roleassistant/None; env observations appended asusermessages are ignored. Testtest_env_observation_messages_not_counted.grpo.py:403(typed config)LengthPenaltyConfig/LengthPenaltyParams/ProfileBandConfig/ProfileBandChannelpydantic models (extra="forbid",Literalforlength_typeand gate channel/field,fin[0, 1],b > a).agent_overridesmerge viamodel_dump(exclude_unset=True); global-band auto-enable fromdefault.model_fields_set.GRPOConfig.length_penalty: Optional[LengthPenaltyConfig] = None; call sites pass the block itself;_postprocess_single_nemo_gym_group(group_size=...)filled fromidentity_num_generations;apply_group_length_penalties(results, block, group_size, tokenizer).grpo.py:3292(sync path)rollout_to_tq(apply_length_penalty: bool = False)forwardscfg.grpo.length_penaltytorun_nemo_gym_rollout_sync;grpo_train_syncsets itTruefor training only. Testtest_grpo_train_sync_applies_length_penalty_to_training_rollouts(torch; compute node).rollouts.py:3030(metrics)apply_group_length_penaltiesreturnslength_penalty/{env_reward_mean,reward_delta_mean,wiped_correct_frac,adjusted_frac,skipped_non_binary_frac}, merged intorollout_metricsnext to the sibling shapers; pre-hook reward kept as batch columnenv_reward(also carried throughdriver_carryon the TQ path);baseline_reward/pct_*computed from anenv_rewardbaseline in both drivers.docs/guides/length-penalty.md:139-175chars, model-only lengths, the metrics /env_reward, and the sync-path support.tests/unit/utils/test_length_penalty.py:595TestNemoGymPostprocessHookadded (adapted to the typed block +group_size; also assertsenv_rewardand the metrics). Needs torch, so it skips locally and runs on the compute node / CI.docs/guides/length-penalty.md:6-7(SingleController)Second round (commit 6):
docs/guides/length-penalty.md:7reply (SC/native guards)("length_penalty", getattr(algo_cfg, "length_penalty", None) is not None)added to theunsupportedlist in_validate_algo_settings, so an SC run with the block set raisesNotImplementedErrorat setup;Noneand PPO configs pass. Native: new_raise_if_length_penalty_enabled_without_nemo_gymingrpo.py, called next to the reward-penalties guard insetup(), raisesValueErrorwhen the block is set andenv.should_use_nemo_gymis false. Tests:test_ppo_setup.py::test_rejects_length_penalty_until_sc_applies_it/test_accepts_absent_length_penalty[grpo,ppo];test_grpo.py::test_raise_if_length_penalty_enabled_without_nemo_gym_{noops_when_unset,raises,allows_nemo_gym}. Guide updated.rollouts.py:3030reply (per-group metric bias)length_penalty/wiped_correct_fracis now a per-row rate (wiped correct rollouts / rollouts), so the async path's per-group averaging matches the batched paths; the other four metrics were already per-row. Testtest_metrics_match_batched_and_per_group(reviewer's all-wrong + one-wiped example: 1/8 either way). Guide definition changed to "fraction of rollouts".Follow-up: SingleController path
nemo_rl/experience/rollout_manager.py(SC Gym path) applies_apply_effort_shapingandapply_reward_penaltiesper streamed row so that recovery checkpoints never hold a raw rewardthat finalization later trains on. Length penalties are group-level (they need every sibling's
reward and length), so they cannot be applied at the same point without either (a) applying them at
group finalization and accepting that streamed recovery rows carry the pre-penalty reward, or (b)
re-running the penalty on group reassembly after recovery. This needs a design decision, so SC is
not hooked in this PR;
grpo.length_penaltyis a no-op there (documented in the guide).Risks
GRPOConfig.length_penaltybecomes a typed field (LengthPenaltyConfig,extra="forbid"); configs thatalready carried a well-formed
grpo.length_penaltyblock keep working, configs with unknown keys or badvalues now fail at load instead of at the first rollout.
default.enabled/agent_overrides/profile_bandactivate them; otherwise the hook is a dict lookup.
rewardmetrics include the adjustments.🤖 Generated with Claude Code