Repository navigation
Conversation
… default Add a third grpo.masked_reward_policy value, exclude_infra, and make it the default. A rollout that the environment masked still votes in its group's baseline and std only when its top-level failure_kind is listed in the new grpo.policy_failure_kinds (default: agent_timeout, agent_command_not_allowed, agent_limit, agent_invalid_output, model_context_limit). Any other masked rollout, including one with no failure_kind, gets dataset weight 0 where RL reads the Gym result, so the include population from #4160 leaves it out on the legacy sync, async, TQ and SingleController trainers. include and exclude behave as before. The SingleController token-capture path carries the raw failure_kind through the recovery ledger (schema v4) and ReassemblyRequest.infra_failure. infra_failure_sample_rate reports the share of rollouts left out. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
exclude_infra stays available as an opt-in value; the default stays include, as in #4160. Update the config default, the example and reference configs, the GRPO guide and the default test. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
yfw
left a comment
There was a problem hiding this comment.
Self review of this draft, left as a comment. There are four inline comments:
- (C1) No test fails if a trainer stops passing
grpo.policy_failure_kindsto its rollout code. Four trainer-level tests and small edits to existing rollout tests close the gap (test_grpo.py#L9586). - (C2) The finalizer test has no row with flag
False. One more parametrize case closes the gap (test_rollout_reassembler.py#L295). - (C3) SingleController drops infrastructure-failed rows from the statistics but logs no
infra_failure_sample_rate(payload.py#L184). - (C4) The YAML comment says that names "may be added", but a recipe list replaces the five defaults (grpo_math_1B.yaml#L16-L17).
These parts are correct at head:
- The worked-example table in the PR description matches a recomputation with the real
calculate_baseline_and_std_per_prompt. - The
includeandexcludevalues ofgrpo.masked_reward_policybehave exactly as before. - The opt-out
env.should_mask_flagged_samples: falseholds on every path. - All tests in the 9 changed test files pass on CPU, except 3 existing tests that need a GPU, which this review did not run. The 22 unchanged test files that use a changed API also pass. Every new test runs in a CI lane.
Generated by Claude Code
…ind forwarding - SingleController now logs infra_failure_sample_rate under exclude_infra: TQReplayBuffer.commit adds it to the rollout metrics on the native path, and finalize_group adds it to the finalizer metrics on the token-capture path. The guide now says that every trainer reports it. - New tests check that each trainer passes grpo.policy_failure_kinds on: legacy grpo_train, the async collector, the TQ sync actor, SingleController setup (TQReplayBuffer and RolloutManager), the RolloutManager store, both places inside the rollout code, and the prompt-group seal. - The finalizer test gets a row whose infra_failure flag is False. - Comments, the guide and the example configurations now say that a recipe that sets grpo.policy_failure_kinds replaces the list (OmegaConf.merge replaces lists), so it must list the defaults with any new name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
|
/ok to test 23cf9bf |
|
/ok to test 23cf9bf |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
Static review of all 27 changed files and their affected trainer/data-plane paths found no demonstrated correctness regression. The current source addresses the earlier forwarding, finalizer False-row, SingleController metric, and list-replacement feedback.
Two nonblocking suggestions remain:
- (C5) Provide a controlled training comparison for the opt-in policy.
- (C6) Shorten repeated in-function policy explanations while retaining their invariants.
Tests and linters were not executed. Upstream Gym source was unavailable in the mount. Two child reports were unavailable despite terminal completion status; their assigned paths were inspected directly before consolidation.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
The full rule lives in docs/guides/grpo.md and in the rollouts.infra_failure_flags docstring. The call sites now keep a short pointer and their local invariant. Comments only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
|
/ok to test dfe2133 |
…comparison A failed judge comparison is still dropped. A rollout whose comparisons all failed is now masked alone through the judge failsafe (reward 0.0, mask_sample, failure_kind judge_failed), and its siblings keep their scores. Before, it failed the whole group. The masked rollout is left out of the group's length and style adjustments. When every rollout is left uncovered, the whole group fails as a judge failure, as before. - The server now waits for every comparison to finish or fail, within cohort_evaluation_timeout_s. A non-judge error still fails the group at once and cancels the comparisons still running. - The fixed_baseline path follows the same rule. /compare has no per-response mask, so it still returns 503 when a response keeps no comparison. - Checkpoints save and restore a member masked alone (masked_alone). - New metric: genrm_masked_rollout_rate_per_group. The masked rollout's reward is a placeholder, so a trainer must keep it out of the group statistics: in NeMo RL, grpo.masked_reward_policy exclude_infra (NVIDIA-NeMo/RL#4968) or exclude. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
…kind-advantages Main added rollout recovery schema v4 (token-capture receipts, #4124) and v5 (logical_selection, #4200), so failure_kind moves to schema v6. The ledger still loads v2 to v5, with failure_kind=None. Conflicts kept both sides: failure_kind next to logical_selection in the ledger, SiblingSealResult and both seal calls; infra_failure next to the Forest fields in ReassemblyRequest; the infra metric next to the logical owner counts; the infra sample_mask next to the sampler top-k copy. Main moved row building into _publish_rows, so finalize_group now passes infra_failure in. With forest_training, each segment row takes its rollout's flag, and infra_failure_sample_rate counts rollouts. Tests: the v3 load test is now a v5 load test; the finalizer test sets capture_key, as main's receipts need; a new Forest test checks that the segment rows of a flagged rollout get sample_mask 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
|
/ok to test ab0a647 |
What does this PR do ?
Adds a third
grpo.masked_reward_policyvalue,exclude_infra. With it, a rollout that the environment masked for an infrastructure reason, or with nofailure_kindat all, no longer moves the baseline and advantages of the valid rollouts in its group. The default staysinclude, so existing recipes do not change.This is the follow-up promised in #4160.
Issues
Related to #4061 and follows up on #4160. On the Gym side, NVIDIA-NeMo/Gym#4882 is adding
failure_kindto masked results that lack one.Problem
#4160 added
grpo.masked_reward_policy: include | exclude, withincludeas the default. Underinclude, a masked rollout is still part of its group's baseline and std. It never trains, but its reward still moves the advantages of its valid siblings, including when that reward measured nothing.Example: OSWorld masks evaluator errors and gives them no
failure_kind. Take a group with rewards[1, 1, 0], where the 0 is an evaluator error. Underinclude, the two valid rollouts get raw advantages[0.5, 0.5]instead of[0, 0].excludeavoids this, but it also drops masks that the policy itself caused (timeouts, step limits, context overflow) and trainer-filtered rows. Those rewards are real measurements of the policy.Change
exclude_infra(the default staysinclude). A masked rollout still counts in its group's baseline and std only if its top-levelfailure_kindis in the newgrpo.policy_failure_kinds. The default list isagent_timeout,agent_command_not_allowed,agent_limit,agent_invalid_outputandmodel_context_limit.<server>:<kind>names. Only that rollout leaves; the rest of its group still trains.OmegaConf.mergereplaces lists. So a recipe that adds a name must also list the defaults.mask_sample.includeandexcludebehave exactly as before.env.should_mask_flagged_samples: false, nothing is masked, so nothing is treated as infra.include(default)exclude_infraexcludepolicy_failure_kindsHow it works
No new data-plane or trainer column is needed.
loss_multiplier), the same as an unusable over-length prompt. Theincludepopulation from feat(grpo): configurable masked-reward participation across the four trainers #4160 (_include_valid_mask) then leaves it out on all four trainers: legacy sync, async, TQ and SingleController.grpo.py:_votes_by_dataset_weight()is true forincludeandexclude_infra. The estimator, dynamic-sampling admission and the logprob mask use it.exclude_infra_policy_kinds()returns the policy list only underexclude_infra.rollouts.infra_failure_flags()marks the rows. The group builder sets theirloss_multiplierto 0 and logsinfra_failure_sample_rate.record_to_train_batchsetssample_maskto 0.TQReplayBuffercarries the policy list and logsinfra_failure_sample_rate.failure_kindinto the recovery ledger.ReassemblyRequest.infra_failuremakesfinalize_groupwritesample_mask0 and loginfra_failure_sample_rate. Withforest_training, every segment row of a flagged rollout gets the 0, and the rate counts rollouts, not rows.includepath now also coversexclude_infra.A search of
nemo_rl/found no other reader that treatsloss_multiplier == 0as "prompt too long".Worked example (GRPO estimator, leave-one-out baseline,
normalize_rewards: true)The group has five rollouts:
agent_timeout.includeexclude_infraexcludeinclude, the valid rollouts' advantages depend on E, which measured nothing.exclude_infradrops E and keeps D, because a timeout is a real failure of the policy.What changes when a recipe sets
exclude_infraThe default does not change, so existing recipes behave as before. A recipe that opts in changes training wherever its groups contain masked rollouts without a policy kind.
Known cases:
failure_kind: judge_failed; their reward is a 0.0 placeholder.exclude. These rows never trained under any value. Gym#4882 tracks giving them kinds.critpt_custom_grader:response_incompleteandpartner_if:evidence_too_large. A recipe can list them ingrpo.policy_failure_kinds, together with the defaults.agent_timeout: listed as a policy kind, although wall time also includes slow model servers or VMs.Usage
Before your PR is "Ready for review"
Pre checks:
docs/guides/grpo.md, plusexamples/configs/grpo_math_1B.yaml.Additional Information
New tests
test_trainers_infra_masked_reward_never_moves_valid_siblings:grpo_train,async_grpo_trainandgrpo_train_syncunder all three values.include.test_advantage_stage_infra_masked_reward_never_moves_valid_siblings: the same check on the SingleController advantage stage, with the GRPO, GDPO and Reinforce++ estimators (9 cases).grpo.policy_failure_kindson: legacygrpo_train, the async collector, the TQ sync actor, SingleController setup,RolloutManager, both places inside the rollout code, and the prompt-group seal.infra_failure_flags;infra_failure_sample_rateon the SingleController native and token-capture paths;include);exclude_inframatchingincludeon every row that is not infra.exclude_infratrainer cases and the ingestion test fail.Training check (a small real run, with injected masks)
Two GRPO jobs ran this PR at
dfe2133bd. Qwen3-0.6B trained on GSM8K through NeMo Gym'smath_with_judgeserver, with the legacy sync trainer: 16 prompts × 8 samples, 5 steps, one GB200 node per job, the same seed and data. The jobs differ only ingrpo.masked_reward_policy:includeorexclude_infra(defaultpolicy_failure_kinds).At RL's current Gym pin, no environment puts a policy kind on a masked row. So a local test patch in
math_with_judge(not part of this PR) masked about a third of the rows with a reward of 0. A hash of the prompt and the sample index picked the rows, so both jobs masked the same rows. The kinds took the normal Gym → RL path.exclude_infrainfra_failure_sample_ratetoken_mult_prob_error(1.011 to 1.014) were about the same in both jobs.Local results (CPU workstation, on
95719f8e8, at23cf9bf1b; the headdfe2133bdchanges only comments)tests/unit/single_controllerandtests/unit/test_config_v2.py, from the repo root: 2,358 passed.--nemo-gym-only): 125 passed and 3 errors. The 3 errors come from tests that start a real vLLM GPU worker, which fails on this workstation ("NVIDIA driver too old"). They are environment errors, not caused by this change.🤖 Generated with Claude Code