Skip to content

feat(grpo): add exclude_infra masked-reward policy for infra-masked rollouts - #4968

Open
yfw wants to merge 5 commits into
mainfrom
yifuw/masked-failure-kind-advantages
Open

yfw wants to merge 5 commits into
mainfrom
yifuw/masked-failure-kind-advantages

Conversation

@yfw

@yfw yfw commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Adds a third grpo.masked_reward_policy value, exclude_infra. With it, a rollout that the environment masked for an infrastructure reason, or with no failure_kind at all, no longer moves the baseline and advantages of the valid rollouts in its group. The default stays include, so existing recipes do not change.

This is the follow-up promised in #4160.

Issues

Related to #4061 and follows up on #4160. On the Gym side, NVIDIA-NeMo/Gym#4882 is adding failure_kind to masked results that lack one.

Problem

#4160 added grpo.masked_reward_policy: include | exclude, with include as the default. Under include, a masked rollout is still part of its group's baseline and std. It never trains, but its reward still moves the advantages of its valid siblings, including when that reward measured nothing.

Example: OSWorld masks evaluator errors and gives them no failure_kind. Take a group with rewards [1, 1, 0], where the 0 is an evaluator error. Under include, the two valid rollouts get raw advantages [0.5, 0.5] instead of [0, 0].

exclude avoids this, but it also drops masks that the policy itself caused (timeouts, step limits, context overflow) and trainer-filtered rows. Those rewards are real measurements of the policy.

Change

  • New opt-in value exclude_infra (the default stays include). A masked rollout still counts in its group's baseline and std only if its top-level failure_kind is in the new grpo.policy_failure_kinds. The default list is agent_timeout, agent_command_not_allowed, agent_limit, agent_invalid_output and model_context_limit.
  • Every other masked rollout is treated as infra and leaves the statistics. That includes rollouts with no kind, kinds Gym adds later, and namespaced <server>:<kind> names. Only that rollout leaves; the rest of its group still trains.
  • Matching is by exact string, namespaced names included. A recipe that sets the list replaces it, because OmegaConf.merge replaces lists. So a recipe that adds a name must also list the defaults.
  • Loss masking does not change. Under every value, a masked rollout's loss weight stays 0 through mask_sample.
  • include and exclude behave exactly as before.
  • Opt-out still works. With env.should_mask_flagged_samples: false, nothing is masked, so nothing is treated as infra.
  • Validation rollouts are not affected.
Rollout include (default) exclude_infra exclude
Not masked counts counts counts
Trainer-filtered (overlong filtering, seq-logprob error) counts counts —
Env-masked, kind in policy_failure_kinds counts counts —
Env-masked, any other kind or no kind counts — —
Dataset weight 0 (over-length prompt) — — —

How it works

No new data-plane or trainer column is needed.

  • Ingestion: where RL reads the Gym result, an infra-masked rollout gets dataset weight 0 (loss_multiplier), the same as an unusable over-length prompt. The include population from feat(grpo): configurable masked-reward participation across the four trainers #4160 (_include_valid_mask) then leaves it out on all four trainers: legacy sync, async, TQ and SingleController.
  • grpo.py:
    • _votes_by_dataset_weight() is true for include and exclude_infra. The estimator, dynamic-sampling admission and the logprob mask use it.
    • exclude_infra_policy_kinds() returns the policy list only under exclude_infra.
  • Legacy, async and TQ paths: rollouts.infra_failure_flags() marks the rows. The group builder sets their loss_multiplier to 0 and logs infra_failure_sample_rate.
  • SingleController, native path: record_to_train_batch sets sample_mask to 0. TQReplayBuffer carries the policy list and logs infra_failure_sample_rate.
  • SingleController, token-capture path:
    • The rollout manager seals each attempt's raw failure_kind into the recovery ledger.
    • The ledger schema goes to v6 (main has v5) and still loads v2 to v5 ledgers, with no kind.
    • At finalization, ReassemblyRequest.infra_failure makes finalize_group write sample_mask 0 and log infra_failure_sample_rate. With forest_training, every segment row of a flagged rollout gets the 0, and the rate counts rollouts, not rows.
  • SingleController advantage stage: the include path now also covers exclude_infra.

A search of nemo_rl/ found no other reader that treats loss_multiplier == 0 as "prompt too long".

Worked example (GRPO estimator, leave-one-out baseline, normalize_rewards: true)

The group has five rollouts:

  • A = 1, B = 1, C = 0 are valid.
  • D = 0 is masked with agent_timeout.
  • E is masked with no kind (an evaluator error), with reward 0 or 1.
Mode A, B if E=0 A, B if E=1 C if E=0 C if E=1
include +1.500 +0.866 −0.866 −1.500
exclude_infra +1.155 +1.155 −1.155 −1.155
exclude +0.707 +0.707 −1.000 −1.000
  • Under include, the valid rollouts' advantages depend on E, which measured nothing.
  • exclude_infra drops E and keeps D, because a timeout is a real failure of the policy.

What changes when a recipe sets exclude_infra

The default does not change, so existing recipes behave as before. A recipe that opts in changes training wherever its groups contain masked rollouts without a policy kind.

Known cases:

  • Judge failures: environments that mask judge failures without a kind no longer count those rows. The same goes for GenRM cohort failures, which carry failure_kind: judge_failed; their reward is a 0.0 placeholder.
  • SWE agents: swe_agents, anyswe and anyterminal mask limits and timeouts without a kind today, so those rows are treated as infra. That matches exclude. These rows never trained under any value. Gym#4882 tracks giving them kinds.
  • Namespaced kinds: some are arguably policy-caused but default to infra, for example critpt_custom_grader:response_incomplete and partner_if:evidence_too_large. A recipe can list them in grpo.policy_failure_kinds, together with the defaults.
  • agent_timeout: listed as a policy kind, although wall time also includes slow model servers or VMs.

Usage

grpo:
  masked_reward_policy: exclude_infra   # opt-in; the default is include
  policy_failure_kinds:                 # read only by exclude_infra; exact string match;
                                        # a recipe list replaces the defaults, so keep them
    - agent_timeout
    - agent_command_not_allowed
    - agent_limit
    - agent_invalid_output
    - model_context_limit

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Unit tests yes (below); functional/GPU tests not run.
  • Did you add or update any necessary documentation? See the "Masked rewards in advantage statistics" section in docs/guides/grpo.md, plus examples/configs/grpo_math_1B.yaml.

Additional Information

New tests

  • test_trainers_infra_masked_reward_never_moves_valid_siblings:
    • A batch built by the real Gym ingestion code runs through grpo_train, async_grpo_train and grpo_train_sync under all three values.
    • Changing the masked evaluator-error reward from 0 to 1 moves the valid rows only under include.
  • test_advantage_stage_infra_masked_reward_never_moves_valid_siblings: the same check on the SingleController advantage stage, with the GRPO, GDPO and Reinforce++ estimators (9 cases).
  • Forwarding tests check that each trainer passes grpo.policy_failure_kinds on: legacy grpo_train, the async collector, the TQ sync actor, SingleController setup, RolloutManager, both places inside the rollout code, and the prompt-group seal.
  • Focused tests cover:
    • infra_failure_flags;
    • the builder setting dataset weights to 0, including with masking off;
    • the payload path, the finalizer (rows flagged and not flagged) and the receipt path;
    • infra_failure_sample_rate on the SingleController native and token-capture paths;
    • a ledger round trip and the v5 load;
    • Forest segment rows taking their rollout's flag;
    • the config default (include);
    • exclude_infra matching include on every row that is not infra.
  • Mutation checks:
    • With the ingestion zeroing disabled, the three exclude_infra trainer cases and the ingestion test fail.
    • Removing any of the 8 forwarding arguments, the prompt-group seal's kind, the finalizer's row flag or either SingleController metric makes its new test fail.

Training check (a small real run, with injected masks)

Two GRPO jobs ran this PR at dfe2133bd. Qwen3-0.6B trained on GSM8K through NeMo Gym's math_with_judge server, with the legacy sync trainer: 16 prompts × 8 samples, 5 steps, one GB200 node per job, the same seed and data. The jobs differ only in grpo.masked_reward_policy: include or exclude_infra (default policy_failure_kinds).

At RL's current Gym pin, no environment puts a policy kind on a masked row. So a local test patch in math_with_judge (not part of this PR) masked about a third of the rows with a reward of 0. A hash of the prompt and the sample index picked the rows, so both jobs masked the same rows. The kinds took the normal Gym → RL path.

Step Masked Policy kind Infra kind No kind Left out of the statistics under exclude_infra infra_failure_sample_rate
1 43 24 13 6 19 of 128 0.148
2 44 18 20 6 26 0.203
3 42 17 20 5 25 0.195
4 44 19 19 6 25 0.195
5 40 16 18 6 24 0.188
  • In each job, the logged advantages match a separate recomputation from the dumped rewards under that job's own rule, within 3e-7. They do not match the other rules (differences up to 1.27). In each job, 41 to 72 of about 86 unmasked rows per step get a different advantage under the other rule.
  • No masked row carried any loss under either policy.
  • Step time and token_mult_prob_error (1.011 to 1.014) were about the same in both jobs.
  • Not shown: training quality, convergence or held-out accuracy; real environments that emit these kinds; the async, TransferQueue and SingleController paths (unit tests only). The two jobs sampled differently, so their reward curves are not a controlled comparison.

Local results (CPU workstation, on 95719f8e8, at 23cf9bf1b; the head dfe2133bd changes only comments)

  • Unit tests, every touched test file plus tests/unit/single_controller and tests/unit/test_config_v2.py, from the repo root: 2,358 passed.
  • Gym-only lane (--nemo-gym-only): 125 passed and 3 errors. The 3 errors come from tests that start a real vLLM GPU worker, which fails on this workstation ("NVIDIA driver too old"). They are environment errors, not caused by this change.
  • The repo pre-commit hooks (ruff, import order, ruff format, pyrefly) pass on the changed files.
  • Not run locally: GPU functional tests.

🤖 Generated with Claude Code

… default

Add a third grpo.masked_reward_policy value, exclude_infra, and make it the
default. A rollout that the environment masked still votes in its group's
baseline and std only when its top-level failure_kind is listed in the new
grpo.policy_failure_kinds (default: agent_timeout, agent_command_not_allowed,
agent_limit, agent_invalid_output, model_context_limit). Any other masked
rollout, including one with no failure_kind, gets dataset weight 0 where RL
reads the Gym result, so the include population from #4160 leaves it out on
the legacy sync, async, TQ and SingleController trainers. include and
exclude behave as before.

The SingleController token-capture path carries the raw failure_kind
through the recovery ledger (schema v4) and ReassemblyRequest.infra_failure.
infra_failure_sample_rate reports the share of rollouts left out.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 28, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Oct 28, 2026
exclude_infra stays available as an opt-in value; the default stays include,
as in #4160. Update the config default, the example and reference configs,
the GRPO guide and the default test.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@yfw yfw changed the title feat(grpo): keep infra-masked rollouts out of advantage statistics by default feat(grpo): add exclude_infra masked-reward policy for infra-masked rollouts Oct 28, 2026

@yfw yfw left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self review of this draft, left as a comment. There are four inline comments:

  1. (C1) No test fails if a trainer stops passing grpo.policy_failure_kinds to its rollout code. Four trainer-level tests and small edits to existing rollout tests close the gap (test_grpo.py#L9586).
  2. (C2) The finalizer test has no row with flag False. One more parametrize case closes the gap (test_rollout_reassembler.py#L295).
  3. (C3) SingleController drops infrastructure-failed rows from the statistics but logs no infra_failure_sample_rate (payload.py#L184).
  4. (C4) The YAML comment says that names "may be added", but a recipe list replaces the five defaults (grpo_math_1B.yaml#L16-L17).

These parts are correct at head:

  • The worked-example table in the PR description matches a recomputation with the real calculate_baseline_and_std_per_prompt.
  • The include and exclude values of grpo.masked_reward_policy behave exactly as before.
  • The opt-out env.should_mask_flagged_samples: false holds on every path.
  • All tests in the 9 changed test files pass on CPU, except 3 existing tests that need a GPU, which this review did not run. The 22 unchanged test files that use a changed API also pass. Every new test runs in a CI lane.

Generated by Claude Code

Comment thread tests/unit/algorithms/test_grpo.py Resolved
Comment thread nemo_rl/experience/payload.py Resolved
Comment thread examples/configs/grpo_math_1B.yaml Resolved
…ind forwarding

- SingleController now logs infra_failure_sample_rate under exclude_infra:
  TQReplayBuffer.commit adds it to the rollout metrics on the native path, and
  finalize_group adds it to the finalizer metrics on the token-capture path.
  The guide now says that every trainer reports it.
- New tests check that each trainer passes grpo.policy_failure_kinds on:
  legacy grpo_train, the async collector, the TQ sync actor, SingleController
  setup (TQReplayBuffer and RolloutManager), the RolloutManager store, both
  places inside the rollout code, and the prompt-group seal.
- The finalizer test gets a row whose infra_failure flag is False.
- Comments, the guide and the example configurations now say that a recipe
  that sets grpo.policy_failure_kinds replaces the list (OmegaConf.merge
  replaces lists), so it must list the defaults with any new name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@yfw

yfw commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 23cf9bf

@yfw
yfw marked this pull request as ready for review October 29, 2026 19:05
@yfw yfw added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Oct 29, 2026
@yfw

yfw commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 23cf9bf

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of all 27 changed files and their affected trainer/data-plane paths found no demonstrated correctness regression. The current source addresses the earlier forwarding, finalizer False-row, SingleController metric, and list-replacement feedback.

Two nonblocking suggestions remain:

  1. (C5) Provide a controlled training comparison for the opt-in policy.
  2. (C6) Shorten repeated in-function policy explanations while retaining their invariants.

Tests and linters were not executed. Upstream Gym source was unavailable in the mount. Two child reports were unavailable despite terminal completion status; their assigned paths were inspected directly before consolidation.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread nemo_rl/algorithms/grpo.py Resolved
The full rule lives in docs/guides/grpo.md and in the
rollouts.infra_failure_flags docstring. The call sites now keep a short
pointer and their local invariant. Comments only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@yfw

yfw commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test dfe2133

yfw added a commit to NVIDIA-NeMo/Gym that referenced this pull request Oct 29, 2026
…comparison

A failed judge comparison is still dropped. A rollout whose comparisons
all failed is now masked alone through the judge failsafe (reward 0.0,
mask_sample, failure_kind judge_failed), and its siblings keep their
scores. Before, it failed the whole group. The masked rollout is left out
of the group's length and style adjustments. When every rollout is left
uncovered, the whole group fails as a judge failure, as before.

- The server now waits for every comparison to finish or fail, within
  cohort_evaluation_timeout_s. A non-judge error still fails the group at
  once and cancels the comparisons still running.
- The fixed_baseline path follows the same rule. /compare has no
  per-response mask, so it still returns 503 when a response keeps no
  comparison.
- Checkpoints save and restore a member masked alone (masked_alone).
- New metric: genrm_masked_rollout_rate_per_group.

The masked rollout's reward is a placeholder, so a trainer must keep it
out of the group statistics: in NeMo RL, grpo.masked_reward_policy
exclude_infra (NVIDIA-NeMo/RL#4968) or exclude.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
…kind-advantages

Main added rollout recovery schema v4 (token-capture receipts, #4124) and
v5 (logical_selection, #4200), so failure_kind moves to schema v6. The
ledger still loads v2 to v5, with failure_kind=None.

Conflicts kept both sides: failure_kind next to logical_selection in the
ledger, SiblingSealResult and both seal calls; infra_failure next to the
Forest fields in ReassemblyRequest; the infra metric next to the logical
owner counts; the infra sample_mask next to the sampler top-k copy.

Main moved row building into _publish_rows, so finalize_group now passes
infra_failure in. With forest_training, each segment row takes its
rollout's flag, and infra_failure_sample_rate counts rollouts.

Tests: the v3 load test is now a v5 load test; the finalizer test sets
capture_key, as main's receipts need; a new Forest test checks that the
segment rows of a flagged rollout get sample_mask 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@yfw

yfw commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test ab0a647

@terrykong terrykong added CI:L1 Run doctests, unit tests, and functional tests CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) CI:L1 Run doctests, unit tests, and functional tests labels Nov 5, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants