Repository navigation
fix(grpo): group advantages by rollout identity - #3882
Conversation
aroshanghias-nvd
left a comment
There was a problem hiding this comment.
Self-review of my own PR, run with this repo's /review-pr-team skill (rl-expert, bug-finder, test-agent, design-reviewer, comment-reviewer, devil-advocate). Posting the surviving findings for transparency. The adversarial pass cut or downgraded 12 of 15 clusters and disputed 1 outright, including two of my own — what's left is what survived it.
2 action items, 3 follow-ups.
AI-1 — ruff format is not clean on 4 of the 8 changed files
Every offending hunk is a line this PR adds:
| File | Lines |
|---|---|
nemo_rl/algorithms/utils.py |
126-128 |
nemo_rl/algorithms/grpo_sync.py |
237-239 |
tests/unit/algorithms/test_grpo.py |
2234-2236, 2279-2281 |
tests/unit/algorithms/test_utils.py |
46-48, 79-81, 92-94 |
Reproduced three times independently, twice at the v0.9.9 pinned in .pre-commit-config.yaml. ruff check passes; only format fails. Action: uv run ruff format .. Inline suggestions cover the first hunk in each file.
This is invisible right now because Request NVSkills CI is skipped, not passing — the copy-pr-bot gate means the lint job has never run on this branch.
AI-2 — this PR orphans a full-batch prompt flatten in grpo_train
input_ids was the GRPO group key. Both consumers are now reward_group_ids, so grpo.py:2995-3000 builds an unused [B, S_prompt] tensor every generation batch. Details and the required deletion set inline — note it is not a single-block delete.
Follow-up 1 — attach CI:L2
labels is currently empty. The cicd skill maps "changes that could affect convergence" to CI:L2, which this is by definition — it changes the GRPO baseline denominator.
Follow-up 2 — the "usually not visible" framing in the description is too narrow
The description argues collisions mainly matter for multimodal and stateful environments. That understates it: any text-only dataset containing duplicate prompt strings previously pooled those rows into one baseline group and will now get separate per-instance baselines. Advantages change on those runs too, so this is not a text-recipe no-op.
Worth one cheap number in the description instead of a full A/B: for one step on one text and one VLM recipe, report len(torch.unique(prompt_ids_for_adv, dim=0)) against num_prompts_per_step. Equal for the first, strictly less for the second, is the whole empirical case for the PR.
Follow-up 3 — prompt_ids_for_adv is now a write-only column
Removing the last readers (_advantage_input_fields, and the overwrite at grpo_sync.py:702) leaves payload.py:91, sync_rollout_actor.py:363 and data_plane/schema.py:54 still producing and shipping the column, and AdvantageConfig.prompt_ids_field with zero readers repo-wide. Retaining the write is defensible for a correctness-only PR that promises no schema change — but a follow-up to drop the column, or a comment saying why it stays, would save the next reader from re-deriving that it is dead. Not asking for it here.
Context — no action.
The core fix was checked rather than assumed. Invariants confirmed independently by more than one agent:
- Row order is preserved on all three legacy rollout branches, which positional grouping now depends on and token grouping did not: the sync path re-sorts by original index (
rollouts.py:836-860), the async path relies onasyncio.gatherordering (:1573), and the NeMo-Gym path returns in input-row order (:2587). repeated_batchisbatch.repeat_interleave(G), so it is group-contiguous and divisible at everybuild_rollout_group_idscall site.rollout_group_idssurvivesselect_indices/from_batches/slice, andnext_rollout_group_idresets in lockstep withbatch_cache, so cached generation batches cannot collide.- Multimodal dedup does not corrupt
repeated_batch.size—PackedTensor.__len__returns logical rows. [B,1]is correct at every consumer, including thecalculate_advantages_on_gpubranch where group IDs, rewards and mask all move to the same device together.- SC cannot see a partial group: dynamic sampling and overlong filtering are rejected outright at
single_controller_utils/config.py:739. - Sample-ID producers all mint
f"{uuid4()}_g{i}", and uuid4 contains no underscore, sorsplit("_g", 1)is unambiguous.
test_apply_dynamic_sampling_rebuilds_group_ids_across_generation_batches is the one test here that genuinely fails on the old behaviour.
Two design notes worth recording: carrying identity as a batch key in legacy grpo.py rides select_indices/concat for free — the TQ path has to rebuild IDs at grpo_sync.py:237 precisely because it lacks that — and keeping both new helpers as pure functions is what lets the bug be pinned in six lines with no fixtures.
Merge is currently blocked by the copy-pr-bot runner gate, which needs a vetter and is not self-serviceable. All checks that did run (DCO, copyright, secrets-detector, submodule fast-forward, semantic title) pass.
Findings dropped after adversarial review: a sample-ID-grammar refactor (its motivating scenario turned out to be invented — the redispatch path reuses the same group_id), two suggested tests (one cosmetic, one whose pytest.raises(match=...) would have failed on arrival — [0, 1] is a regex character class), a docstring-completeness ask (D417 is in this repo's ruff ignore list), and an exception-type nit (the friendly error already fires, via the unpack ValueError).
Keep independent prompt draws and rewritten rollout histories in their original GRPO groups. Carry stable keys through legacy dynamic sampling and async replay, and reuse explicit TQ group tags in sync advantage consumers. Adapt the identity regression tests from PR #4495 while keeping its separate valid-mask dependency out of scope. Co-authored-by: Lin Jia <linj@nvidia.com> Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
fc1a3c1 to
5c029e9
Compare
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
|
/ok to test e5e26cd |
NVIDIA-NeMo#3882 switched calculate_baseline_and_std_per_prompt to group by prompt_group_ids and removed input_ids from _grpo_train_impl; NVIDIA-NeMo#4553, merged right after, added the env-reward baseline call still passing input_ids. On main that is an undefined name (ruff F821, failing Lint check) and a NameError whenever grpo.length_penalty sets env_reward. Use repeated_batch[PROMPT_GROUP_IDS_KEY] there, and prompt_group_ids (instead of prompt token ids) for the same baseline in grpo_sync, so the env-reward baseline groups rows exactly like the main baseline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com> (cherry picked from commit 5328e0c)
…lit baselines GRPO's baselines key on the explicit prompt-group ids since #3882 and #4620, so a per-rollout string in the harness prompt no longer splits a group into singleton baselines. The comment and the warning now say what the condition does mean: the rollouts of a group do not share a prompt prefix, so the engine's prefix cache and anything keyed on first-prompt tokens see one group per rollout. The test asserts the new wording. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: alexandery <alexandery@nvidia.com>
…lit baselines GRPO's baselines key on the explicit prompt-group ids since #3882 and #4620, so a per-rollout string in the harness prompt no longer splits a group into singleton baselines. The comment and the warning now say what the condition does mean: the rollouts of a group do not share a prompt prefix, so the engine's prefix cache and anything keyed on first-prompt tokens see one group per rollout. The test asserts the new wording. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: alexandery <alexandery@nvidia.com>
Two defects in how grpo.masked_reward_policy separates the advantage statistics from the training loss, found in the review of NVIDIA-NeMo#4160: 1. Fractional loss weights reached the statistics (BF-4160-1). Under `exclude` the estimator and the pre-logprob sites received the loss weights themselves as `valid_mask`, but calculate_baseline_and_std_per_prompt and the GDPO/Reinforce++ normalizations count rows: two valid rows with rewards [0, 1] and weights [.5, .5] produced baselines [0, 1] and zero advantages. Participation is now binary everywhere (`_binary_participation`): a nonzero weight votes with full weight and stays a training weight only. One mask feeds every statistics site of each trainer: the legacy loops' dynamic-sampling admission, baseline/std, env_baseline and compute_advantage; the TransferQueue driver's baseline/std, env_baseline (previously still all-ones) and admission through one `statistics_valid_mask`; and the SingleController stage, which binarizes the raw sample mask (include) or the final sample mask (exclude), so token-capture placeholders never vote under either policy. 2. Statistics rows lost their logprobs under top-k/top-p filtering (root-4160-kl). The workers zero every logprob outside token_mask * sample_mask, for the policy and the reference pass alike, and the drivers handed them the loss weights. A loss-masked row that votes under `include` therefore reached Reinforce++'s KL-in-reward with a zero KL term and shifted the batch normalization of the trainable rows. The policy and reference logprob passes now receive the statistics participation mask (`_logprob_sample_mask`): the sync loop's logprob_data, an async-loop view of train_data, and the TransferQueue actor's bulk `sample_mask`, which only the worker logprob passes read (the driver writes the final training mask back after advantages). The training sample_mask is unchanged; under `exclude` both masks select the same rows. The `exclude` policy also folds truncated rows in under overlong_filtering at the pre-logprob sites, so one population serves admission, baseline and estimator; `include` keeps them voting, as docs/guides/grpo.md describes. Tests: TestStatisticsParticipation (binary helper, both policies, fractional baseline [.5, .5]); fractional rows in the legacy/TQ admission test, the fractional-weight step with independent expected advantages; the logprob passes receive the statistics mask; an include + KL-in-reward + top-k step whose fake policy applies mask_filtered_logprobs_outside_tokens and whose unmasked-row advantages match an independent reference; the estimator-side worker-filter regression in test_advantage_validity.py; SC binary participation with a placeholder row. The NVIDIA-NeMo#4490 driver tests are parametrized over the policy, and the TQ dynamic-sampling test rows now carry the GROUP_ID_TAG that main's NVIDIA-NeMo#3882 requires. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
…pisode ids Move the Gym pin from 120d516f, a merge of the pre-merge checkpoint stack that is on no Gym branch head and does not descend from main's pin, to a6bf27ff on NVIDIA-NeMo/Gym amahishi/rl-gym-turn-recovery-pin: - Gym main at 9ef5ed0f, which carries the merged checkpoint stack (#3882-#3889) and descends from main's pin da9f6714; - NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions; - NVIDIA-NeMo/Gym#4690: name the episodes legacy /run sessions own in the agent commit reply; - the checkpoint test agent the turn-recovery functional test runs, with its participant installed once per app now that Gym installs it twice. Relock with uv 0.11.18: Gym moves ray and jinja2 out of its core dependencies and adds requests; every resolved package keeps its version. Gym's coordination now sends each control body pre-encoded as data rather than json, so the adapter test's in-process client forwards both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
…L needs Move the Gym pin from 120d516f, a merge of the pre-merge checkpoint stack that is on no Gym branch head and does not descend from main's pin, to bf631d0a on NVIDIA-NeMo/Gym amahishi/rl-gym-turn-recovery-pin: - Gym main at 9ef5ed0f, which carries the merged checkpoint stack (#3882-#3889) and descends from main's pin da9f6714; - NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions; - NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in the agent commit reply (run_episode_ids); - the checkpoint test agent the turn-recovery functional test runs, with its participant installed once per app now that Gym installs it twice. - NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources server setup, and checkpointing for Blackjack, indirect prompt injection and the proof refinement agent. Relock with uv 0.11.18: Gym moves ray and jinja2 out of its core dependencies and adds requests; every resolved package keeps its version. Gym's coordination now sends each control body pre-encoded as data rather than json, so the adapter test's in-process client forwards both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
…L needs Move the Gym pin from main's 6f295d7d to bf631d0a on NVIDIA-NeMo/Gym amahishi/rl-gym-turn-recovery-pin: - Gym main at 9ef5ed0f, which carries the merged checkpoint stack (#3882-#3889) and descends from main's pin 6f295d7d; - NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions; - NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in the agent commit reply (run_episode_ids); - the checkpoint test agent the turn-recovery functional test runs, with its participant installed once per app now that Gym installs it twice. - NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources server setup, and checkpointing for Blackjack, indirect prompt injection and the proof refinement agent. Gym adds requests to its core dependencies, so the lock records it for nemo-gym; requests is already locked, and every resolved package keeps its version. Gym's coordination now sends each control body pre-encoded as data rather than json, so the adapter test's in-process client forwards both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Two defects in how grpo.masked_reward_policy separates the advantage statistics from the training loss, found in the review of NVIDIA-NeMo#4160: 1. Fractional loss weights reached the statistics (BF-4160-1). Under `exclude` the estimator and the pre-logprob sites received the loss weights themselves as `valid_mask`, but calculate_baseline_and_std_per_prompt and the GDPO/Reinforce++ normalizations count rows: two valid rows with rewards [0, 1] and weights [.5, .5] produced baselines [0, 1] and zero advantages. Participation is now binary everywhere (`_binary_participation`): a nonzero weight votes with full weight and stays a training weight only. One mask feeds every statistics site of each trainer: the legacy loops' dynamic-sampling admission, baseline/std, env_baseline and compute_advantage; the TransferQueue driver's baseline/std, env_baseline (previously still all-ones) and admission through one `statistics_valid_mask`; and the SingleController stage, which binarizes the raw sample mask (include) or the final sample mask (exclude), so token-capture placeholders never vote under either policy. 2. Statistics rows lost their logprobs under top-k/top-p filtering (root-4160-kl). The workers zero every logprob outside token_mask * sample_mask, for the policy and the reference pass alike, and the drivers handed them the loss weights. A loss-masked row that votes under `include` therefore reached Reinforce++'s KL-in-reward with a zero KL term and shifted the batch normalization of the trainable rows. The policy and reference logprob passes now receive the statistics participation mask (`_logprob_sample_mask`): the sync loop's logprob_data, an async-loop view of train_data, and the TransferQueue actor's bulk `sample_mask`, which only the worker logprob passes read (the driver writes the final training mask back after advantages). The training sample_mask is unchanged; under `exclude` both masks select the same rows. The `exclude` policy also folds truncated rows in under overlong_filtering at the pre-logprob sites, so one population serves admission, baseline and estimator; `include` keeps them voting, as docs/guides/grpo.md describes. Tests: TestStatisticsParticipation (binary helper, both policies, fractional baseline [.5, .5]); fractional rows in the legacy/TQ admission test, the fractional-weight step with independent expected advantages; the logprob passes receive the statistics mask; an include + KL-in-reward + top-k step whose fake policy applies mask_filtered_logprobs_outside_tokens and whose unmasked-row advantages match an independent reference; the estimator-side worker-filter regression in test_advantage_validity.py; SC binary participation with a placeholder row. The NVIDIA-NeMo#4490 driver tests are parametrized over the policy, and the TQ dynamic-sampling test rows now carry the GROUP_ID_TAG that main's NVIDIA-NeMo#3882 requires. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
…L needs Move the Gym pin from main's 6f295d7d to bf631d0a on NVIDIA-NeMo/Gym amahishi/rl-gym-turn-recovery-pin: - Gym main at 9ef5ed0f, which carries the merged checkpoint stack (#3882-#3889) and descends from main's pin 6f295d7d; - NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions; - NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in the agent commit reply (run_episode_ids); - the checkpoint test agent the turn-recovery functional test runs, with its participant installed once per app now that Gym installs it twice. - NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources server setup, and checkpointing for Blackjack, indirect prompt injection and the proof refinement agent. Gym adds requests to its core dependencies, so the lock records it for nemo-gym; requests is already locked, and every resolved package keeps its version. Gym's coordination now sends each control body pre-encoded as data rather than json, so the adapter test's in-process client forwards both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
…L needs Move the Gym pin from main's 6f295d7d to bf631d0a on NVIDIA-NeMo/Gym amahishi/rl-gym-turn-recovery-pin: - Gym main at 9ef5ed0f, which carries the merged checkpoint stack (#3882-#3889) and descends from main's pin 6f295d7d; - NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions; - NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in the agent commit reply (run_episode_ids); - the checkpoint test agent the turn-recovery functional test runs, with its participant installed once per app now that Gym installs it twice. - NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources server setup, and checkpointing for Blackjack, indirect prompt injection and the proof refinement agent. Gym adds requests to its core dependencies, so the lock records it for nemo-gym; requests is already locked, and every resolved package keeps its version. Gym's coordination now sends each control body pre-encoded as data rather than json, so the adapter test's in-process client forwards both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
…L needs Move the Gym pin from main's 6f295d7d to bf631d0a on NVIDIA-NeMo/Gym amahishi/rl-gym-turn-recovery-pin: - Gym main at 9ef5ed0f, which carries the merged checkpoint stack (#3882-#3889) and descends from main's pin 6f295d7d; - NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions; - NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in the agent commit reply (run_episode_ids); - the checkpoint test agent the turn-recovery functional test runs, with its participant installed once per app now that Gym installs it twice. - NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources server setup, and checkpointing for Blackjack, indirect prompt injection and the proof refinement agent. Gym adds requests to its core dependencies, so the lock records it for nemo-gym; requests is already locked, and every resolved package keeps its version. Gym's coordination now sends each control body pre-encoded as data rather than json, so the adapter test's in-process client forwards both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
…L needs Move the Gym pin from main's 6f295d7d to bf631d0a on NVIDIA-NeMo/Gym amahishi/rl-gym-turn-recovery-pin: - Gym main at 9ef5ed0f, which carries the merged checkpoint stack (#3882-#3889) and descends from main's pin 6f295d7d; - NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions; - NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in the agent commit reply (run_episode_ids); - the checkpoint test agent the turn-recovery functional test runs, with its participant installed once per app now that Gym installs it twice. - NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources server setup, and checkpointing for Blackjack, indirect prompt injection and the proof refinement agent. Gym adds requests to its core dependencies, so the lock records it for nemo-gym; requests is already locked, and every resolved package keeps its version. Gym's coordination now sends each control body pre-encoded as data rather than json, so the adapter test's in-process client forwards both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
…L needs Move the Gym pin from main's 6f295d7d to bf631d0a on NVIDIA-NeMo/Gym amahishi/rl-gym-turn-recovery-pin: - Gym main at 9ef5ed0f, which carries the merged checkpoint stack (#3882-#3889) and descends from main's pin 6f295d7d; - NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions; - NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in the agent commit reply (run_episode_ids); - the checkpoint test agent the turn-recovery functional test runs, with its participant installed once per app now that Gym installs it twice. - NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources server setup, and checkpointing for Blackjack, indirect prompt injection and the proof refinement agent. Gym adds requests to its core dependencies, so the lock records it for nemo-gym; requests is already locked, and every resolved package keeps its version. Gym's coordination now sends each control body pre-encoded as data rather than json, so the adapter test's in-process client forwards both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
…L needs Move the Gym pin from main's 6f295d7d to bf631d0a on NVIDIA-NeMo/Gym amahishi/rl-gym-turn-recovery-pin: - Gym main at 9ef5ed0f, which carries the merged checkpoint stack (#3882-#3889) and descends from main's pin 6f295d7d; - NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions; - NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in the agent commit reply (run_episode_ids); - the checkpoint test agent the turn-recovery functional test runs, with its participant installed once per app now that Gym installs it twice. - NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources server setup, and checkpointing for Blackjack, indirect prompt injection and the proof refinement agent. Gym adds requests to its core dependencies, so the lock records it for nemo-gym; requests is already locked, and every resolved package keeps its version. Gym's coordination now sends each control body pre-encoded as data rather than json, so the adapter test's in-process client forwards both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
What does this PR do?
Fix GRPO credit assignment by grouping completions by rollout sampling identity instead of tokenized prompt equality.
Problem
GRPO baselines are defined over completions sampled together for one prompt instance. Using initial prompt tokens as the group key can merge independent draws with identical tokens, or split one draw if an environment rewrites its returned history.
For example, two independent 16-generation groups may have identical initial token rows but different images or environment state:
The correct within-group RLOO advantages are zero for both groups. Token-based grouping instead creates one 32-sample group: a zero-reward sample receives a baseline of
16/31, and a one-reward sample receives a baseline of15/31, producing false negative and positive advantages.Fix
group_idtag, preserving multimodal tags and existing sample-ID format.[B, K]equality-key contract; production GRPO passes[B, 1]group IDs. Preserve the utility's argument name and three-value return contract.No user-facing configuration changes, sample-ID format changes, checkpoint-schema changes, or dependency/container changes. Valid-mask semantics are unchanged; #4490 remains separate.
Issues
None.
Usage
No configuration changes are required.
Before your PR is ready for review
git diff --checkpassed.Additional information
Local tests used an isolated macOS CPU environment (Python 3.13.14, PyTorch 2.13.0) with
--noconftest; this is not the repository's full GPU test harness. Coverage includes the complete existingtest_grpo.pymodule, targeted group-identity/data-plane/actor/Single Controller/validity tests, and focused baseline/std and trivial-distribution utility tests.CI:Lfastremains suitable: this change does not touch container inputs,pyproject.toml, oruv.lock.This fixes incorrect GRPO credit assignment. It does not claim to resolve entropy collapse or policy degeneration generally. Full repository CI remains the merge gate.