Skip to content

fix(grpo): group advantages by rollout identity - #3882

Merged
yfw merged 2 commits into
mainfrom
aroshanghias/fix-grpo-rollout-group-identity
Oct 9, 2026
Merged

yfw merged 2 commits into
mainfrom
aroshanghias/fix-grpo-rollout-group-identity

Conversation

@aroshanghias-nvd

@aroshanghias-nvd aroshanghias-nvd commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Fix GRPO credit assignment by grouping completions by rollout sampling identity instead of tokenized prompt equality.

Problem

GRPO baselines are defined over completions sampled together for one prompt instance. Using initial prompt tokens as the group key can merge independent draws with identical tokens, or split one draw if an environment rewrites its returned history.

For example, two independent 16-generation groups may have identical initial token rows but different images or environment state:

Group A: image/state A, rewards [0, 0, ..., 0]
Group B: image/state B, rewards [1, 1, ..., 1]

The correct within-group RLOO advantages are zero for both groups. Token-based grouping instead creates one 32-sample group: a zero-reward sample receives a baseline of 16/31, and a one-reward sample receives a baseline of 15/31, producing false negative and positive advantages.

Fix

  • Assign explicit per-draw group IDs before legacy synchronous rollouts. Carry them through dynamic-sampling filtering, caching, concatenation, and slicing, with cumulative IDs across generation batches.
  • Group legacy asynchronous training rows by their sampled per-prompt replay-buffer entry, independently of returned prompt tokens.
  • Stamp the existing rollout-group UUID in TransferQueue metadata's group_id tag, preserving multimodal tags and existing sample-ID format.
  • Use group tags for TransferQueue baseline/std, dynamic-sampling decisions, and advantage estimation. Factorize the final selected metadata after caches are combined, so independent UUIDs cannot collide through temporary dense IDs.
  • Share the current Single Controller group-tag helpers with the synchronous TransferQueue path, without reintroducing sample-ID parsing or changing Single Controller estimator behavior.
  • Remove obsolete prompt-token flattening and actor-to-driver transfer. Retain the current Single Controller prompt-token field/configuration used by training dumps.
  • Document the generic [B, K] equality-key contract; production GRPO passes [B, 1] group IDs. Preserve the utility's argument name and three-value return contract.

No user-facing configuration changes, sample-ID format changes, checkpoint-schema changes, or dependency/container changes. Valid-mask semantics are unchanged; #4490 remains separate.

Issues

None.

Usage

No configuration changes are required.

Before your PR is ready for review

  • Followed contributor guidelines.
  • Added regression coverage for independent equal-token prompts, rewritten histories, sync/async estimator plumbing, and dynamic-sampling cache/filter/concat/slice behavior.
  • Verified TransferQueue group tags coexist with multimodal metadata and obsolete prompt carry is absent.
  • Local CPU checks: 302 passed, one CUDA-only test skipped.
  • Ruff 0.9.9 lint/import/format checks, Python compilation, and git diff --check passed.
  • Linux/GPU CI and project-wide pyrefly checks remain pending.

Additional information

Local tests used an isolated macOS CPU environment (Python 3.13.14, PyTorch 2.13.0) with --noconftest; this is not the repository's full GPU test harness. Coverage includes the complete existing test_grpo.py module, targeted group-identity/data-plane/actor/Single Controller/validity tests, and focused baseline/std and trivial-distribution utility tests.

CI:Lfast remains suitable: this change does not touch container inputs, pyproject.toml, or uv.lock.

This fixes incorrect GRPO credit assignment. It does not claim to resolve entropy collapse or policy degeneration generally. Full repository CI remains the merge gate.

@aroshanghias-nvd
aroshanghias-nvd requested review from a team as code owners August 27, 2026 19:52
@copy-pr-bot

copy-pr-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@aroshanghias-nvd
aroshanghias-nvd requested a review from yfw August 27, 2026 19:53

@aroshanghias-nvd aroshanghias-nvd left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review of my own PR, run with this repo's /review-pr-team skill (rl-expert, bug-finder, test-agent, design-reviewer, comment-reviewer, devil-advocate). Posting the surviving findings for transparency. The adversarial pass cut or downgraded 12 of 15 clusters and disputed 1 outright, including two of my own — what's left is what survived it.

2 action items, 3 follow-ups.

AI-1 — ruff format is not clean on 4 of the 8 changed files

Every offending hunk is a line this PR adds:

File Lines
nemo_rl/algorithms/utils.py 126-128
nemo_rl/algorithms/grpo_sync.py 237-239
tests/unit/algorithms/test_grpo.py 2234-2236, 2279-2281
tests/unit/algorithms/test_utils.py 46-48, 79-81, 92-94

Reproduced three times independently, twice at the v0.9.9 pinned in .pre-commit-config.yaml. ruff check passes; only format fails. Action: uv run ruff format .. Inline suggestions cover the first hunk in each file.

This is invisible right now because Request NVSkills CI is skipped, not passing — the copy-pr-bot gate means the lint job has never run on this branch.

AI-2 — this PR orphans a full-batch prompt flatten in grpo_train

input_ids was the GRPO group key. Both consumers are now reward_group_ids, so grpo.py:2995-3000 builds an unused [B, S_prompt] tensor every generation batch. Details and the required deletion set inline — note it is not a single-block delete.

Follow-up 1 — attach CI:L2

labels is currently empty. The cicd skill maps "changes that could affect convergence" to CI:L2, which this is by definition — it changes the GRPO baseline denominator.

Follow-up 2 — the "usually not visible" framing in the description is too narrow

The description argues collisions mainly matter for multimodal and stateful environments. That understates it: any text-only dataset containing duplicate prompt strings previously pooled those rows into one baseline group and will now get separate per-instance baselines. Advantages change on those runs too, so this is not a text-recipe no-op.

Worth one cheap number in the description instead of a full A/B: for one step on one text and one VLM recipe, report len(torch.unique(prompt_ids_for_adv, dim=0)) against num_prompts_per_step. Equal for the first, strictly less for the second, is the whole empirical case for the PR.

Follow-up 3 — prompt_ids_for_adv is now a write-only column

Removing the last readers (_advantage_input_fields, and the overwrite at grpo_sync.py:702) leaves payload.py:91, sync_rollout_actor.py:363 and data_plane/schema.py:54 still producing and shipping the column, and AdvantageConfig.prompt_ids_field with zero readers repo-wide. Retaining the write is defensible for a correctness-only PR that promises no schema change — but a follow-up to drop the column, or a comment saying why it stays, would save the next reader from re-deriving that it is dead. Not asking for it here.


Context — no action.

The core fix was checked rather than assumed. Invariants confirmed independently by more than one agent:

  • Row order is preserved on all three legacy rollout branches, which positional grouping now depends on and token grouping did not: the sync path re-sorts by original index (rollouts.py:836-860), the async path relies on asyncio.gather ordering (:1573), and the NeMo-Gym path returns in input-row order (:2587).
  • repeated_batch is batch.repeat_interleave(G), so it is group-contiguous and divisible at every build_rollout_group_ids call site.
  • rollout_group_ids survives select_indices / from_batches / slice, and next_rollout_group_id resets in lockstep with batch_cache, so cached generation batches cannot collide.
  • Multimodal dedup does not corrupt repeated_batch.size — PackedTensor.__len__ returns logical rows.
  • [B,1] is correct at every consumer, including the calculate_advantages_on_gpu branch where group IDs, rewards and mask all move to the same device together.
  • SC cannot see a partial group: dynamic sampling and overlong filtering are rejected outright at single_controller_utils/config.py:739.
  • Sample-ID producers all mint f"{uuid4()}_g{i}", and uuid4 contains no underscore, so rsplit("_g", 1) is unambiguous.

test_apply_dynamic_sampling_rebuilds_group_ids_across_generation_batches is the one test here that genuinely fails on the old behaviour.

Two design notes worth recording: carrying identity as a batch key in legacy grpo.py rides select_indices/concat for free — the TQ path has to rebuild IDs at grpo_sync.py:237 precisely because it lacks that — and keeping both new helpers as pure functions is what lets the bug be pinned in six lines with no fixtures.

Merge is currently blocked by the copy-pr-bot runner gate, which needs a vetter and is not self-serviceable. All checks that did run (DCO, copyright, secrets-detector, submodule fast-forward, semantic title) pass.

Findings dropped after adversarial review: a sample-ID-grammar refactor (its motivating scenario turned out to be invented — the redispatch path reuses the same group_id), two suggested tests (one cosmetic, one whose pytest.raises(match=...) would have failed on arrival — [0, 1] is a regex character class), a docstring-completeness ask (D417 is in this repo's ruff ignore list), and an exception-type nit (the friendly error already fires, via the unpack ValueError).

Comment thread nemo_rl/algorithms/utils.py Resolved
Comment thread nemo_rl/algorithms/grpo_sync.py Resolved
Comment thread tests/unit/algorithms/test_grpo.py Resolved
Comment thread tests/unit/algorithms/test_utils.py Resolved
Comment thread nemo_rl/algorithms/grpo.py Resolved
@aroshanghias-nvd aroshanghias-nvd added the CI:L2 Run doctests, unit tests, functional tests, and convergence tests label Aug 28, 2026
Comment thread nemo_rl/algorithms/single_controller.py Resolved
Comment thread nemo_rl/algorithms/utils.py Resolved
Comment thread nemo_rl/algorithms/single_controller.py Resolved
@yfw yfw added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:L2 Run doctests, unit tests, functional tests, and convergence tests labels Sep 1, 2026
Keep independent prompt draws and rewritten rollout histories in their original GRPO groups. Carry stable keys through legacy dynamic sampling and async replay, and reuse explicit TQ group tags in sync advantage consumers.

Adapt the identity regression tests from PR #4495 while keeping its separate valid-mask dependency out of scope.

Co-authored-by: Lin Jia <linj@nvidia.com>
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
@aroshanghias-nvd
aroshanghias-nvd force-pushed the aroshanghias/fix-grpo-rollout-group-identity branch from fc1a3c1 to 5c029e9 Compare October 8, 2026 17:56
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
@aroshanghias-nvd

Copy link
Copy Markdown
Contributor Author

/ok to test e5e26cd

@yfw
yfw merged commit 724344f into main Oct 9, 2026
95 checks passed
@yfw
yfw deleted the aroshanghias/fix-grpo-rollout-group-identity branch October 9, 2026 01:45
yuekaizhang pushed a commit to yuekaizhang/RL that referenced this pull request Oct 9, 2026
NVIDIA-NeMo#3882 switched calculate_baseline_and_std_per_prompt to group by
prompt_group_ids and removed input_ids from _grpo_train_impl; NVIDIA-NeMo#4553,
merged right after, added the env-reward baseline call still passing
input_ids. On main that is an undefined name (ruff F821, failing Lint
check) and a NameError whenever grpo.length_penalty sets env_reward.

Use repeated_batch[PROMPT_GROUP_IDS_KEY] there, and prompt_group_ids
(instead of prompt token ids) for the same baseline in grpo_sync, so the
env-reward baseline groups rows exactly like the main baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>
(cherry picked from commit 5328e0c)
alexandery-nvidia added a commit that referenced this pull request Oct 10, 2026
…lit baselines

GRPO's baselines key on the explicit prompt-group ids since #3882 and #4620,
so a per-rollout string in the harness prompt no longer splits a group into
singleton baselines. The comment and the warning now say what the condition
does mean: the rollouts of a group do not share a prompt prefix, so the
engine's prefix cache and anything keyed on first-prompt tokens see one
group per rollout. The test asserts the new wording.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: alexandery <alexandery@nvidia.com>
alexandery-nvidia added a commit that referenced this pull request Oct 12, 2026
…lit baselines

GRPO's baselines key on the explicit prompt-group ids since #3882 and #4620,
so a per-rollout string in the harness prompt no longer splits a group into
singleton baselines. The comment and the warning now say what the condition
does mean: the rollouts of a group do not share a prompt prefix, so the
engine's prefix cache and anything keyed on first-prompt tokens see one
group per rollout. The test asserts the new wording.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: alexandery <alexandery@nvidia.com>
linj-glitch added a commit to linj-glitch/RL that referenced this pull request Oct 13, 2026
Two defects in how grpo.masked_reward_policy separates the advantage
statistics from the training loss, found in the review of NVIDIA-NeMo#4160:

1. Fractional loss weights reached the statistics (BF-4160-1). Under
   `exclude` the estimator and the pre-logprob sites received the loss
   weights themselves as `valid_mask`, but calculate_baseline_and_std_per_prompt
   and the GDPO/Reinforce++ normalizations count rows: two valid rows with
   rewards [0, 1] and weights [.5, .5] produced baselines [0, 1] and zero
   advantages. Participation is now binary everywhere (`_binary_participation`):
   a nonzero weight votes with full weight and stays a training weight only.
   One mask feeds every statistics site of each trainer: the legacy loops'
   dynamic-sampling admission, baseline/std, env_baseline and
   compute_advantage; the TransferQueue driver's baseline/std, env_baseline
   (previously still all-ones) and admission through one
   `statistics_valid_mask`; and the SingleController stage, which binarizes
   the raw sample mask (include) or the final sample mask (exclude), so
   token-capture placeholders never vote under either policy.

2. Statistics rows lost their logprobs under top-k/top-p filtering
   (root-4160-kl). The workers zero every logprob outside
   token_mask * sample_mask, for the policy and the reference pass alike, and
   the drivers handed them the loss weights. A loss-masked row that votes
   under `include` therefore reached Reinforce++'s KL-in-reward with a zero
   KL term and shifted the batch normalization of the trainable rows. The
   policy and reference logprob passes now receive the statistics
   participation mask (`_logprob_sample_mask`): the sync loop's logprob_data,
   an async-loop view of train_data, and the TransferQueue actor's bulk
   `sample_mask`, which only the worker logprob passes read (the driver
   writes the final training mask back after advantages). The training
   sample_mask is unchanged; under `exclude` both masks select the same rows.

The `exclude` policy also folds truncated rows in under overlong_filtering at
the pre-logprob sites, so one population serves admission, baseline and
estimator; `include` keeps them voting, as docs/guides/grpo.md describes.

Tests: TestStatisticsParticipation (binary helper, both policies, fractional
baseline [.5, .5]); fractional rows in the legacy/TQ admission test, the
fractional-weight step with independent expected advantages; the logprob
passes receive the statistics mask; an include + KL-in-reward + top-k step
whose fake policy applies mask_filtered_logprobs_outside_tokens and whose
unmasked-row advantages match an independent reference; the estimator-side
worker-filter regression in test_advantage_validity.py; SC binary
participation with a placeholder row. The NVIDIA-NeMo#4490 driver tests are
parametrized over the policy, and the TQ dynamic-sampling test rows now
carry the GROUP_ID_TAG that main's NVIDIA-NeMo#3882 requires.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
macandro96 added a commit that referenced this pull request Oct 16, 2026
…pisode ids

Move the Gym pin from 120d516f, a merge of the pre-merge checkpoint stack
that is on no Gym branch head and does not descend from main's pin, to
a6bf27ff on NVIDIA-NeMo/Gym amahishi/rl-gym-turn-recovery-pin:

- Gym main at 9ef5ed0f, which carries the merged checkpoint stack
  (#3882-#3889) and descends from main's pin da9f6714;
- NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions;
- NVIDIA-NeMo/Gym#4690: name the episodes legacy /run sessions own in
  the agent commit reply;
- the checkpoint test agent the turn-recovery functional test runs, with
  its participant installed once per app now that Gym installs it twice.

Relock with uv 0.11.18: Gym moves ray and jinja2 out of its core
dependencies and adds requests; every resolved package keeps its version.

Gym's coordination now sends each control body pre-encoded as data
rather than json, so the adapter test's in-process client forwards both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
macandro96 added a commit that referenced this pull request Oct 17, 2026
…L needs

Move the Gym pin from 120d516f, a merge of the pre-merge checkpoint stack
that is on no Gym branch head and does not descend from main's pin, to
bf631d0a on NVIDIA-NeMo/Gym amahishi/rl-gym-turn-recovery-pin:

- Gym main at 9ef5ed0f, which carries the merged checkpoint stack
  (#3882-#3889) and descends from main's pin da9f6714;
- NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions;
- NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in
  the agent commit reply (run_episode_ids);
- the checkpoint test agent the turn-recovery functional test runs, with
  its participant installed once per app now that Gym installs it twice.
- NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources
  server setup, and checkpointing for Blackjack, indirect prompt
  injection and the proof refinement agent.

Relock with uv 0.11.18: Gym moves ray and jinja2 out of its core
dependencies and adds requests; every resolved package keeps its version.

Gym's coordination now sends each control body pre-encoded as data
rather than json, so the adapter test's in-process client forwards both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
macandro96 added a commit that referenced this pull request Oct 22, 2026
…L needs

Move the Gym pin from main's 6f295d7d to bf631d0a on NVIDIA-NeMo/Gym
amahishi/rl-gym-turn-recovery-pin:

- Gym main at 9ef5ed0f, which carries the merged checkpoint stack
  (#3882-#3889) and descends from main's pin 6f295d7d;
- NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions;
- NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in
  the agent commit reply (run_episode_ids);
- the checkpoint test agent the turn-recovery functional test runs, with
  its participant installed once per app now that Gym installs it twice.
- NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources
  server setup, and checkpointing for Blackjack, indirect prompt
  injection and the proof refinement agent.

Gym adds requests to its core dependencies, so the lock records it for
nemo-gym; requests is already locked, and every resolved package keeps its
version.

Gym's coordination now sends each control body pre-encoded as data
rather than json, so the adapter test's in-process client forwards both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
linj-glitch added a commit to linj-glitch/RL that referenced this pull request Oct 23, 2026
Two defects in how grpo.masked_reward_policy separates the advantage
statistics from the training loss, found in the review of NVIDIA-NeMo#4160:

1. Fractional loss weights reached the statistics (BF-4160-1). Under
   `exclude` the estimator and the pre-logprob sites received the loss
   weights themselves as `valid_mask`, but calculate_baseline_and_std_per_prompt
   and the GDPO/Reinforce++ normalizations count rows: two valid rows with
   rewards [0, 1] and weights [.5, .5] produced baselines [0, 1] and zero
   advantages. Participation is now binary everywhere (`_binary_participation`):
   a nonzero weight votes with full weight and stays a training weight only.
   One mask feeds every statistics site of each trainer: the legacy loops'
   dynamic-sampling admission, baseline/std, env_baseline and
   compute_advantage; the TransferQueue driver's baseline/std, env_baseline
   (previously still all-ones) and admission through one
   `statistics_valid_mask`; and the SingleController stage, which binarizes
   the raw sample mask (include) or the final sample mask (exclude), so
   token-capture placeholders never vote under either policy.

2. Statistics rows lost their logprobs under top-k/top-p filtering
   (root-4160-kl). The workers zero every logprob outside
   token_mask * sample_mask, for the policy and the reference pass alike, and
   the drivers handed them the loss weights. A loss-masked row that votes
   under `include` therefore reached Reinforce++'s KL-in-reward with a zero
   KL term and shifted the batch normalization of the trainable rows. The
   policy and reference logprob passes now receive the statistics
   participation mask (`_logprob_sample_mask`): the sync loop's logprob_data,
   an async-loop view of train_data, and the TransferQueue actor's bulk
   `sample_mask`, which only the worker logprob passes read (the driver
   writes the final training mask back after advantages). The training
   sample_mask is unchanged; under `exclude` both masks select the same rows.

The `exclude` policy also folds truncated rows in under overlong_filtering at
the pre-logprob sites, so one population serves admission, baseline and
estimator; `include` keeps them voting, as docs/guides/grpo.md describes.

Tests: TestStatisticsParticipation (binary helper, both policies, fractional
baseline [.5, .5]); fractional rows in the legacy/TQ admission test, the
fractional-weight step with independent expected advantages; the logprob
passes receive the statistics mask; an include + KL-in-reward + top-k step
whose fake policy applies mask_filtered_logprobs_outside_tokens and whose
unmasked-row advantages match an independent reference; the estimator-side
worker-filter regression in test_advantage_validity.py; SC binary
participation with a placeholder row. The NVIDIA-NeMo#4490 driver tests are
parametrized over the policy, and the TQ dynamic-sampling test rows now
carry the GROUP_ID_TAG that main's NVIDIA-NeMo#3882 requires.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
macandro96 added a commit that referenced this pull request Oct 26, 2026
…L needs

Move the Gym pin from main's 6f295d7d to bf631d0a on NVIDIA-NeMo/Gym
amahishi/rl-gym-turn-recovery-pin:

- Gym main at 9ef5ed0f, which carries the merged checkpoint stack
  (#3882-#3889) and descends from main's pin 6f295d7d;
- NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions;
- NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in
  the agent commit reply (run_episode_ids);
- the checkpoint test agent the turn-recovery functional test runs, with
  its participant installed once per app now that Gym installs it twice.
- NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources
  server setup, and checkpointing for Blackjack, indirect prompt
  injection and the proof refinement agent.

Gym adds requests to its core dependencies, so the lock records it for
nemo-gym; requests is already locked, and every resolved package keeps its
version.

Gym's coordination now sends each control body pre-encoded as data
rather than json, so the adapter test's in-process client forwards both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
macandro96 added a commit that referenced this pull request Oct 27, 2026
…L needs

Move the Gym pin from main's 6f295d7d to bf631d0a on NVIDIA-NeMo/Gym
amahishi/rl-gym-turn-recovery-pin:

- Gym main at 9ef5ed0f, which carries the merged checkpoint stack
  (#3882-#3889) and descends from main's pin 6f295d7d;
- NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions;
- NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in
  the agent commit reply (run_episode_ids);
- the checkpoint test agent the turn-recovery functional test runs, with
  its participant installed once per app now that Gym installs it twice.
- NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources
  server setup, and checkpointing for Blackjack, indirect prompt
  injection and the proof refinement agent.

Gym adds requests to its core dependencies, so the lock records it for
nemo-gym; requests is already locked, and every resolved package keeps its
version.

Gym's coordination now sends each control body pre-encoded as data
rather than json, so the adapter test's in-process client forwards both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
macandro96 added a commit that referenced this pull request Oct 28, 2026
…L needs

Move the Gym pin from main's 6f295d7d to bf631d0a on NVIDIA-NeMo/Gym
amahishi/rl-gym-turn-recovery-pin:

- Gym main at 9ef5ed0f, which carries the merged checkpoint stack
  (#3882-#3889) and descends from main's pin 6f295d7d;
- NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions;
- NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in
  the agent commit reply (run_episode_ids);
- the checkpoint test agent the turn-recovery functional test runs, with
  its participant installed once per app now that Gym installs it twice.
- NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources
  server setup, and checkpointing for Blackjack, indirect prompt
  injection and the proof refinement agent.

Gym adds requests to its core dependencies, so the lock records it for
nemo-gym; requests is already locked, and every resolved package keeps its
version.

Gym's coordination now sends each control body pre-encoded as data
rather than json, so the adapter test's in-process client forwards both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
macandro96 added a commit that referenced this pull request Oct 31, 2026
…L needs

Move the Gym pin from main's 6f295d7d to bf631d0a on NVIDIA-NeMo/Gym
amahishi/rl-gym-turn-recovery-pin:

- Gym main at 9ef5ed0f, which carries the merged checkpoint stack
  (#3882-#3889) and descends from main's pin 6f295d7d;
- NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions;
- NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in
  the agent commit reply (run_episode_ids);
- the checkpoint test agent the turn-recovery functional test runs, with
  its participant installed once per app now that Gym installs it twice.
- NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources
  server setup, and checkpointing for Blackjack, indirect prompt
  injection and the proof refinement agent.

Gym adds requests to its core dependencies, so the lock records it for
nemo-gym; requests is already locked, and every resolved package keeps its
version.

Gym's coordination now sends each control body pre-encoded as data
rather than json, so the adapter test's in-process client forwards both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
macandro96 added a commit that referenced this pull request Oct 31, 2026
…L needs

Move the Gym pin from main's 6f295d7d to bf631d0a on NVIDIA-NeMo/Gym
amahishi/rl-gym-turn-recovery-pin:

- Gym main at 9ef5ed0f, which carries the merged checkpoint stack
  (#3882-#3889) and descends from main's pin 6f295d7d;
- NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions;
- NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in
  the agent commit reply (run_episode_ids);
- the checkpoint test agent the turn-recovery functional test runs, with
  its participant installed once per app now that Gym installs it twice.
- NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources
  server setup, and checkpointing for Blackjack, indirect prompt
  injection and the proof refinement agent.

Gym adds requests to its core dependencies, so the lock records it for
nemo-gym; requests is already locked, and every resolved package keeps its
version.

Gym's coordination now sends each control body pre-encoded as data
rather than json, so the adapter test's in-process client forwards both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
macandro96 added a commit that referenced this pull request Nov 2, 2026
…L needs

Move the Gym pin from main's 6f295d7d to bf631d0a on NVIDIA-NeMo/Gym
amahishi/rl-gym-turn-recovery-pin:

- Gym main at 9ef5ed0f, which carries the merged checkpoint stack
  (#3882-#3889) and descends from main's pin 6f295d7d;
- NVIDIA-NeMo/Gym#3894: checkpoint Workplace Assistant sessions;
- NVIDIA-NeMo/Gym#4690: name the episodes an agent's own /run drives in
  the agent commit reply (run_episode_ids);
- the checkpoint test agent the turn-recovery functional test runs, with
  its participant installed once per app now that Gym installs it twice.
- NVIDIA-NeMo/Gym#3895-#3898: Gymnasium servers on the shared resources
  server setup, and checkpointing for Blackjack, indirect prompt
  injection and the proof refinement agent.

Gym adds requests to its core dependencies, so the lock records it for
nemo-gym; requests is already locked, and every resolved package keeps its
version.

Gym's coordination now sends each control body pre-encoded as data
rather than json, so the adapter test's in-process client forwards both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants