Skip to content

fix(sc): checkpoint monotonic NeMo Gym task IDs - #4245

Merged
terrykong merged 4 commits into
NVIDIA-NeMo:mainfrom
mehraakash:akamehra/sc-gym-task-index
Oct 30, 2026
Merged

terrykong merged 4 commits into
NVIDIA-NeMo:mainfrom
mehraakash:akamehra/sc-gym-task-index

Conversation

@mehraakash

@mehraakash mehraakash commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Give Single Controller's NeMo Gym rollout attempts a fresh, monotonically increasing _ng_task_index, and persist the next value in trainer checkpoints and rollout-only snapshots.

  • Allocate one numeric task ID per run_rollout attempt; all sibling rows share it.
  • Override missing/reused dataset task IDs without mutating the source sample.
  • Keep the same ID when missing rows are retried inside an attempt; allocate a new ID on a new attempt (including partial-sibling recovery).
  • Save/restore next_nemo_gym_task_index in GRPOSaveState; older checkpoints default to zero.
  • Carry the counter in the same rollout-snapshot cut as the data-plane state; latest/bootstrap recovery restores that value. Older snapshots without it retain the trainer counter.
  • Preserve main's _ng_group_id, _ng_group_attempt, _ng_rollout_index, and token-capture rollout IDs.

This is the narrow task-ID/counter change from e919a519, adapted to current main. No telemetry, masking, advantage-estimation, recipe, or submodule changes.

Validation

Current head: review follow-up (28 October 2026)

Rebased onto main 1849f2256725b783b34c9f91ae9444f93bb26587; current PR head is fcaaffbefadf6f11d23894d015839e5d4a656a60.

  • Addressed the review's attempt-identity comment: explain why Gym needs a fresh task ID per rollout attempt, why sibling/row retries keep that ID, and why it is not the async collector's dataloader position.
  • Preserved both main's reserve_prompt_group and this PR's task-counter get/set methods in the checkpointing fake. Updated the periodic snapshot test to mc.logger.log_dir.
  • Rebase also preserves main's adaptive PPO clip state and bootstrap reward compatibility tests alongside this PR's counter coverage.
  • 1,641 CPU pytest tests passed, covering the full Single Controller suite and experience rollout-manager, generation-failure, rollouts, and redispatch suites.
  • Both existing pure GRPO checkpoint compatibility/round-trip test functions passed separately with their original assertions. Their module-wide Ray actor fixture stalled locally, so these two were invoked directly without that unrelated fixture; they are not included in the 1,641 pytest count.
  • Pinned Ruff lint/import, Ruff formatting, EOF and trailing-whitespace hooks passed on all 15 changed Python files; git diff --check passed. Full local type/config hooks remain unverified.
  • Local runtime: macOS CPU, Python 3.13.13 and PyTorch 2.10; temporary uv overlays used current-main Lens, multi-storage-client and pytest-asyncio pins. This is below the project's Python minimum and is not the locked GPU image. No dependency/submodule pin changes, GPU test, or training/convergence claim.
  • Fresh CI:L0 validation is being requested for the new head; no CI success is claimed yet.

Earlier revisions (historical validation)

  • 1,189 local CPU unit tests passed after review fixes: Single Controller suite, rollout-manager/generation-failure/redispatch suites, and GRPO checkpoint compatibility/round-trip tests.
  • Repository pre-commit formatting/import/whitespace hooks and git diff --check passed on the review fixes. Local type/config hooks remain unverified because the available test environment is below the project's Python minimum; CI lint passed for the initial head, and CI is retriggered for the fix head.
  • Tests used an existing macOS CPU environment (Python 3.13.13, PyTorch 2.10), not the locked GPU training image. No GPU training or convergence test was run.
  • Team self-review completed with the upstream /review-pr-team skill using Codex agents, including an adversarial pass. Addressed the snapshot persistence finding and optional internal-retry identity test; bounded independent re-review found no further issues.
  • CI follow-up: fixed two stale test fixtures in fc778271db0667ae3a73458942c5f1dff5821c6b: initialize the counter in the runtime-agent-ref rollout test, and expose the native counter interface in the Megatron train-pump stub. The expanded local Single Controller + experience suites passed (1,475 passed, 1 skipped). No checks, assertions, or coverage thresholds were relaxed. Full L0 CI has been retriggered for this head; its result remains pending.

Usage and compatibility

No configuration change. Native rollouts are unchanged. The counter is checkpoint-local, not a globally unique ID across independent runs or rollback to an older checkpoint. The snapshot manifest gains a backward-compatible optional counter field; legacy artifacts cannot reconstruct historical attempt counts that were never saved.

Checklist

  • Read contributor guidelines; signed-off commit.
  • Added focused regression tests.
  • Ran local CPU unit tests.
  • GPU functional tests (not run locally).
  • No user-facing documentation change required for this bug fix.

@mehraakash
mehraakash requested review from a team as code owners September 22, 2026 21:35
@mehraakash mehraakash added the CI:L0 Run doctests and unit tests label Sep 22, 2026
@mehraakash
mehraakash requested a review from a team as a code owner September 22, 2026 21:35
@copy-pr-bot

copy-pr-bot Bot commented Sep 22, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@mehraakash

Copy link
Copy Markdown
Contributor Author

/ok to test 0d915e9

@mehraakash mehraakash left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review performed with the upstream /review-pr-team skill using Codex agents: RL/design, bugs, tests/existing comments, and an independent devil's-advocate pass. One P2 correctness finding and one optional P3 test improvement survived the adversarial review. Both will be addressed before human review. The change remains limited to per-attempt NeMo Gym task identity and checkpoint continuity; no telemetry, masking, or recipe changes. Validation so far: 1,177 CPU tests passed; CI lint passed. OCI-HSG GPU validation is being prepared and is not yet claimed. No reward-corruption or convergence claim is made.

Comment thread nemo_rl/algorithms/single_controller.py Resolved
@mehraakash

Copy link
Copy Markdown
Contributor Author

/ok to test d7028b2

@mehraakash

Copy link
Copy Markdown
Contributor Author

/ok to test fc77827

@saumishr
saumishr requested review from ananthsub and yfw October 7, 2026 20:08

@terrykong terrykong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR needs a rebase on main. It conflicts in one file, tests/unit/single_controller/test_checkpointing.py, in two spots:

  1. _FakeRolloutManager: keep both main's new reserve_prompt_group and this PR's get/set_next_nemo_gym_task_index.
  2. test_periodic_snapshot_restores_dispatch_cursor_and_task_counter: keep this PR's block, but write mc.logger.log_dir = ... instead of mc.logger["log_dir"] = .... main turned the logger config into a BaseModel (#4278), so the dict-style line fails after the merge.

The change itself looks right. Giving each attempt its own ID stops Gym keys from repeating when a dataset ID repeats (every epoch), and gives Gym an ID when the dataset row has none. The counter is saved together with the dispatch cursor, in both trainer checkpoints and rollout-only snapshots. It is restored the same way the sampler restores its dispatch index, and old checkpoints and snapshots still load. The tests cover save, restore, bootstrap, old manifests, and the retry-within-an-attempt case. We found no correctness bugs.

One note, no action in this PR: as the description says, the counter is per checkpoint. IDs handed out after the last save are handed out again after a resume, and a fresh run starts at 0. Any Gym agent that caches by (task_index, rollout_index) in a folder that outlives the run (Harbor's harbor_jobs_dir today) could return an old result. No NeMo-RL recipe uses Harbor. If that changes, the fix belongs in Gym: build the job name from _ng_rollout_id or a per-run ID.

Visual explainer: https://terrykong.github.io/gh-pages-poc/terryk/pr-4245-gym-attempt-ids.html

Generated by Claude Code

Comment thread nemo_rl/experience/rollout_manager.py Resolved
mehraakash and others added 4 commits October 28, 2026 22:38
Signed-off-by: Akash Mehra <akamehra@nvidia.com>
Signed-off-by: Akash Mehra <akamehra@nvidia.com>
Signed-off-by: Akash Mehra <akamehra@nvidia.com>
Signed-off-by: Akash Mehra <mehra.akash@gmail.com>
@mehraakash
mehraakash force-pushed the akamehra/sc-gym-task-index branch from fc77827 to fcaaffb Compare October 29, 2026 05:54
@mehraakash

Copy link
Copy Markdown
Contributor Author

/ok to test fcaaffb

@terrykong
terrykong enabled auto-merge (squash) October 30, 2026 08:19
@terrykong
terrykong merged commit 2bb62a8 into NVIDIA-NeMo:main Oct 30, 2026
96 of 98 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:L0 Run doctests and unit tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants