Skip to content

feat(sc): restart dead generation shards and re-admit them at the next refit - #3592

Merged
terrykong merged 1 commit into
mainfrom
feat/sc-resiliency-04-restart-readmission
Sep 14, 2026
Merged

terrykong merged 1 commit into
mainfrom
feat/sc-resiliency-04-restart-readmission

Conversation

@terrykong

Copy link
Copy Markdown
Collaborator

Note

Mirror of #3473, originally opened by @asolergi-nv from the asolergi-nv/RL fork.
The branch was re-pushed to NVIDIA-NeMo/RL unchanged (same head SHA) so the four parts
can be linked as a proper PR stack — gh stack cannot operate on fork branches.
Please review and comment here. The original #3473 is left open for history.

Part 4/4 of #3454

What this fixes

After #3472 a lost shard is dropped and the run continues on the rest — but the fleet only ever shrinks. A job that sheds a few transient failures over a long run ends permanently smaller, and nothing ever brings capacity back.

Two things were missing, and the second was a genuine defect:

1. Re-admission was structurally impossible. reconcile_communicator keyed off "is anything absent", so the moment a restarted shard stopped being absent the answer was False, no rebuild happened, and the shard stayed out of both the communicator and the refit dispatch for the rest of the run. Restarting an engine would have achieved nothing.

2. Nothing promoted a restarted shard back into service. GenerationFleetMonitor.report_refit — documented as "a completed refit is the only way back into the serving set" — was never called anywhere in nemo_rl. A restarted shard therefore reached STALE, rejoined the communicator, received weights, and then sat idle for the rest of the run. The fleet recovered on paper and not in throughput. Every supervisor unit test passed because each one called report_refit by hand; the tests now drive it through the controller, which is what makes the wiring observable.

What it does

Membership is compared against what was built, not against "is anything absent", so a shard leaving and a shard returning are one operation in opposite directions. desired_membership() expresses it as "what it should be" rather than "what changed", which is what makes both directions fall out of one comparison. An unrecorded membership means the full fleet, not "unknown".

EngineSupervisor restarts dead shards, one background task each. Handover is entirely through fleet-health states:

DEAD ─(restart starts)→ RESTARTING ─(engine up)→ STALE ─(refit)→ HEALTHY

RESTARTING is absent from collectives, so a rebuild landing mid-restart correctly excludes it. STALE is present but not serving — that is what lets the refit write current weights before the shard takes traffic again.

The restart is not awaited. Reloading a model takes minutes, and the loop it runs from
also drives the rollout pump, the watchdog and the refit; blocking it would stall training
that the surviving shards can still do.

RayWorkerGroup.recreate_worker replays the recorded creation call for one worker rather than re-deriving it — re-deriving means reproducing the placement group, bundle, venv and rank bookkeeping, a second implementation that would drift. The old actor is killed first: usually already dead, but an unresponsive one still holding the GPU would make the replacement fail on memory rather than on anything informative.

Testing

  • 18 supervisor tests: state handover, the attempt cap and retirement, failed restarts not propagating, and that tick() returns while the restart is still blocked.
  • 8 recreate_worker tests pinning the replay — same bundle and placement group, old actor killed first, name uniquified per incarnation, and the recorded spec left unmutated so names do not compound as -r1-r2-r3.
  • RESTART_DEAD_SHARDS=true mode added to the recovery functional test and registered in the SingleController lane. It asserts a restart was attempted, an engine came back, two rebuilds happened (one dropping the shard, one taking it back), and a stale -> healthy transition occurred. Completion plus one rebuild would only prove the fleet shrank and carried on.

Issues

List issues that this PR closes (syntax):

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

@terrykong
terrykong requested review from a team as code owners August 12, 2026 00:58
@copy-pr-bot

copy-pr-bot Bot commented Aug 12, 2026

Copy link
Copy Markdown

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@terrykong

Copy link
Copy Markdown
Collaborator Author

PR stack for #3454 (SingleController resiliency)

Mirror of #3473. Part 4/4 of the stack (#3593) — see #3589 for the full stack map. Based on #3591, so the diff here is only this layer (+1153 −65, 21 files) rather than the cumulative +10139 across 56 files the original showed.

Part Mirror PR Original
1/4 contain rollout failures #3589 #3470
2/4 fleet health + Gym router #3590 #3471
3/4 keep training on shard death #3591 #3472
4/4 restart + re-admission #3592 #3473

Branch pushed unchanged from @asolergi-nv's fork (bce9eb03ed9a502282724d2069f7a55758310212). Review here; #3473 stays open for history.

@github-actions

Copy link
Copy Markdown

❌ Submodule Fast-Forward Check Failed

Check based on commit: 911e5d0 (PR #3592 from feat/sc-resiliency-04-restart-readmission)

❌ Submodules that need attention:

Gym: ❌ PR branch is BEHIND main branch
TARGET (main branch): https://github.com/NVIDIA-NeMo/Gym/commits/c3bac96314a59f28b896f597eb9845d175bb0252/
CURRENT (PR #3592 from feat/sc-resiliency-04-restart-readmission): https://github.com/NVIDIA-NeMo/Gym/commits/473f446f71ec7c1243eb1517fe2440a2b37fe68b/

Megatron-Bridge: ❌ PR branch is BEHIND main branch
TARGET (main branch): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/8c46dc4259080c510b7455f43e836fdff222c5d3/
CURRENT (PR #3592 from feat/sc-resiliency-04-restart-readmission): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/573e088c9c6740082c39744e03dc5b009e730ed4/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

1 similar comment
@github-actions

Copy link
Copy Markdown

❌ Submodule Fast-Forward Check Failed

Check based on commit: 911e5d0 (PR #3592 from feat/sc-resiliency-04-restart-readmission)

❌ Submodules that need attention:

Gym: ❌ PR branch is BEHIND main branch
TARGET (main branch): https://github.com/NVIDIA-NeMo/Gym/commits/c3bac96314a59f28b896f597eb9845d175bb0252/
CURRENT (PR #3592 from feat/sc-resiliency-04-restart-readmission): https://github.com/NVIDIA-NeMo/Gym/commits/473f446f71ec7c1243eb1517fe2440a2b37fe68b/

Megatron-Bridge: ❌ PR branch is BEHIND main branch
TARGET (main branch): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/8c46dc4259080c510b7455f43e836fdff222c5d3/
CURRENT (PR #3592 from feat/sc-resiliency-04-restart-readmission): https://github.com/NVIDIA-NeMo/Megatron-Bridge/commits/573e088c9c6740082c39744e03dc5b009e730ed4/

Please ensure all submodule commits are fast-forwards of the main branch before merging.

@asolergi-nv
asolergi-nv force-pushed the feat/sc-resiliency-04-restart-readmission branch from f65ee2d to bac923e Compare August 24, 2026 10:28
asolergi-nv added a commit that referenced this pull request Aug 25, 2026
**Dead code that asserted the opposite of the live check (mine).** The FREEZE_VICTIM block
at :528 terminates on every path -- four `exit 1` and an `exit 0` -- so the second
FREEZE_VICTIM block below it and the frozen branch of the final PASS were unreachable. They
also claimed "the run survived" and "finished all N steps", which is precisely what that
variant does not do; leftover from before the variant was consolidated. Twenty lines
deleted and the PASS collapsed to its one live branch. Worth doing here rather than later:
#3592 edits a string inside the region that was dead.

**getattr probing where the attribute is declared.** `pp_comm_group` is a class attribute
on AbstractPolicyWorker, which MegatronPolicyWorkerImpl inherits, so
`getattr(self, "pp_comm_group", None)` could never take its default. Verified at runtime,
not just read. Same smell already removed for `_refit_membership` in 8779394.

**A forward reference that outlived the thing it pointed at.** "Recovery modes arrive with
the communicator rebuild" sat on `on_dead_shard` in both the config and the exemplar. The
rebuild has since landed, and recovery is not selected through that field at all -- the
reconcile rebuilds over the survivors whenever a shard becomes absent, whatever it says.

**An undocumented parameter among documented siblings.** `refit_timeout_s` was missing from
the factory's Args block. The reviewer also expected it missing on both synchronizers; it
is already documented there, so only the factory needed it.

Found by review on #3591.

Signed-off-by: asolergibert <asolergibert@nvidia.com>
Base automatically changed from feat/sc-resiliency-03-elastic-recovery to main August 28, 2026 19:32
@terrykong
terrykong requested review from a team as code owners August 28, 2026 19:32
@asolergi-nv asolergi-nv added the CI:L1 Run doctests, unit tests, and functional tests label Aug 29, 2026
@copy-pr-bot

copy-pr-bot Bot commented Aug 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@asolergi-nv

Copy link
Copy Markdown
Contributor

/ok to test aa100a2

@asolergi-nv

Copy link
Copy Markdown
Contributor

/ok to test 6e54437

@asolergi-nv

Copy link
Copy Markdown
Contributor

/ok to test 5b4a732

@terrykong terrykong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rebase first — this PR currently conflicts with main (mergeable: CONFLICTING). Everything below is pinned to head 5b4a732, so some line numbers will drift once it lands.

28 comments. The design is careful and the state machine is well argued with itself — most of what follows is about the edges where the argument does not quite reach.

Correctness — these change training silently

  1. A shard promoted to HEALTHY by a refit that never sent it weights. A restart finishing during a refit turns the shard STALE after membership was settled, and _promote_refit_shards asks "is this shard STALE?" rather than "was it in the group I just sent weights to?". Until this PR nothing could turn a shard STALE mid-refit, so those two questions had the same answer. The shard then serves rollouts from the checkpoint it loaded off disk, and the per-shard weight_version metric cannot catch it — the promotion is what writes that number.
  2. A restarted engine inherits the dead engine's record. mark_loaded clears the two probe counters but leaves state_before_partial and consecutive_reported_failures set, so a fresh process comes back SUSPECT at 2 of 3 on the counter that condemns it. On the condemn-silent-participant path this is guaranteed rather than unlucky, because the test that selects the shard is a test that the field is set.
  3. RAY_process_group_cleanup_enabled never reaches the raylet. It is read by the raylet, not the driver, so setting it from Python is a no-op on Slurm and k8s — and that setting is what the EngineCore reaping depends on.

Recovery robustness

  • The restart runs on asyncio.to_thread with no timeout. A hang leaves the shard in RESTARTING for the rest of the run, never retried and never charged an attempt, and can hold the process open at exit.
  • drain() reads like a shutdown safety net. Nothing outside the tests calls it, and its timeout cannot bound the thread it waits on.
  • mark_restart_failed takes no reason, unlike both its siblings, so after a shard is retired the record still shows what originally killed it rather than why five reloads failed.
  • Nothing logs the target GPU before a restart is issued, so an orphaned EngineCore still holding the memory is only visible from inside the attempt that is already failing — and not at all when the replacement cannot start.

Tests and CI

  • No CI lane collects test_engine_supervisor.py or test_engine_reaping_env.py, so 25 of this PR's new tests never run, and run_unit.sh treats pytest's "no tests collected" exit 5 as success. Worth fixing before the rest: it is the finding that hides the others.
  • Three smaller ones: an assertion that also passes when zero restarts happened, no case pinning what happens when a replacement engine dies, and a parametrize that checks a parameter against itself.

Config and interface

  • restart_dead_shards appears in neither SingleController exemplar YAML, so a user reading either file cannot tell restart exists.
  • One ++ Hydra override in the recovery test, needed only because of the above. ++ turns off the unknown-key check, so a renamed field would silently fall back to defaults and the test would pass while proving nothing.
  • restart_shard is the one backend capability found by duck typing rather than declared on GenerationInterface, and the rebuild-skip rule is now byte-identical in both synchronizers.

Comment cleanup (7 comments)

Several new comments read as notes to a reviewer rather than to a future reader — internal run ids, which merge deleted a method, where a line used to sit. The behaviour they describe is worth keeping; the run that found it is not. Each comment carries a suggested trim that keeps the rule and drops the history.

Beyond this PR: the same pattern appears about 88 more times in files already on main from the earlier PRs in this stack. Not this PR's problem — worth one sweep later.

Walkthroughs

Seven pages, one question each, for the mechanisms that are hard to see from the diff alone. Two are linked from the comments they explain.

Comment thread nemo_rl/models/generation/__init__.py Resolved
Comment thread nemo_rl/__init__.py Resolved
asolergi-nv added a commit that referenced this pull request Sep 8, 2026
…ailure history on reload

Three review findings from PR #3592, all rooted in the same thing: restart is
the first event that can change fleet membership *while* a refit is in flight.

1. `_promote_refit_shards` promoted every STALE shard. Asking "is this shard
   STALE?" was sufficient until restart existed, because nothing could turn a
   shard STALE mid-refit. Now `mark_loaded` does, at whatever moment the reload
   lands, and STALE is deliberately not in `_ABSENT_STATES` -- so a shard that
   was absent when the communicator was built could still be promoted, returning
   it to service holding the checkpoint it read off disk instead of the trainer's
   weights. Capture the eligible set once, when membership settles, and filter
   promotion by it. Captured twice, in fact: `_recover_from_failed_refit`
   condemns the silent participant and rebuilds over the survivors, so the retry
   does not run against the first attempt's membership.

2. `mark_loaded` cleared the two probe counters but left
   `consecutive_reported_failures` and `state_before_partial` set. Both describe
   the process that died; neither is evidence about the fresh one taking its
   place, and `report_refit` reads both -- so a clean replacement came back
   SUSPECT, one failure away from being condemned again.

3. `test_engine_supervisor.py` and `test_engine_reaping_env.py` sat under
   `tests/unit/models/generation/`, which the `Other` lane ignores and the
   `Models_*` lanes also ignore, so 25 tests ran in no lane at all. Moved to
   `tests/unit/single_controller/`, where the code they cover lives. Both files
   walked `parents[4]` to the repo root, which the move breaks; now `parents[3]`.

Signed-off-by: asolergibert <asolergibert@nvidia.com>
asolergi-nv added a commit that referenced this pull request Sep 8, 2026
…apping

Structural findings from the PR #3592 review, plus the test and comment fixes
that go with them. No behaviour change intended; the suites that cover these
paths pass unchanged.

`should_rebuild()` moves into `membership.py`. Both hardened synchronizers had
the same ten lines, byte for byte, comments included -- which is the bug class
this stack keeps hitting, and the reason `desired_membership` was extracted in
the first place. Not a move toward unifying the synchronizers: `_desired_membership`'s
worker-group fallback stays where it is, because the two transports genuinely
differ there (Dynamo degrades, nccl_reshard requires a worker group and should
fail loudly without one).

`shard_liveness_ref()` moves the shard-to-worker mapping onto the backend.
`restart_shard` already asked the worker group which worker leads a shard;
`_probe_generation_fleet` reached through the generation object and did the
arithmetic itself, so the control loop knew the layout and assumed every backend
has a `worker_group` -- the same assumption that broke the Dynamo L1 lane with
`'DynamoGeneration' object has no attribute 'worker_group'`. Declaring it on
GenerationInterface removes the assumption instead of guarding it, and the test
fakes collapse from four nested SimpleNamespaces to one stub.

The supervisor's counters move from `supervisor/` to `gen_fleet/` and `metrics()`
becomes `as_metrics()`, matching GenerationFleetHealth -- the two dicts are merged
two lines apart and `gen_fleet/restart_attempts` already carried restart state.
Worth doing now rather than later: the rename is free forever, the key prefix is
only free until something points at it.

Tests:
- the attempt-cap assertion was `len(...) <= max_attempts`, which also holds at
  zero -- the failure it exists to catch. Asserts the actual list now.
- a replacement engine dying again was untested on both sides. STALE is
  deliberately not absent, so a second death has to be able to condemn it; the
  gap is new because this PR is what creates STALE-from-restart.
- dropped a `parametrize` over one value that fed the test its own expected
  value, and the stray `[retired]` in the id that came with it.

Comments: dropped the run ids, commit SHAs, stack positions and exact NVML
readings that were notes to a reviewer rather than to a future reader. The rules
they were attached to are kept; git keeps the history. The `seed=None` error
string stays -- it is what someone hitting it will search for. Scoped to this
PR's own comments; the same pattern on main is a separate sweep.

Signed-off-by: asolergibert <asolergibert@nvidia.com>
@asolergi-nv
asolergi-nv requested a review from a team as a code owner September 9, 2026 10:02
@asolergi-nv

Copy link
Copy Markdown
Contributor

/ok to test 17e9f2c

@asolergi-nv

Copy link
Copy Markdown
Contributor

/ok to test e0c9e73

…t refit

Squashed from the PR's original history. The branch carried 251 commits, 112 of
them merges of feat/sc-resiliency-01/02/03 while that stack was in flight; those
three landed on main as squashes, so their individual commits exist nowhere in
main and a plain rebase would have replayed ~139 commits whose content main
already has. The net diff was always this: 39 files, +3103/-226.

A dead generation shard used to be gone for the rest of the run. This restarts
it and hands it back through the fleet-health states:

    DEAD --(restart starts)--> RESTARTING --(engine up)--> STALE --(refit)--> HEALTHY

RESTARTING is absent from collectives, so a rebuild mid-restart leaves the shard
out. STALE is present but not serving, which is what lets the next refit write
current weights into it before it takes traffic again.

What this contains beyond the core feature, all of it review-driven:

- The refit promotes only shards it actually reached. A restart finishing
  mid-transfer turns a shard STALE, which is deliberately not absent, so "is it
  STALE" stopped meaning "was it in the refit". The eligible set is captured when
  membership settles, and captured again after a failed-refit recovery, because
  that recovery rebuilds over the survivors.
- The weight version is stamped where the weights land, not by the promotion.
  Nothing turns a shard STALE on a refit that succeeds, so a healthy fleet
  reported version 0 forever and the alert built on it was firing on every shard.
- A restarted engine no longer inherits the dead one's failure record.
- The restart runs on a dedicated daemon thread bounded by restart_timeout_s,
  with restart_backoff_s between attempts, a reason carried into the health
  record, and a GPU reading taken before each attempt from the shard's own node.
- RAY_process_group_cleanup_enabled is exported in ray.sub, where the raylet
  actually reads it; the driver-side setting was inert on every multi-node run.
- Router failures are counted once per drain window rather than once per request,
  so one brief outage with several requests in flight can no longer condemn a
  shard in a single tick.
- restart_shard, shard_liveness_ref and log_shard_gpu_state are declared on
  GenerationInterface, and the Megatron backend implements the liveness probe.

Signed-off-by: asolergibert <asolergibert@nvidia.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@asolergi-nv
asolergi-nv force-pushed the feat/sc-resiliency-04-restart-readmission branch from 564953a to a815a8c Compare September 14, 2026 13:14
@asolergi-nv

Copy link
Copy Markdown
Contributor

/ok to test a815a8c

@terrykong
terrykong merged commit 53bce05 into main Sep 14, 2026
269 of 273 checks passed
@terrykong
terrykong deleted the feat/sc-resiliency-04-restart-readmission branch September 14, 2026 23:22
# minutes and nothing blocks it, so mark_loaded can turn a shard STALE mid-transfer
# -- and STALE is not absent, so asking again at promotion time would include a
# shard the communicator was deliberately built without.
participants = self._refit_participants()

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 action item.

TL;DR — participants is read after _reconcile_refit_membership has already taken its own read and awaited, so a shard that finishes restarting in that gap lands in participants while the communicator was built without it — the exact outcome this comment says it prevents.

The comment here, and _refit_participants's docstring, both state the contract:

Read once, here, because the answer changes underneath a refit. […] asking again at promotion time would include a shard the communicator was deliberately built without.

Right idea, one step short: reading here is still a second read, taken after the reconcile's own read and after an await.

  1. L3443 captures absent = self._gen_fleet.absent_shards(), and L3447-L3449 builds the communicator from that set — over an await asyncio.to_thread(...), which yields the loop.
  2. The engine supervisor runs on that loop. engine_supervisor.py L174 calls mark_loaded, flipping a restarted shard RESTARTING → STALE. STALE is not absent.
  3. Back here, L4705 re-reads the monitor — L3596 asks absent_shards() again, "as of right now".
  4. That shard is now STALE, so not absent, so in participants — but it is not in the communicator and receives nothing.
  5. _record_refit_landed L3644-L3650 promotes shards that are STALE and in participants, so it goes back into the serving set holding the checkpoint it read off disk.

There is a companion page walking this through with a diagram: how a shard gets marked healthy by a refit that never sent it any weights. It now covers the guard this PR added and the window it leaves.

Reachability is narrow: it needs async_rl.generation_fleet_health.enabled: true (off by default) and a restart completing inside that window. But that is the configuration the restart feature exists to serve, so it is the one that matters.

AI-1

Make _reconcile_refit_membership return the absent set it reconciled against (alongside rebuilt), and have _refit_participants take that set as an argument instead of calling absent_shards() itself. Then the participant set and the communicator both come from the single read at L3443, with no window between them. No suggestion block because the fix changes the function's return type plus two call sites, which is not one contiguous region.

Both call sites need it: L4700-L4705 and the recovery re-snapshot at L4739, where the rebuild happens inside _recover_from_failed_refit and participants is read after it returns — same shape, and by the comment's own note it is the likelier of the two windows.

potatowarriors added a commit to potatowarriors/RL that referenced this pull request Oct 6, 2026
- 3rdparty/Gym-workspace/Gym: bb9b26de (0.5.0rc0) -> v0.6.0 (3045a793), matching the
  standalone Gym the alpha eval/configs were validated on (2026-09-15 decision: v0.6.0 fixed).
- pyproject: override openai==2.44.0. Gym v0.6.0 pins openai==2.44.0 while sglang 0.5.12.post1
  pins 2.6.1, and workspace members are resolved in every split; SGLang is unused on the alpha
  track. Upstream r0.8.0 moved sglang to a newer fork instead -- revisit on the v0.8 rebase.
- uv.lock regenerated with uv 0.11.28 (+252/-105): openai 2.6.1 -> 2.44.0, Gym 0.6.0 deps added;
  torch/vllm 0.25.1/transformers/ray/TE/flashinfer-python unchanged.
- nemo_rl/environments/nemo_gym.py: backport of upstream r0.8.0 (NVIDIA-NeMo#3592) -- build the per-agent
  counter after run_examples() (Gym >= 0.6 resolves task_source -> agent_ref there) and fail
  readably on unresolved rows. Datasets collated with Gym 0.6 carry only task_source and would
  otherwise KeyError on row["agent_ref"].

Verified (CPU only, CUDA_VISIBLE_DEVICES=""): nemo_gym 0.6.0 imports + NeMo-RL call sites;
NeMo-RL gym unit tests 69 passed (default) + 1 passed (nemo_gym-marked, 2 GPU-only not run);
alpha gym configs 4/4 valid; swe_pivot 184 / single_step_tool_use 30 / terminus_judge 84 passed;
vllm & mcore extras sync from cache, AlphaBridge/MegatronPolicyWorker/alpha vLLM plugin import.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:L1 Run doctests, unit tests, and functional tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants