Repository navigation
feat(sc): restart dead generation shards and re-admit them at the next refit - #3592
Conversation
|
Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
PR stack for #3454 (SingleController resiliency)Mirror of #3473. Part 4/4 of the stack (#3593) — see #3589 for the full stack map. Based on #3591, so the diff here is only this layer (+1153 −65, 21 files) rather than the cumulative +10139 across 56 files the original showed.
Branch pushed unchanged from @asolergi-nv's fork ( |
4852047 to
9dbc8cf
Compare
❌ Submodule Fast-Forward Check FailedCheck based on commit: 911e5d0 (PR #3592 from ❌ Submodules that need attention:Gym: ❌ PR branch is BEHIND main branch Megatron-Bridge: ❌ PR branch is BEHIND main branch Please ensure all submodule commits are fast-forwards of the main branch before merging. |
1 similar comment
❌ Submodule Fast-Forward Check FailedCheck based on commit: 911e5d0 (PR #3592 from ❌ Submodules that need attention:Gym: ❌ PR branch is BEHIND main branch Megatron-Bridge: ❌ PR branch is BEHIND main branch Please ensure all submodule commits are fast-forwards of the main branch before merging. |
f65ee2d to
bac923e
Compare
**Dead code that asserted the opposite of the live check (mine).** The FREEZE_VICTIM block at :528 terminates on every path -- four `exit 1` and an `exit 0` -- so the second FREEZE_VICTIM block below it and the frozen branch of the final PASS were unreachable. They also claimed "the run survived" and "finished all N steps", which is precisely what that variant does not do; leftover from before the variant was consolidated. Twenty lines deleted and the PASS collapsed to its one live branch. Worth doing here rather than later: #3592 edits a string inside the region that was dead. **getattr probing where the attribute is declared.** `pp_comm_group` is a class attribute on AbstractPolicyWorker, which MegatronPolicyWorkerImpl inherits, so `getattr(self, "pp_comm_group", None)` could never take its default. Verified at runtime, not just read. Same smell already removed for `_refit_membership` in 8779394. **A forward reference that outlived the thing it pointed at.** "Recovery modes arrive with the communicator rebuild" sat on `on_dead_shard` in both the config and the exemplar. The rebuild has since landed, and recovery is not selected through that field at all -- the reconcile rebuilds over the survivors whenever a shard becomes absent, whatever it says. **An undocumented parameter among documented siblings.** `refit_timeout_s` was missing from the factory's Args block. The reviewer also expected it missing on both synchronizers; it is already documented there, so only the factory needed it. Found by review on #3591. Signed-off-by: asolergibert <asolergibert@nvidia.com>
|
/ok to test aa100a2 |
|
/ok to test 6e54437 |
|
/ok to test 5b4a732 |
terrykong
left a comment
There was a problem hiding this comment.
Please rebase first — this PR currently conflicts with main (mergeable: CONFLICTING). Everything below is pinned to head 5b4a732, so some line numbers will drift once it lands.
28 comments. The design is careful and the state machine is well argued with itself — most of what follows is about the edges where the argument does not quite reach.
Correctness — these change training silently
- A shard promoted to HEALTHY by a refit that never sent it weights. A restart finishing during a refit turns the shard
STALEafter membership was settled, and_promote_refit_shardsasks "is this shard STALE?" rather than "was it in the group I just sent weights to?". Until this PR nothing could turn a shardSTALEmid-refit, so those two questions had the same answer. The shard then serves rollouts from the checkpoint it loaded off disk, and the per-shardweight_versionmetric cannot catch it — the promotion is what writes that number. - A restarted engine inherits the dead engine's record.
mark_loadedclears the two probe counters but leavesstate_before_partialandconsecutive_reported_failuresset, so a fresh process comes backSUSPECTat 2 of 3 on the counter that condemns it. On the condemn-silent-participant path this is guaranteed rather than unlucky, because the test that selects the shard is a test that the field is set. RAY_process_group_cleanup_enablednever reaches the raylet. It is read by the raylet, not the driver, so setting it from Python is a no-op on Slurm and k8s — and that setting is what the EngineCore reaping depends on.
Recovery robustness
- The restart runs on
asyncio.to_threadwith no timeout. A hang leaves the shard inRESTARTINGfor the rest of the run, never retried and never charged an attempt, and can hold the process open at exit. drain()reads like a shutdown safety net. Nothing outside the tests calls it, and its timeout cannot bound the thread it waits on.mark_restart_failedtakes no reason, unlike both its siblings, so after a shard is retired the record still shows what originally killed it rather than why five reloads failed.- Nothing logs the target GPU before a restart is issued, so an orphaned EngineCore still holding the memory is only visible from inside the attempt that is already failing — and not at all when the replacement cannot start.
Tests and CI
- No CI lane collects
test_engine_supervisor.pyortest_engine_reaping_env.py, so 25 of this PR's new tests never run, andrun_unit.shtreats pytest's "no tests collected" exit 5 as success. Worth fixing before the rest: it is the finding that hides the others. - Three smaller ones: an assertion that also passes when zero restarts happened, no case pinning what happens when a replacement engine dies, and a
parametrizethat checks a parameter against itself.
Config and interface
restart_dead_shardsappears in neither SingleController exemplar YAML, so a user reading either file cannot tell restart exists.- One
++Hydra override in the recovery test, needed only because of the above.++turns off the unknown-key check, so a renamed field would silently fall back to defaults and the test would pass while proving nothing. restart_shardis the one backend capability found by duck typing rather than declared onGenerationInterface, and the rebuild-skip rule is now byte-identical in both synchronizers.
Comment cleanup (7 comments)
Several new comments read as notes to a reviewer rather than to a future reader — internal run ids, which merge deleted a method, where a line used to sit. The behaviour they describe is worth keeping; the run that found it is not. Each comment carries a suggested trim that keeps the rule and drops the history.
Beyond this PR: the same pattern appears about 88 more times in files already on main from the earlier PRs in this stack. Not this PR's problem — worth one sweep later.
Walkthroughs
Seven pages, one question each, for the mechanisms that are hard to see from the diff alone. Two are linked from the comments they explain.
- why a returning shard was shut out
- what actually gets recreated
- restart timing: what blocks on what
- promoted without weights — finding 1 above, start to finish
- old failures carry over — finding 2 above
- to_thread vs a dedicated daemon thread
- what a restart cannot do when a whole node dies
…ailure history on reload Three review findings from PR #3592, all rooted in the same thing: restart is the first event that can change fleet membership *while* a refit is in flight. 1. `_promote_refit_shards` promoted every STALE shard. Asking "is this shard STALE?" was sufficient until restart existed, because nothing could turn a shard STALE mid-refit. Now `mark_loaded` does, at whatever moment the reload lands, and STALE is deliberately not in `_ABSENT_STATES` -- so a shard that was absent when the communicator was built could still be promoted, returning it to service holding the checkpoint it read off disk instead of the trainer's weights. Capture the eligible set once, when membership settles, and filter promotion by it. Captured twice, in fact: `_recover_from_failed_refit` condemns the silent participant and rebuilds over the survivors, so the retry does not run against the first attempt's membership. 2. `mark_loaded` cleared the two probe counters but left `consecutive_reported_failures` and `state_before_partial` set. Both describe the process that died; neither is evidence about the fresh one taking its place, and `report_refit` reads both -- so a clean replacement came back SUSPECT, one failure away from being condemned again. 3. `test_engine_supervisor.py` and `test_engine_reaping_env.py` sat under `tests/unit/models/generation/`, which the `Other` lane ignores and the `Models_*` lanes also ignore, so 25 tests ran in no lane at all. Moved to `tests/unit/single_controller/`, where the code they cover lives. Both files walked `parents[4]` to the repo root, which the move breaks; now `parents[3]`. Signed-off-by: asolergibert <asolergibert@nvidia.com>
…apping Structural findings from the PR #3592 review, plus the test and comment fixes that go with them. No behaviour change intended; the suites that cover these paths pass unchanged. `should_rebuild()` moves into `membership.py`. Both hardened synchronizers had the same ten lines, byte for byte, comments included -- which is the bug class this stack keeps hitting, and the reason `desired_membership` was extracted in the first place. Not a move toward unifying the synchronizers: `_desired_membership`'s worker-group fallback stays where it is, because the two transports genuinely differ there (Dynamo degrades, nccl_reshard requires a worker group and should fail loudly without one). `shard_liveness_ref()` moves the shard-to-worker mapping onto the backend. `restart_shard` already asked the worker group which worker leads a shard; `_probe_generation_fleet` reached through the generation object and did the arithmetic itself, so the control loop knew the layout and assumed every backend has a `worker_group` -- the same assumption that broke the Dynamo L1 lane with `'DynamoGeneration' object has no attribute 'worker_group'`. Declaring it on GenerationInterface removes the assumption instead of guarding it, and the test fakes collapse from four nested SimpleNamespaces to one stub. The supervisor's counters move from `supervisor/` to `gen_fleet/` and `metrics()` becomes `as_metrics()`, matching GenerationFleetHealth -- the two dicts are merged two lines apart and `gen_fleet/restart_attempts` already carried restart state. Worth doing now rather than later: the rename is free forever, the key prefix is only free until something points at it. Tests: - the attempt-cap assertion was `len(...) <= max_attempts`, which also holds at zero -- the failure it exists to catch. Asserts the actual list now. - a replacement engine dying again was untested on both sides. STALE is deliberately not absent, so a second death has to be able to condemn it; the gap is new because this PR is what creates STALE-from-restart. - dropped a `parametrize` over one value that fed the test its own expected value, and the stray `[retired]` in the id that came with it. Comments: dropped the run ids, commit SHAs, stack positions and exact NVML readings that were notes to a reviewer rather than to a future reader. The rules they were attached to are kept; git keeps the history. The `seed=None` error string stays -- it is what someone hitting it will search for. Scoped to this PR's own comments; the same pattern on main is a separate sweep. Signed-off-by: asolergibert <asolergibert@nvidia.com>
|
/ok to test 17e9f2c |
|
/ok to test e0c9e73 |
…t refit
Squashed from the PR's original history. The branch carried 251 commits, 112 of
them merges of feat/sc-resiliency-01/02/03 while that stack was in flight; those
three landed on main as squashes, so their individual commits exist nowhere in
main and a plain rebase would have replayed ~139 commits whose content main
already has. The net diff was always this: 39 files, +3103/-226.
A dead generation shard used to be gone for the rest of the run. This restarts
it and hands it back through the fleet-health states:
DEAD --(restart starts)--> RESTARTING --(engine up)--> STALE --(refit)--> HEALTHY
RESTARTING is absent from collectives, so a rebuild mid-restart leaves the shard
out. STALE is present but not serving, which is what lets the next refit write
current weights into it before it takes traffic again.
What this contains beyond the core feature, all of it review-driven:
- The refit promotes only shards it actually reached. A restart finishing
mid-transfer turns a shard STALE, which is deliberately not absent, so "is it
STALE" stopped meaning "was it in the refit". The eligible set is captured when
membership settles, and captured again after a failed-refit recovery, because
that recovery rebuilds over the survivors.
- The weight version is stamped where the weights land, not by the promotion.
Nothing turns a shard STALE on a refit that succeeds, so a healthy fleet
reported version 0 forever and the alert built on it was firing on every shard.
- A restarted engine no longer inherits the dead one's failure record.
- The restart runs on a dedicated daemon thread bounded by restart_timeout_s,
with restart_backoff_s between attempts, a reason carried into the health
record, and a GPU reading taken before each attempt from the shard's own node.
- RAY_process_group_cleanup_enabled is exported in ray.sub, where the raylet
actually reads it; the driver-side setting was inert on every multi-node run.
- Router failures are counted once per drain window rather than once per request,
so one brief outage with several requests in flight can no longer condemn a
shard in a single tick.
- restart_shard, shard_liveness_ref and log_shard_gpu_state are declared on
GenerationInterface, and the Megatron backend implements the liveness probe.
Signed-off-by: asolergibert <asolergibert@nvidia.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
564953a to
a815a8c
Compare
|
/ok to test a815a8c |
| # minutes and nothing blocks it, so mark_loaded can turn a shard STALE mid-transfer | ||
| # -- and STALE is not absent, so asking again at promotion time would include a | ||
| # shard the communicator was deliberately built without. | ||
| participants = self._refit_participants() |
There was a problem hiding this comment.
1 action item.
TL;DR — participants is read after _reconcile_refit_membership has already taken its own read and awaited, so a shard that finishes restarting in that gap lands in participants while the communicator was built without it — the exact outcome this comment says it prevents.
The comment here, and _refit_participants's docstring, both state the contract:
Read once, here, because the answer changes underneath a refit. […] asking again at promotion time would include a shard the communicator was deliberately built without.
Right idea, one step short: reading here is still a second read, taken after the reconcile's own read and after an await.
- L3443 captures
absent = self._gen_fleet.absent_shards(), and L3447-L3449 builds the communicator from that set — over anawait asyncio.to_thread(...), which yields the loop. - The engine supervisor runs on that loop.
engine_supervisor.pyL174 callsmark_loaded, flipping a restarted shard RESTARTING → STALE. STALE is not absent. - Back here, L4705 re-reads the monitor — L3596 asks
absent_shards()again, "as of right now". - That shard is now STALE, so not absent, so in
participants— but it is not in the communicator and receives nothing. _record_refit_landedL3644-L3650 promotes shards that are STALE and inparticipants, so it goes back into the serving set holding the checkpoint it read off disk.
There is a companion page walking this through with a diagram: how a shard gets marked healthy by a refit that never sent it any weights. It now covers the guard this PR added and the window it leaves.
Reachability is narrow: it needs async_rl.generation_fleet_health.enabled: true (off by default) and a restart completing inside that window. But that is the configuration the restart feature exists to serve, so it is the one that matters.
AI-1
Make _reconcile_refit_membership return the absent set it reconciled against (alongside rebuilt), and have _refit_participants take that set as an argument instead of calling absent_shards() itself. Then the participant set and the communicator both come from the single read at L3443, with no window between them. No suggestion block because the fix changes the function's return type plus two call sites, which is not one contiguous region.
Both call sites need it: L4700-L4705 and the recovery re-snapshot at L4739, where the rebuild happens inside _recover_from_failed_refit and participants is read after it returns — same shape, and by the comment's own note it is the likelier of the two windows.
- 3rdparty/Gym-workspace/Gym: bb9b26de (0.5.0rc0) -> v0.6.0 (3045a793), matching the standalone Gym the alpha eval/configs were validated on (2026-09-15 decision: v0.6.0 fixed). - pyproject: override openai==2.44.0. Gym v0.6.0 pins openai==2.44.0 while sglang 0.5.12.post1 pins 2.6.1, and workspace members are resolved in every split; SGLang is unused on the alpha track. Upstream r0.8.0 moved sglang to a newer fork instead -- revisit on the v0.8 rebase. - uv.lock regenerated with uv 0.11.28 (+252/-105): openai 2.6.1 -> 2.44.0, Gym 0.6.0 deps added; torch/vllm 0.25.1/transformers/ray/TE/flashinfer-python unchanged. - nemo_rl/environments/nemo_gym.py: backport of upstream r0.8.0 (NVIDIA-NeMo#3592) -- build the per-agent counter after run_examples() (Gym >= 0.6 resolves task_source -> agent_ref there) and fail readably on unresolved rows. Datasets collated with Gym 0.6 carry only task_source and would otherwise KeyError on row["agent_ref"]. Verified (CPU only, CUDA_VISIBLE_DEVICES=""): nemo_gym 0.6.0 imports + NeMo-RL call sites; NeMo-RL gym unit tests 69 passed (default) + 1 passed (nemo_gym-marked, 2 GPU-only not run); alpha gym configs 4/4 valid; swe_pivot 184 / single_step_tool_use 30 / terminus_judge 84 passed; vllm & mcore extras sync from cache, AlphaBridge/MegatronPolicyWorker/alpha vLLM plugin import. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Note
Mirror of #3473, originally opened by @asolergi-nv from the
asolergi-nv/RLfork.The branch was re-pushed to
NVIDIA-NeMo/RLunchanged (same head SHA) so the four partscan be linked as a proper PR stack —
gh stackcannot operate on fork branches.Please review and comment here. The original #3473 is left open for history.
Part 4/4 of #3454
What this fixes
After #3472 a lost shard is dropped and the run continues on the rest — but the fleet only ever shrinks. A job that sheds a few transient failures over a long run ends permanently smaller, and nothing ever brings capacity back.
Two things were missing, and the second was a genuine defect:
1. Re-admission was structurally impossible.
reconcile_communicatorkeyed off "is anything absent", so the moment a restarted shard stopped being absent the answer wasFalse, no rebuild happened, and the shard stayed out of both the communicator and the refit dispatch for the rest of the run. Restarting an engine would have achieved nothing.2. Nothing promoted a restarted shard back into service.
GenerationFleetMonitor.report_refit— documented as "a completed refit is the only way back into the serving set" — was never called anywhere innemo_rl. A restarted shard therefore reachedSTALE, rejoined the communicator, received weights, and then sat idle for the rest of the run. The fleet recovered on paper and not in throughput. Every supervisor unit test passed because each one calledreport_refitby hand; the tests now drive it through the controller, which is what makes the wiring observable.What it does
Membership is compared against what was built, not against "is anything absent", so a shard leaving and a shard returning are one operation in opposite directions.
desired_membership()expresses it as "what it should be" rather than "what changed", which is what makes both directions fall out of one comparison. An unrecorded membership means the full fleet, not "unknown".EngineSupervisorrestarts dead shards, one background task each. Handover is entirely through fleet-health states:RESTARTINGis absent from collectives, so a rebuild landing mid-restart correctly excludes it.STALEis present but not serving — that is what lets the refit write current weights before the shard takes traffic again.The restart is not awaited. Reloading a model takes minutes, and the loop it runs from
also drives the rollout pump, the watchdog and the refit; blocking it would stall training
that the surviving shards can still do.
RayWorkerGroup.recreate_workerreplays the recorded creation call for one worker rather than re-deriving it — re-deriving means reproducing the placement group, bundle, venv and rank bookkeeping, a second implementation that would drift. The old actor is killed first: usually already dead, but an unresponsive one still holding the GPU would make the replacement fail on memory rather than on anything informative.Testing
tick()returns while the restart is still blocked.recreate_workertests pinning the replay — same bundle and placement group, old actor killed first, name uniquified per incarnation, and the recorded spec left unmutated so names do not compound as-r1-r2-r3.RESTART_DEAD_SHARDS=truemode added to the recovery functional test and registered in the SingleController lane. It asserts a restart was attempted, an engine came back, two rebuilds happened (one dropping the shard, one taking it back), and astale -> healthytransition occurred. Completion plus one rebuild would only prove the fleet shrank and carried on.Issues
List issues that this PR closes (syntax):
Usage
# Add a code snippet demonstrating how to use thisBefore your PR is "Ready for review"
Pre checks:
Additional Information