Repository navigation
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
338247c to
9dd85ca
Compare
305608c to
f3fd7e2
Compare
9dd85ca to
f73800c
Compare
f3fd7e2 to
37213bc
Compare
f73800c to
e7716c5
Compare
37213bc to
74c9462
Compare
e7716c5 to
e7c8156
Compare
d4290d4 to
fdf1938
Compare
33e004b to
a6808da
Compare
fdf1938 to
3228bf5
Compare
3228bf5 to
aa8e870
Compare
a6808da to
523addf
Compare
aa8e870 to
390601e
Compare
…o engines A refit of the prefill/decode engines now keeps in-flight KV handoffs whole. The Megatron worker's sleep is split into a pause (stop scheduling, keep admitted requests' KV) and a suspend; pause_for_refit runs the pause alone, pinned_handoff_block_count reads the prefill KV blocks still pinned for decode handoffs on the inference loop, and a wake of a paused-only engine just unpauses it. MegatronDynamoGeneration pauses the prefill engines first, so no new handoff starts, waits until the decode engines have pulled every block prefill still pins (PinnedHandoffTimeout past the deadline), suspends prefill, then decode; it resumes decode first, since decode is the reader of prefill's KV, and a failed suspend restores in that order too. Teardown attests decode termination to prefill: MegatronDynamoGeneration reads each decode engine's handoff owner (dynamo_endpoint_metadata), stops the decode engines, then has a prefill engine's rank zero run Megatron-LM's handoff-recovery hook (confirm_decode_terminated) against every listening prefill coordinator (dynamo_coordinator_address) so the blocks pinned for those owners are released before the prefill engines go down. A decode engine that did not stop cleanly is not attested. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Laura Dang <laurad@nvidia.com>
…tation check_alive probes every engine separately and treats a worker whose actor died (RayActorError) as dead, naming every dead engine and chaining the first error. The shutdown path calls rank zero's endpoint methods through the engines' worker handles with the control timeout (handoff owners and prefill coordinators read at once under one deadline), so a stuck engine is logged and still shut down instead of hanging the teardown. The MegatronGeneration endpoint wrappers go, since MegatronDynamoGeneration talks to each engine's rank zero directly, and init_collective no longer checks the membership it just built. confirm_decode_terminated attests every prefill coordinator concurrently and logs each failure with its address, so one unreachable coordinator no longer keeps the rest from releasing their pinned KV blocks; the overall wait is one per-coordinator timeout plus slack. An untagged prepare_for_generation no longer treats a paused engine as serving; it wakes it. The role-to-component mapping moves to dynamo_config, where the worker's endpoint start reads it too. Tests fake a dead actor in the liveness probe, hang endpoint calls during shutdown, and drive the pause, suspend and wake sequence against an engine that enforces Megatron-LM's control-signal states. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Laura Dang <laurad@nvidia.com>
390601e to
9b07997
Compare
…es by role The decode-then-prefill resume order was not needed. The engines keep their KV on the GPU across suspend and resume (the persist cache mode the handoff requires), and a handoff that reaches a decode engine still waking is queued and read into buffers the resume does not touch. In the weight synchronizer's sequence the kv_cache phase already wakes every engine at once, so the ordered wave never ran on the normal path and only shaped the restore after a failed suspend. Resume is now one wave in record order. The helper that picks the engines suspended after prefill now selects the decode role explicitly instead of "not prefill". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Laura Dang <laurad@nvidia.com>
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
One shutdown failure-isolation bug needs a fix (C1): a failed coordinator-address lookup suppresses termination notices to healthy prefill engines.
Reviewed all six changed files and their directly affected refit, worker-control, liveness, and teardown paths. All three delegated inspections completed; two report bodies weren't delivered, so those topics were inspected directly. This was a static review: tests and linters weren't run. The upstream Megatron source and later stack heads weren't available for verification.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
…through live prefill - Refit pin wait: reads, liveness probes and sleeps share one deadline. A spent deadline, a pin read or a probe it cuts off raises PinnedHandoffTimeout with the last known counts. Every poll probes the fleet, so a dead engine or stopped endpoint fails the wait as itself. The counts are printed every 10 s while it waits. - PinnedHandoffTimeout is a RuntimeError, so the controller's refit await no longer replaces it with a "rank not answering" abort. - The wait's message and docs name both ways a pin is released: by the decode engine that claimed it, or by Megatron-LM's prefill coordinator once an unclaimed offer passes its 300 s timeout. - Shutdown: each prefill coordinator address is read separately, and attestation goes through a prefill engine that answered. Failed owner or address reads are logged and teardown continues. No prefill engine is asked when there is no decode owner to attest. - Worker: _pause, _sleep and _wake set their flags before the barrier, so a failed barrier does not leave the engine marked awake. - confirm_decode_terminated catches only the wait running out or being cancelled, and documents the precondition it is called under at shutdown. - Tests for each of the above, including Ray-gone shutdown, a pin read or probe cut off by the deadline, and a stopped endpoint mid-wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Laura Dang <laurad@nvidia.com>
Signed-off-by: Laura Dang <laurad@nvidia.com>
What this changes
PR 4 of 6: keep in-flight KV handoffs intact during refit and release pinned prefill blocks when decode engines shut down. This extends the existing
MegatronDynamoGenerationinnemo_rl/models/generation/megatron/dynamo_generation.py; engine creation, endpoint registration, and refit fan-out are provided by the preceding PRs.Refit ordering
PinnedHandoffTimeout.prepare_for_generationalso wakes a paused engine.Shutdown and liveness
check_alive, alongside the existing endpoint-task checks.Diagrams come from the design doc "Megatron Prefill/Decode behind Dynamo in NeMo RL".
Stack
main ← PR 1 (#4801) ← PR 2 (#4912) ← PR 3 (#5023) ← PR 4 (#5024, this) ← PR 5 (#5025) ← PR 6 (#5097)
Review this PR against PR 3. The Dynamo generation switch remains guarded here; PR 5 wires rollouts through the frontend, and PR 6 adds NeMo-Gym serving.
Tests
Coverage is in the existing unit suites:
test_megatron_dynamo_generation.py: ordered pause/drain/suspend and one-wave resume, pinned-handoff timeout, recovery after suspension failure, shutdown/attestation order, partial-startup cleanup, dead actors, and bounded shutdown calls.test_megatron_dynamo_worker_mixin.py: coordinator attestation and worker event/engine state handling.test_megatron_generation.py: pause/suspend/wake state transitions, including abandoned pauses and untagged generation preparation.Before your PR is "Ready for review"