Skip to content

feat(megatron): Dynamo engine fleet with refit fan-out and ordered pause - #5024

Open
lauradang wants to merge 5 commits into
laurad/pd-sc-3-adapter-sidecarfrom
laurad/pd-sc-4-worker-pool
Open

lauradang wants to merge 5 commits into
laurad/pd-sc-3-adapter-sidecarfrom
laurad/pd-sc-4-worker-pool

Conversation

@lauradang

@lauradang lauradang commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

PR 4 of 6: keep in-flight KV handoffs intact during refit and release pinned prefill blocks when decode engines shut down. This extends the existing MegatronDynamoGeneration in nemo_rl/models/generation/megatron/dynamo_generation.py; engine creation, endpoint registration, and refit fan-out are provided by the preceding PRs.

Refit ordering

  • Pause prefill scheduling so no new handoff starts, while decode continues pulling the KV blocks prefill has pinned.
  • Wait for the pinned-block counts to reach zero, then suspend prefill followed by decode. One deadline covers the pause, drain, and suspend sequence; undrained pins raise PinnedHandoffTimeout.
  • Resume every engine in one wave after refit. No role order is needed on the way back up: the engines keep their KV on the GPU across suspend and resume (persist mode), and a handoff that reaches a decode engine still waking is queued and read into buffers the resume does not touch. If suspension fails, restore every engine the operation reached the same way, preserving the original error.
  • Split the Megatron worker's pause and suspend transitions so a paused engine retains admitted requests' KV and can either suspend or resume without repeating the pause. Untagged prepare_for_generation also wakes a paused engine.

Ordered pause

Shutdown and liveness

  • Read decode handoff-owner metadata, stop decode endpoints and engines, then report successfully stopped owners to the listening prefill coordinators before stopping prefill and the shared services. A decode engine that did not stop cleanly is not attested.
  • Bound worker control calls during shutdown. Read owner metadata and coordinator addresses concurrently within their respective deadlines, and attest each owner's termination to all prefill coordinators concurrently so an unreachable coordinator does not block the others.
  • Keep teardown best effort and idempotent, including cleanup after partial startup.
  • Report dead Ray actors by engine ID in check_alive, alongside the existing endpoint-task checks.

Shutdown

Diagrams come from the design doc "Megatron Prefill/Decode behind Dynamo in NeMo RL".

Stack

main ← PR 1 (#4801) ← PR 2 (#4912) ← PR 3 (#5023) ← PR 4 (#5024, this) ← PR 5 (#5025) ← PR 6 (#5097)

Review this PR against PR 3. The Dynamo generation switch remains guarded here; PR 5 wires rollouts through the frontend, and PR 6 adds NeMo-Gym serving.

Tests

Coverage is in the existing unit suites:

  • test_megatron_dynamo_generation.py: ordered pause/drain/suspend and one-wave resume, pinned-handoff timeout, recovery after suspension failure, shutdown/attestation order, partial-startup cleanup, dead actors, and bounded shutdown calls.
  • test_megatron_dynamo_worker_mixin.py: coordinator attestation and worker event/engine state handling.
  • test_megatron_generation.py: pause/suspend/wake state transitions, including abandoned pauses and untagged generation preparation.

Before your PR is "Ready for review"

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? (CPU unit suites only; GPU path not yet run on this base)
  • Did you add or update any necessary documentation? (arrives with PR 5)

@copy-pr-bot

copy-pr-bot Bot commented Nov 2, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@lauradang
lauradang force-pushed the laurad/pd-sc-3-adapter-sidecar branch from 338247c to 9dd85ca Compare November 2, 2026 17:50
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch from 305608c to f3fd7e2 Compare November 2, 2026 17:50
@lauradang
lauradang force-pushed the laurad/pd-sc-3-adapter-sidecar branch from 9dd85ca to f73800c Compare November 4, 2026 03:00
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch from f3fd7e2 to 37213bc Compare November 4, 2026 03:00
@lauradang lauradang changed the title feat(dynamo): Megatron worker pool behind the Dynamo frontend feat(megatron): Dynamo engine fleet with refit fan-out and ordered pause Nov 4, 2026
@lauradang
lauradang force-pushed the laurad/pd-sc-3-adapter-sidecar branch from f73800c to e7716c5 Compare November 4, 2026 03:39
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch from 37213bc to 74c9462 Compare November 4, 2026 03:39
@lauradang
lauradang force-pushed the laurad/pd-sc-3-adapter-sidecar branch from e7716c5 to e7c8156 Compare November 4, 2026 05:36
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch 2 times, most recently from d4290d4 to fdf1938 Compare November 5, 2026 09:26
@lauradang
lauradang force-pushed the laurad/pd-sc-3-adapter-sidecar branch 2 times, most recently from 33e004b to a6808da Compare November 5, 2026 09:27
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch from fdf1938 to 3228bf5 Compare November 5, 2026 09:27
@terrykong terrykong added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Nov 5, 2026
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch from 3228bf5 to aa8e870 Compare November 5, 2026 21:41
@lauradang
lauradang force-pushed the laurad/pd-sc-3-adapter-sidecar branch from a6808da to 523addf Compare November 5, 2026 21:41
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch from aa8e870 to 390601e Compare November 5, 2026 23:59
lauradang and others added 2 commits November 5, 2026 16:03
…o engines

A refit of the prefill/decode engines now keeps in-flight KV handoffs whole.
The Megatron worker's sleep is split into a pause (stop scheduling, keep
admitted requests' KV) and a suspend; pause_for_refit runs the pause alone,
pinned_handoff_block_count reads the prefill KV blocks still pinned for
decode handoffs on the inference loop, and a wake of a paused-only engine
just unpauses it. MegatronDynamoGeneration pauses the prefill engines first,
so no new handoff starts, waits until the decode engines have pulled every
block prefill still pins (PinnedHandoffTimeout past the deadline), suspends
prefill, then decode; it resumes decode first, since decode is the reader of
prefill's KV, and a failed suspend restores in that order too.

Teardown attests decode termination to prefill: MegatronDynamoGeneration
reads each decode engine's handoff owner (dynamo_endpoint_metadata), stops
the decode engines, then has a prefill engine's rank zero run Megatron-LM's
handoff-recovery hook (confirm_decode_terminated) against every listening
prefill coordinator (dynamo_coordinator_address) so the blocks pinned for
those owners are released before the prefill engines go down. A decode
engine that did not stop cleanly is not attested.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Laura Dang <laurad@nvidia.com>
…tation

check_alive probes every engine separately and treats a worker whose actor
died (RayActorError) as dead, naming every dead engine and chaining the first
error. The shutdown path calls rank zero's endpoint methods through the
engines' worker handles with the control timeout (handoff owners and prefill
coordinators read at once under one deadline), so a stuck engine is logged and
still shut down instead of hanging the teardown. The MegatronGeneration
endpoint wrappers go, since MegatronDynamoGeneration talks to each engine's
rank zero directly, and init_collective no longer checks the membership it just
built.

confirm_decode_terminated attests every prefill coordinator concurrently and
logs each failure with its address, so one unreachable coordinator no longer
keeps the rest from releasing their pinned KV blocks; the overall wait is one
per-coordinator timeout plus slack.

An untagged prepare_for_generation no longer treats a paused engine as
serving; it wakes it. The role-to-component mapping moves to dynamo_config,
where the worker's endpoint start reads it too.

Tests fake a dead actor in the liveness probe, hang endpoint calls during
shutdown, and drive the pause, suspend and wake sequence against an engine that
enforces Megatron-LM's control-signal states.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Laura Dang <laurad@nvidia.com>
@lauradang
lauradang force-pushed the laurad/pd-sc-4-worker-pool branch from 390601e to 9b07997 Compare November 6, 2026 00:05
…es by role

The decode-then-prefill resume order was not needed. The engines keep
their KV on the GPU across suspend and resume (the persist cache mode the
handoff requires), and a handoff that reaches a decode engine still waking
is queued and read into buffers the resume does not touch. In the weight
synchronizer's sequence the kv_cache phase already wakes every engine at
once, so the ordered wave never ran on the normal path and only shaped the
restore after a failed suspend. Resume is now one wave in record order.

The helper that picks the engines suspended after prefill now selects the
decode role explicitly instead of "not prefill".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Laura Dang <laurad@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@lauradang
lauradang marked this pull request as ready for review November 6, 2026 22:40

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One shutdown failure-isolation bug needs a fix (C1): a failed coordinator-address lookup suppresses termination notices to healthy prefill engines.

Reviewed all six changed files and their directly affected refit, worker-control, liveness, and teardown paths. All three delegated inspections completed; two report bodies weren't delivered, so those topics were inspected directly. This was a static review: tests and linters weren't run. The upstream Megatron source and later stack heads weren't available for verification.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

lauradang and others added 2 commits November 6, 2026 21:02
…through live prefill

- Refit pin wait: reads, liveness probes and sleeps share one deadline.
  A spent deadline, a pin read or a probe it cuts off raises
  PinnedHandoffTimeout with the last known counts. Every poll probes the
  fleet, so a dead engine or stopped endpoint fails the wait as itself.
  The counts are printed every 10 s while it waits.
- PinnedHandoffTimeout is a RuntimeError, so the controller's refit
  await no longer replaces it with a "rank not answering" abort.
- The wait's message and docs name both ways a pin is released: by the
  decode engine that claimed it, or by Megatron-LM's prefill coordinator
  once an unclaimed offer passes its 300 s timeout.
- Shutdown: each prefill coordinator address is read separately, and
  attestation goes through a prefill engine that answered. Failed owner
  or address reads are logged and teardown continues. No prefill engine
  is asked when there is no decode owner to attest.
- Worker: _pause, _sleep and _wake set their flags before the barrier,
  so a failed barrier does not leave the engine marked awake.
- confirm_decode_terminated catches only the wait running out or being
  cancelled, and documents the precondition it is called under at
  shutdown.
- Tests for each of the above, including Ray-gone shutdown, a pin read
  or probe cut off by the deadline, and a stopped endpoint mid-wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Laura Dang <laurad@nvidia.com>
Signed-off-by: Laura Dang <laurad@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants