Skip to content

feat(dynamo): run generation fleet health behind the Dynamo frontend - #4955

Draft
jthomson04 wants to merge 7 commits into
NVIDIA-NeMo:jthomson04/dynamo-chat-parityfrom
jthomson04:jthomson04/dynamo-fleet-health
Draft

jthomson04 wants to merge 7 commits into
NVIDIA-NeMo:jthomson04/dynamo-chat-parityfrom
jthomson04:jthomson04/dynamo-fleet-health

Conversation

@jthomson04

@jthomson04 jthomson04 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #4954 → #4979 → #4905 → #4978. Its base is jthomson04/dynamo-chat-parity, a copy of #4954's branch, so the diff shows only the top commit, feat(dynamo): run generation fleet health behind the Dynamo frontend.

What

With policy.generation.dynamo: true, async_rl.generation_fleet_health was refused at startup. The frontend picks engines itself, so it kept routing to a shard that fleet health had quarantined, and a restarted shard never registered with it.

  • GenerationInterface.set_serving_shards(serving_shards, *, timeout_s). A no-op by default, since a backend that dispatches its own requests reads the fleet selector on every request. The SingleController calls it on every fleet probe, next to the generation router's membership push, and after a refit returns shards to service, before generation resumes.
  • VllmGeneration behind Dynamo.
    • Shards that leave service unregister their engines (Dynamo 1.5.1's Endpoint.unregister_endpoint_instance). This removes only the discovery record, so requests in flight finish.
    • Shards that return register again.
    • A restarted shard starts its endpoint the first time it is back in service. That happens only after a refit has given it current weights; until then it holds dummy weights.
    • Each shard is best effort: one that doesn't answer within the probe timeout keeps its routing until the next probe. The Dynamo namespace and engine environment are kept as plain fields, so the SingleController's pickled copy can start a restarted shard's endpoint.
  • Probes. An engine whose Dynamo endpoint task has stopped now fails its fleet probe (is_alive raises) instead of looking healthy while getting no traffic.
  • Router rejection kept. generation_router is still refused with Dynamo; its message now says the frontend routes only to shards in service.

Rollouts through the frontend report no per-shard outcome, so the probes alone judge the shards. This is documented in the guide.

Tests

  • Unit:
    • set_serving_shards with a shard that leaves, one that dies, one that doesn't answer, and a restarted one that returns;
    • the worker's routing transitions against a fake Endpoint, and the probe failing on a stopped endpoint;
    • the refit-recovery test now checks the serving set is pushed before resume.
  • Functional (Dynamo L1, full mode): grpo_sc_generation_shard_recovery.sh runs with dynamo=true in two variants. One kills a shard and checks it leaves the frontend's routing. The other, RESTART_DEAD_SHARDS=true, also checks the replacement rejoins. Both need at least 3 GPUs. CI's runners have 2, so these skip there, as all 7 native variants in L1_Functional_Tests_SingleController_2 already do. They were run on GB200 instead (see Verification).

Verification

  • Both fleet-health overlays, one GB200 node (4 GPUs: 2 Megatron trainers, 2 vLLM shards behind the Dynamo frontend), at this PR's head e57aaf8ab6. Both pass all 24 steps. The table counts requests by the engine the frontend's KV router chose, taken from its per-request log.

    Variant Before the kill After the kill Every request succeeded
    Plain kill 1287 / 1273 survivor: 10240 yes
    RESTART_DEAD_SHARDS=true 1287 / 1273 survivor 5158, replacement 5082 yes
    • Plain kill: shard 0 is marked dead and leaves the frontend's routing.
    • Restart: the replacement starts, holds dummy weights while stale, and rejoins the routing only once the next refit gives it current weights. It then takes half the traffic.
  • Local, one GPU, Qwen3-0.6B, real Dynamo 1.5.1 frontend:

    Step Frontend instances /v1/completions
    Registered at startup 1 200
    set_serving_shards(set()) 0, at once 503
    set_serving_shards({0}) 1 200
  • Unit tests: 1501 passed (SingleController, refit, NeMo-Gym, Dynamo config) plus the vLLM-lane engine tests.

  • Types: pyrefly 0.24.2 reports 0 errors.

🤖 Generated with Claude Code

`async_rl.generation_fleet_health` was refused with `dynamo: true`: the
frontend picks engines itself, so it kept routing to a shard fleet health
had quarantined, and a restarted shard never registered with it.

GenerationInterface gains `set_serving_shards`, a no-op for backends that
dispatch their own requests. The SingleController calls it on every fleet
probe and after a refit returns shards to service. VllmGeneration, behind
Dynamo, unregisters the engines of shards that left service and registers
the ones that came back, starting a restarted shard's endpoint once a refit
has given it current weights. Unregistering removes only the discovery
record, so requests in flight finish. An engine whose Dynamo endpoint stops
now fails its fleet probe.

The Dynamo L1 suite reruns the shard-recovery test with Dynamo, with and
without restarts, and checks the frontend's routing follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04 jthomson04 added the CI:L1 Run doctests, unit tests, and functional tests label Oct 27, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 27, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test e57aaf8

@github-actions github-actions Bot added Documentation Improvements or additions to documentation CI Relating to CI labels Oct 27, 2026
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: e57aaf8 (PR #4955 from jthomson04/dynamo-fleet-health)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Brings in NVIDIA-NeMo#4954's restack onto NVIDIA-NeMo#4905's merge of upstream main (e95b722).
No conflicts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test c9fd1c1

@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: c9fd1c1 (PR #4955 from jthomson04/dynamo-fleet-health)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Restacks onto NVIDIA-NeMo#4954 after NVIDIA-NeMo#4905's split (vLLM refactor NVIDIA-NeMo#4978, Dynamo core
NVIDIA-NeMo#4905, NeMo-Gym through Dynamo NVIDIA-NeMo#4979) and the Gym#4817/#5161 rebase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04

Copy link
Copy Markdown
Contributor Author

/ok to test ecafdfb

Brings in NVIDIA-NeMo#4978's review fixes through the stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 34a620c (PR #4955 from jthomson04/dynamo-fleet-health)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Restacks onto NVIDIA-NeMo#4954 after NVIDIA-NeMo#4979's review fixes and the Gym#4817/#5161 restack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 124e9e0 (PR #4955 from jthomson04/dynamo-fleet-health)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Restacks onto NVIDIA-NeMo#4954 after NVIDIA-NeMo#4905's rollout-only split and NVIDIA-NeMo#4979's restack. The
frontend-path test is NVIDIA-NeMo#4905's folded one; this PR's fleet-health tests and
guide wording are kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 87783b0 (PR #4955 from jthomson04/dynamo-fleet-health)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: f8ea8cf (PR #4955 from jthomson04/dynamo-fleet-health)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

@jthomson04
jthomson04 changed the base branch from main to jthomson04/dynamo-chat-parity November 2, 2026 04:10
@terrykong terrykong added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:L1 Run doctests, unit tests, and functional tests labels Nov 5, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) CI Relating to CI Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants