Repository navigation
feat(dynamo): run generation fleet health behind the Dynamo frontend - #4955
Draft
jthomson04 wants to merge 7 commits into
Draft
jthomson04 wants to merge 7 commits into
jthomson04 wants to merge 7 commits into
Conversation
`async_rl.generation_fleet_health` was refused with `dynamo: true`: the frontend picks engines itself, so it kept routing to a shard fleet health had quarantined, and a restarted shard never registered with it. GenerationInterface gains `set_serving_shards`, a no-op for backends that dispatch their own requests. The SingleController calls it on every fleet probe and after a refit returns shards to service. VllmGeneration, behind Dynamo, unregisters the engines of shards that left service and registers the ones that came back, starting a restarted shard's endpoint once a refit has given it current weights. Unregistering removes only the discovery record, so requests in flight finish. An engine whose Dynamo endpoint stops now fails its fleet probe. The Dynamo L1 suite reruns the shard-recovery test with Dynamo, with and without restarts, and checks the frontend's routing follows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
Author
|
/ok to test e57aaf8 |
Brings in NVIDIA-NeMo#4954's restack onto NVIDIA-NeMo#4905's merge of upstream main (e95b722). No conflicts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Contributor
Author
|
/ok to test c9fd1c1 |
Restacks onto NVIDIA-NeMo#4954 after NVIDIA-NeMo#4905's split (vLLM refactor NVIDIA-NeMo#4978, Dynamo core NVIDIA-NeMo#4905, NeMo-Gym through Dynamo NVIDIA-NeMo#4979) and the Gym#4817/#5161 rebase. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Contributor
Author
|
/ok to test ecafdfb |
Brings in NVIDIA-NeMo#4978's review fixes through the stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Restacks onto NVIDIA-NeMo#4954 after NVIDIA-NeMo#4979's review fixes and the Gym#4817/#5161 restack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Restacks onto NVIDIA-NeMo#4954 after NVIDIA-NeMo#4905's rollout-only split and NVIDIA-NeMo#4979's restack. The frontend-path test is NVIDIA-NeMo#4905's folded one; this PR's fleet-health tests and guide wording are kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
jthomson04
changed the base branch from
main
to
jthomson04/dynamo-chat-parity
November 2, 2026 04:10
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
With
policy.generation.dynamo: true,async_rl.generation_fleet_healthwas refused at startup. The frontend picks engines itself, so it kept routing to a shard that fleet health had quarantined, and a restarted shard never registered with it.GenerationInterface.set_serving_shards(serving_shards, *, timeout_s). A no-op by default, since a backend that dispatches its own requests reads the fleet selector on every request. The SingleController calls it on every fleet probe, next to the generation router's membership push, and after a refit returns shards to service, before generation resumes.VllmGenerationbehind Dynamo.Endpoint.unregister_endpoint_instance). This removes only the discovery record, so requests in flight finish.is_aliveraises) instead of looking healthy while getting no traffic.generation_routeris still refused with Dynamo; its message now says the frontend routes only to shards in service.Rollouts through the frontend report no per-shard outcome, so the probes alone judge the shards. This is documented in the guide.
Tests
set_serving_shardswith a shard that leaves, one that dies, one that doesn't answer, and a restarted one that returns;Endpoint, and the probe failing on a stopped endpoint;grpo_sc_generation_shard_recovery.shruns withdynamo=truein two variants. One kills a shard and checks it leaves the frontend's routing. The other,RESTART_DEAD_SHARDS=true, also checks the replacement rejoins. Both need at least 3 GPUs. CI's runners have 2, so these skip there, as all 7 native variants inL1_Functional_Tests_SingleController_2already do. They were run on GB200 instead (see Verification).Verification
Both fleet-health overlays, one GB200 node (4 GPUs: 2 Megatron trainers, 2 vLLM shards behind the Dynamo frontend), at this PR's head
e57aaf8ab6. Both pass all 24 steps. The table counts requests by the engine the frontend's KV router chose, taken from its per-request log.RESTART_DEAD_SHARDS=trueLocal, one GPU, Qwen3-0.6B, real Dynamo 1.5.1 frontend:
/v1/completionsset_serving_shards(set())set_serving_shards({0})Unit tests: 1501 passed (SingleController, refit, NeMo-Gym, Dynamo config) plus the vLLM-lane engine tests.
Types: pyrefly 0.24.2 reports 0 errors.
🤖 Generated with Claude Code