Repository navigation
feat(sidecar): withdraw vLLM workers on engine health loss and replacement - #16753
Draft
JulienDarve wants to merge 3 commits into
Draft
JulienDarve wants to merge 3 commits into
JulienDarve wants to merge 3 commits into
Conversation
This was referenced Oct 18, 2026
Contributor
✅ Dynamo PR CI passed — run 39632560285 (attempt 1) on
|
21 of 87 tasks
JulienDarve
force-pushed
the
jdarve/dis-3175-vllm-health-identity
branch
from
October 21, 2026 16:10
710dcbd to
31ccd00
Compare
…ement Watch vLLM's Control and Inference health continuously and track each engine lifetime as its own connection. NOT_SERVING withdraws the worker and rejects new requests while in-flight requests and the gRPC connection stay intact; the worker re-registers on the same connection once the engine serves again with an unchanged instance_id. A failed or closed health stream reconnects and compares GetServerInfo.instance_id: an unchanged ID resumes the same engine, and a new ID is a replacement that receives fresh KV publishers and refreshed model metadata. Startup requires a non-empty instance_id and a SERVING watch before registration. LoRA discovery records are withdrawn during reconnects and reconciled before re-registration. Part of #14897 Closes DIS-3175 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
…ace [skip ci] - Start the startup deadline before metadata bootstrap and keep it through registration, so bootstrap, health and recovery share one budget. A later recovery episode still starts a fresh deadline. - Recover a failed startup gate the same way as serving: an unchanged engine recovers in place and keeps its publishers; a changed engine is a replacement. A recovery that observes another instance cancels the stale connection so it reconnects instead of retrying in place. - Describe the vLLM behavior in the shared --grpc-startup-deadline-secs help. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
JulienDarve
force-pushed
the
jdarve/dis-3175-vllm-health-identity
branch
from
October 29, 2026 17:23
bd468ca to
2a2b179
Compare
…health-identity [skip ci] Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
This was referenced Oct 30, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The vLLM sidecar now watches engine health continuously and treats each engine lifetime as its own connection, using the PR 3 recovery hooks.
Health.Watchon the Control and Inference services reports Serving, NotServing, or Lost (the stream failed or closed). A quiet but healthy stream stays Serving.GetServerInfo.instance_idand re-registers on the same connection.instance_idmeans the same engine. A newinstance_idis a replacement: the worker attaches fresh KV publishers and refreshed model metadata.--grpc-startup-deadline-secsbudget (default 30 minutes). A later recovery gets a fresh deadline. The shared flag's help text now says that vLLM applies it once; SGLang and TensorRT-LLM still apply it to each startup phase.instance_idis a replacement. A recovery that sees anotherinstance_iddrops that connection and reconnects instead of retrying in place.instance_id, a SERVING watch, and an unchanged identity before registration.With live-only KV events, reconnecting to the same engine can still miss KV events, as on main. PR 5 adds replay verification.
Stack 4/6 (#16750 and #16751 merged → #16752 → #16753 → #16754 → #16755).
Where should the reviewer start?
lib/sidecar/vllm/src/recovery.rs(monitor_availability,reconnect), thengenerateadmission inengine.rs.Validation
cargo test --lib -p dynamo-vllm-sidecar: 135 passed;cargo test --lib -p dynamo-sidecar-common: 21 passed. That includes an identity change at startup recovering as a replacement, and a NOT_SERVING engine that keeps an in-flight request and recovers on the same connection.cargo test -p dynamo-sidecar-testkit, with sidecar binaries built from this branch: 22 + 25 passed.cargo test -p dynamo-vllm-mocker --test sidecar: 3 passed.cargo clippy --tests -- -D warnings.Related Issues
Generated with Claude Code