Repository navigation
feat(sidecar): verify vLLM KV replay continuity before registration - #16754
Draft
JulienDarve wants to merge 2 commits into
Draft
JulienDarve wants to merge 2 commits into
JulienDarve wants to merge 2 commits into
Conversation
This was referenced Oct 18, 2026
Contributor
✅ Dynamo PR CI passed — run 39635196647 (attempt 1) on
|
21 of 87 tasks
JulienDarve
force-pushed
the
jdarve/dis-3176-vllm-kv-recovery-gate
branch
from
October 21, 2026 16:10
bdf6cf1 to
e82963d
Compare
JulienDarve
force-pushed
the
jdarve/dis-3176-vllm-kv-recovery-gate
branch
from
October 29, 2026 18:03
58dbc21 to
8348200
Compare
…kv-recovery-gate [skip ci] Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The vLLM sidecar now registers only after every local data-parallel rank's KV recovery reports
Success. Recovery is owned by the listener from PR 2: it rebuilds or catches up each rank's index from vLLM's replay endpoint, retries its own connections, and reportsSuccess,MissingHistoryorFailed. The sidecar acts on those outcomes together with engine health and identity.buffer_steps. Behavior change: KV routing without replay now fails at startup with an explicit error instead of serving from a live-only index. PR 1 configures replay in the launchers.Success, engine health is SERVING, andinstance_idis unchanged. A rank that loses continuity while serving moves back to recovering, which withdraws the worker until the listener repairs the gap.instance_idreturns, ranks still recovering are left alone, ranks atSuccesscount toward registration, and only ranks whose round ran out of transport retries getRestart, capped by the recovery deadline. A differentinstance_idstops and awaits the old publishers, and fresh ones replay from zero. Startup uses the same path.Failedends recovery with an explicit error and never requests an engine Shutdown. While health is lost, the sidecar waits for health first; a rank that then refusesRestart(a final failure) ends recovery the same way. The sidecar checksinstance_idbefore treating a failed rank as final, so an engine replaced while its health stream stayed connected is still recovered as a replacement.MissingHistorykeeps the worker withdrawn until the engine is replaced or the recovery deadline expires; PR 6 adds the Shutdown request.Merge with PR 6. On its own, this PR leaves a long-running engine without a way to recover from missing history. A sidecar restart after vLLM's 10,000-step replay buffer has wrapped would stay withdrawn until something else restarts the engine.
Stack 5/6 (#16750 and #16751 merged → #16752 → #16753 → #16754 → #16755).
Where should the reviewer start?
recovery_sources,wait_for_rankandrecover_in_placeinlib/sidecar/vllm/src/recovery.rs.Validation
cargo test --lib -p dynamo-vllm-sidecar: 140 passed on this PR's code. That includes health loss at startup and while serving that keeps every listener running for the same engine, failed replay that restarts only after the same engine is verified (and exits when the restart is refused), a rank that loses continuity blocking registration, missing history waiting for a replacement, a changedinstance_idrecovering as a replacement even with a failed rank, a paused engine keeping in-flight requests and KV ingestion, and a KV-only gap rejecting new requests without cancelling in-flight ones.868e68e789): backend-common 165 passed, SGLang sidecar 123 passed, the vLLM mocker sidecar tests 3 passed against the mocker's real ZMQ replay, and the sidecar testkit 1 + 37 + 45 passed with binaries built from the stack.cargo clippy --tests -p dynamo-llm -p dynamo-vllm-sidecar -p dynamo-backend-common -- -D warnings.Related Issues
Generated with Claude Code