Skip to content

feat(sidecar): verify vLLM KV replay continuity before registration - #16754

Draft
JulienDarve wants to merge 2 commits into
jdarve/dis-3175-vllm-health-identityfrom
jdarve/dis-3176-vllm-kv-recovery-gate
Draft

JulienDarve wants to merge 2 commits into
jdarve/dis-3175-vllm-health-identityfrom
jdarve/dis-3176-vllm-kv-recovery-gate

Conversation

@JulienDarve

@JulienDarve JulienDarve commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The vLLM sidecar now registers only after every local data-parallel rank's KV recovery reports Success. Recovery is owned by the listener from PR 2: it rebuilds or catches up each rank's index from vLLM's replay endpoint, retries its own connections, and reports Success, MissingHistory or Failed. The sidecar acts on those outcomes together with engine health and identity.

  • Sources: every advertised rank must report a ZMQ msgpack source with a replay endpoint and a positive buffer_steps. Behavior change: KV routing without replay now fails at startup with an explicit error instead of serving from a live-only index. PR 1 configures replay in the launchers.
  • Registration: the worker registers only when every rank reports Success, engine health is SERVING, and instance_id is unchanged. A rank that loses continuity while serving moves back to recovering, which withdraws the worker until the listener repairs the gap.
  • Same-engine recovery: a NOT_SERVING engine and a KV-only gap recover on the existing connection. Ingestion continues and only new requests are rejected. When the health stream is lost, the worker is withdrawn and the listeners keep running with their own bounded retries. When the same instance_id returns, ranks still recovering are left alone, ranks at Success count toward registration, and only ranks whose round ran out of transport retries get Restart, capped by the recovery deadline. A different instance_id stops and awaits the old publishers, and fresh ones replay from zero. Startup uses the same path.
  • Outcomes: while engine health is verified, Failed ends recovery with an explicit error and never requests an engine Shutdown. While health is lost, the sidecar waits for health first; a rank that then refuses Restart (a final failure) ends recovery the same way. The sidecar checks instance_id before treating a failed rank as final, so an engine replaced while its health stream stayed connected is still recovered as a replacement. MissingHistory keeps the worker withdrawn until the engine is replaced or the recovery deadline expires; PR 6 adds the Shutdown request.

Merge with PR 6. On its own, this PR leaves a long-running engine without a way to recover from missing history. A sidecar restart after vLLM's 10,000-step replay buffer has wrapped would stay withdrawn until something else restarts the engine.

  • Supporting changes: the vLLM mocker reports its replay buffer size. The shared sidecar testkit turns off mocker prefix caching, because its KV routing is on by default and it covers request handling rather than KV recovery.

Stack 5/6 (#16750 and #16751 merged → #16752 → #16753 → #16754 → #16755).

Where should the reviewer start?

recovery_sources, wait_for_rank and recover_in_place in lib/sidecar/vllm/src/recovery.rs.

Validation

  • cargo test --lib -p dynamo-vllm-sidecar: 140 passed on this PR's code. That includes health loss at startup and while serving that keeps every listener running for the same engine, failed replay that restarts only after the same engine is verified (and exits when the restart is refused), a rank that loses continuity blocking registration, missing history waiting for a replacement, a changed instance_id recovering as a replacement even with a failed rank, a paused engine keeping in-flight requests and KV ingestion, and a KV-only gap rejecting new requests without cancelling in-flight ones.
  • With the full stack (feat(sidecar): shut down vLLM engines on confirmed missing KV history #16755's head, 868e68e789): backend-common 165 passed, SGLang sidecar 123 passed, the vLLM mocker sidecar tests 3 passed against the mocker's real ZMQ replay, and the sidecar testkit 1 + 37 + 45 passed with binaries built from the stack. cargo clippy --tests -p dynamo-llm -p dynamo-vllm-sidecar -p dynamo-backend-common -- -D warnings.
  • GPU check of the full stack against vLLM 0.32.0: see feat(sidecar): shut down vLLM engines on confirmed missing KV history #16755.
  • Not run: multi-GPU or DP ranks, disaggregated serving, routed requests through the Dynamo frontend, Kubernetes.

Related Issues

Generated with Claude Code

@devin-ai-integration

devin-ai-integration Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 39635196647 (attempt 1) on e82963d545

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 39635196647. Updated on every full-CI run of this PR.

…skip ci]

Turn on replay recovery for every vLLM KV event source and register the worker only after each rank's index is recovered. This is #16754 as one commit on the rebased #16753 after #16751 merged.

Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
JulienDarve added a commit that referenced this pull request Oct 29, 2026
… [skip ci]

On confirmed missing KV history, verify the engine's instance_id, send Control.Shutdown to that engine and wait for its replacement. This is #16755 as one commit on the restacked #16754.

Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
@JulienDarve
JulienDarve force-pushed the jdarve/dis-3176-vllm-kv-recovery-gate branch from 58dbc21 to 8348200 Compare October 29, 2026 18:03
…kv-recovery-gate [skip ci]

Signed-off-by: Julien Darve <jdarve@NVIDIA.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant