Skip to content

feat(sidecar): withdraw vLLM workers on engine health loss and replacement - #16753

Draft
JulienDarve wants to merge 3 commits into
jdarve/dis-3174-worker-recovery-gatefrom
jdarve/dis-3175-vllm-health-identity
Draft

JulienDarve wants to merge 3 commits into
jdarve/dis-3174-worker-recovery-gatefrom
jdarve/dis-3175-vllm-health-identity

Conversation

@JulienDarve

@JulienDarve JulienDarve commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The vLLM sidecar now watches engine health continuously and treats each engine lifetime as its own connection, using the PR 3 recovery hooks.

  • Health: a Health.Watch on the Control and Inference services reports Serving, NotServing, or Lost (the stream failed or closed). A quiet but healthy stream stays Serving.
  • NotServing: the worker is withdrawn and new requests are rejected. When the engine reports SERVING again, the sidecar checks GetServerInfo.instance_id and re-registers on the same connection.
  • Lost: the sidecar reconnects. The same instance_id means the same engine. A new instance_id is a replacement: the worker attaches fresh KV publishers and refreshed model metadata.
  • In-flight requests: recovery never cancels them. Each one finishes, or fails with the engine's own stream error so the frontend can migrate it.
  • Startup deadline: one deadline starts before metadata bootstrap and runs through registration, so bootstrap, health and recovery share the --grpc-startup-deadline-secs budget (default 30 minutes). A later recovery gets a fresh deadline. The shared flag's help text now says that vLLM applies it once; SGLang and TensorRT-LLM still apply it to each startup phase.
  • Startup recovery: a failed startup gate recovers the same way as serving. An unchanged engine recovers in place and keeps its publishers; a changed instance_id is a replacement. A recovery that sees another instance_id drops that connection and reconnects instead of retrying in place.
  • Startup errors: a startup deadline error now includes the last connection failure, for example the service that did not become SERVING.
  • Startup: startup requires a non-empty instance_id, a SERVING watch, and an unchanged identity before registration.
  • LoRA: LoRA discovery records are withdrawn during reconnects and reconciled before re-registration.

With live-only KV events, reconnecting to the same engine can still miss KV events, as on main. PR 5 adds replay verification.

Stack 4/6 (#16750 and #16751 merged → #16752 → #16753 → #16754 → #16755).

Where should the reviewer start?

lib/sidecar/vllm/src/recovery.rs (monitor_availability, reconnect), then generate admission in engine.rs.

Validation

  • cargo test --lib -p dynamo-vllm-sidecar: 135 passed; cargo test --lib -p dynamo-sidecar-common: 21 passed. That includes an identity change at startup recovering as a replacement, and a NOT_SERVING engine that keeps an in-flight request and recovers on the same connection.
  • cargo test -p dynamo-sidecar-testkit, with sidecar binaries built from this branch: 22 + 25 passed. cargo test -p dynamo-vllm-mocker --test sidecar: 3 passed. cargo clippy --tests -- -D warnings.

Related Issues

Generated with Claude Code

@devin-ai-integration

devin-ai-integration Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 39632560285 (attempt 1) on 31ccd00a5d

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 39632560285. Updated on every full-CI run of this PR.

JulienDarve and others added 2 commits October 29, 2026 10:23
…ement

Watch vLLM's Control and Inference health continuously and track each
engine lifetime as its own connection. NOT_SERVING withdraws the worker
and rejects new requests while in-flight requests and the gRPC
connection stay intact; the worker re-registers on the same connection
once the engine serves again with an unchanged instance_id. A failed or
closed health stream reconnects and compares GetServerInfo.instance_id:
an unchanged ID resumes the same engine, and a new ID is a replacement
that receives fresh KV publishers and refreshed model metadata.

Startup requires a non-empty instance_id and a SERVING watch before
registration. LoRA discovery records are withdrawn during reconnects and
reconciled before re-registration.

Part of #14897
Closes DIS-3175

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
…ace [skip ci]

- Start the startup deadline before metadata bootstrap and keep it through
  registration, so bootstrap, health and recovery share one budget. A later
  recovery episode still starts a fresh deadline.
- Recover a failed startup gate the same way as serving: an unchanged engine
  recovers in place and keeps its publishers; a changed engine is a
  replacement. A recovery that observes another instance cancels the stale
  connection so it reconnects instead of retrying in place.
- Describe the vLLM behavior in the shared --grpc-startup-deadline-secs help.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
@JulienDarve
JulienDarve force-pushed the jdarve/dis-3175-vllm-health-identity branch from bd468ca to 2a2b179 Compare October 29, 2026 17:23
…health-identity [skip ci]

Signed-off-by: Julien Darve <jdarve@NVIDIA.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant