Repository navigation
docs(sidecar): add backend support matrix - #17855
Conversation
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
✅ Dynamo PR CI passed — run 41680008296 (attempt 1) on
|
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
|
No actionable comments were generated in the recent review. WalkthroughThe sidecar backend documentation adds a capability matrix for vLLM, SGLang, and TensorRT-LLM. It also describes operational constraints, request migration eligibility, shutdown behavior, and links to engine-specific guides. ChangesSidecar backend reference
Estimated code review effort: 2 (Simple) | ~8 minutes Merge Risk:
|
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Overview
Users evaluating Dynamo sidecars need to see the serving and operational gaps compared with the in-process backends. Add a compact support matrix for vLLM, SGLang, and TensorRT-LLM to the existing Sidecar Backends page.
Key Changes
Core logic: 0 lines; documentation and ownership routing only.
lib/sidecar/SUPPORT_NOTES.md.SUPPORT_NOTES.md, with links from sidecar READMEs and an active-development notice that the matrix and notes may become stale.Validation
Ownership validation covers 100% of tracked files; all 158 CODEOWNERS tests passed. The support notes use the existing Runtime and Sidecar teams, matching the common README.
Initial source and runtime validation at
318ab0390d8e3e1fa7517ea6a1ebf46a7854c236, with independent serving, operations, and recovery experiments. The final matrix was reconciled with main at76d66f76583f4903c65221c9c6dca63c7d0a2cf6, including merged SGLang media and TensorRT-LLM video support.506 sidecar/common CPU tests, 56 migration tests, and 3 shared-worker shutdown tests passed.
The frontend and sidecars at the initial validation revision passed real GPU text, JSON-schema output, output logprobs, and positive-temperature
n=2refusal checks for all three engines. vLLM and SGLang also returned prompt logprobs.Real SGLang HiCache allocated/advertised 4,097 host tokens and reloaded 799 tokens from the host tier after forced eviction. Native engine and Dynamo metrics were scraped separately.
TensorRT-LLM NGram speculation generated 64 tokens; native draft-token handoff and the launcher's custom-config refusal were exercised. A real prefill sidecar followed bounded draining and exited successfully.
Current operator admission tests and live Kubernetes server-side dry runs accepted baseline sidecar mode and rejected GMS, enabled checkpointing, and failover. Planner experiments discovered workers but made no load-scaling decision without FPM.
Documentation lint, asset-path checks, applicable pre-commit hooks,
fern check, andfern docs broken-linkspassed. Documentation updates passed page lint, asset-path checks, Fern configuration/link checks, and applicable pre-commit hooks. Existing Fern/navigation warnings remain outside this change.These experiments do not certify every model, multimodal, storage-tier, or speculative/disaggregated combination. GPU runs used Qwen3-0.6B with SGLang 0.5.22, TensorRT-LLM 1.4.0rc2, and vLLM 0.32.0 installed in a disposable SGLang CUDA image; the vLLM run used that image's torch 2.14.1 rather than vLLM's declared 2.13 pin. No runtime code or automated tests are added to the PR.
Issues and Ongoing Work
Part of #10839, the sidecar feature-parity inventory.
Pull Request Checklist
/pr-checklistskill.Summary by CodeRabbit