Skip to content

docs(sidecar): add backend support matrix - #17855

Merged
tanmayv25 merged 11 commits into
mainfrom
docs/dis-3316-sidecar-support-matrix
Nov 3, 2026
Merged

tanmayv25 merged 11 commits into
mainfrom
docs/dis-3316-sidecar-support-matrix

Conversation

@tanmayv25

@tanmayv25 tanmayv25 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Users evaluating Dynamo sidecars need to see the serving and operational gaps compared with the in-process backends. Add a compact support matrix for vLLM, SGLang, and TensorRT-LLM to the existing Sidecar Backends page.

Key Changes

Core logic: 0 lines; documentation and ownership routing only.

  • Keep the Fern serving, deployment, and operations matrices compact; link backend columns to per-engine sections in lib/sidecar/SUPPORT_NOTES.md.
  • Mark implemented serving features, opt-in migration, and Kubernetes deployments supported. Distinguish engine/container integration requirements from operator sidecar support, and keep Planner/autoscaling unsupported.
  • Keep caveats in SUPPORT_NOTES.md, with links from sidecar READMEs and an active-development notice that the matrix and notes may become stale.

Validation

  • Ownership validation covers 100% of tracked files; all 158 CODEOWNERS tests passed. The support notes use the existing Runtime and Sidecar teams, matching the common README.

  • Initial source and runtime validation at 318ab0390d8e3e1fa7517ea6a1ebf46a7854c236, with independent serving, operations, and recovery experiments. The final matrix was reconciled with main at 76d66f76583f4903c65221c9c6dca63c7d0a2cf6, including merged SGLang media and TensorRT-LLM video support.

  • 506 sidecar/common CPU tests, 56 migration tests, and 3 shared-worker shutdown tests passed.

  • The frontend and sidecars at the initial validation revision passed real GPU text, JSON-schema output, output logprobs, and positive-temperature n=2 refusal checks for all three engines. vLLM and SGLang also returned prompt logprobs.

  • Real SGLang HiCache allocated/advertised 4,097 host tokens and reloaded 799 tokens from the host tier after forced eviction. Native engine and Dynamo metrics were scraped separately.

  • TensorRT-LLM NGram speculation generated 64 tokens; native draft-token handoff and the launcher's custom-config refusal were exercised. A real prefill sidecar followed bounded draining and exited successfully.

  • Current operator admission tests and live Kubernetes server-side dry runs accepted baseline sidecar mode and rejected GMS, enabled checkpointing, and failover. Planner experiments discovered workers but made no load-scaling decision without FPM.

  • Documentation lint, asset-path checks, applicable pre-commit hooks, fern check, and fern docs broken-links passed. Documentation updates passed page lint, asset-path checks, Fern configuration/link checks, and applicable pre-commit hooks. Existing Fern/navigation warnings remain outside this change.

These experiments do not certify every model, multimodal, storage-tier, or speculative/disaggregated combination. GPU runs used Qwen3-0.6B with SGLang 0.5.22, TensorRT-LLM 1.4.0rc2, and vLLM 0.32.0 installed in a disposable SGLang CUDA image; the vLLM run used that image's torch 2.14.1 rather than vLLM's declared 2.13 pin. No runtime code or automated tests are added to the PR.

Issues and Ongoing Work

Part of #10839, the sidecar feature-parity inventory.

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

Summary by CodeRabbit

  • Documentation
    • Added a support matrix for vLLM, SGLang, and TensorRT-LLM, covering serving features, operations, and recovery.
    • Clarified which capabilities are supported, limited, unavailable, or not validated, along with engine-specific constraints.
    • Documented request migration defaults, retry eligibility and exclusions, bounded shutdown draining, and unsupported operator-managed recovery features.
    • Added links to engine-specific launch guides.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation docs labels Nov 2, 2026
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@devin-ai-integration

devin-ai-integration Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 41680008296 (attempt 1) on 0a9f8a05f1

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 41680008296. Updated on every full-CI run of this PR.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@tanmayv25
tanmayv25 marked this pull request as ready for review November 2, 2026 19:40
@tanmayv25
tanmayv25 requested review from a team as code owners November 2, 2026 19:40
@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

The sidecar backend documentation adds a capability matrix for vLLM, SGLang, and TensorRT-LLM. It also describes operational constraints, request migration eligibility, shutdown behavior, and links to engine-specific guides.

Changes

Sidecar backend reference

Layer / File(s) Summary
Capabilities and operational constraints
docs/fern/pages/developer-guide/knowledge-base/concepts/system-architecture/sidecar-backends.md
The page compares serving features and operations across three engines. It documents migration and shutdown constraints and links to engine-specific guides.

Estimated code review effort: 2 (Simple) | ~8 minutes

Merge Risk: Minimal · up to 70355

This documentation-only change adds a sidecar support matrix with no runtime behavior change. No concrete merge-blocking issue remains.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title clearly and concisely identifies the documentation change: adding a backend support matrix for sidecars.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description is thorough and substantively complete. It has an Overview explaining the purpose (a compact support matrix for vLLM, SGLang, and TensorRT-LLM on the Sidecar Backends page), Key Change…
Full details: Description check

Explanation

The description is thorough and substantively complete. It has an Overview explaining the purpose (a compact support matrix for vLLM, SGLang, and TensorRT-LLM on the Sidecar Backends page), Key Changes listing the documentation changes and where caveats live, a detailed Validation section (CODEOWNERS tests, CPU tests, GPU runs with engine versions, Fern and pre-commit checks), and an Issues and Ongoing Work section linking #10839. The Pull Request Checklist is present and checked. The template's 'Closes #XXXX' line is absent, but the author links a related tracking issue instead. The check passes because the template's required sections are present and filled in.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 318ab03 and 703552f.

Files selected for processing: 1
  • docs/fern/pages/developer-guide/knowledge-base/concepts/system-architecture/sidecar-backends.md

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@zhongdaor-nv zhongdaor-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sihp

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@tanmayv25
tanmayv25 requested review from a team as code owners November 2, 2026 22:39
@tanmayv25
tanmayv25 enabled auto-merge (squash) November 3, 2026 00:33
@tanmayv25
tanmayv25 disabled auto-merge November 3, 2026 01:26
Comment thread lib/sidecar/sglang/README.md Resolved
Comment thread lib/sidecar/trtllm/README.md Resolved
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@pull-request-size pull-request-size Bot added size/L and removed size/M labels Nov 3, 2026
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@tanmayv25
tanmayv25 requested review from a team as code owners November 3, 2026 17:00
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@tanmayv25
tanmayv25 enabled auto-merge (squash) November 3, 2026 22:53
@tanmayv25
tanmayv25 merged commit 1c537be into main Nov 3, 2026
113 checks passed
@tanmayv25
tanmayv25 deleted the docs/dis-3316-sidecar-support-matrix branch November 3, 2026 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs documentation Improvements or additions to documentation size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants