Repository navigation
Description
TL;DR
Today, Dynamo backends are designed as a wrapper on top of the inference engines (TRT-LLM, vLLM, SGLang and TokenSpeed). This introduces the following issues:
- We have to maintain these wrappers for all of the inference frameworks.
- Some of the features requires us to access private functions from the inference engines — which makes it fragile for version updates
- We have to release Dynamo containers for each of these workers (
vllm-runtime,sglang-runtimeandtrtllm-runtime) - The users need to learn a new worker CLI to migrate from stock upstream container to Dynamo.
- There is also a performance issue with GIL contention across PyO3 boundary.
This DEP proposes a new sidecar architecture that aims to solve these issues by pulling Dynamo out of the engine process and driving communication over a gRPC API boundary.
Summary
Today, Dynamo is embedded into each engine's container: the stock vLLM/SGLang/TensorRT-LLM image has to include Dynamo's dynamo-backend-common crate and an engine-specific LLMEngine implementation. That entangles our dependencies with the engine's and means we cannot run the stock upstream images directly the way llm-d and AIBRix do.
This DEP proposes a Dynamo Sidecar: a remote LLMEngine on the unified-backend boundary (already in main) that runs alongside an unmodified engine and drives it over an engine RPC. The Dynamo request plane — frontend, KV-aware router, SLA planner, NIXL, KVBM, disaggregation — is reused unchanged, so we get stock-image deployment without giving up our differentiators.
This is an umbrella DEP: it sets the direction and requirements and defers the concrete designs (RPC contract, feature-parity inventory, deployment, KV/disaggregation) to the child DEPs listed under Proposal.
Motivation
-
Dependency entanglement. Installing Dynamo onto stock engine images mixes our dependencies with the engine's and requires a custom per-engine Dynamo container. Crossing the Rust/PyO3 boundary in-process also carries GIL-contention risk we are still validating.
-
The ecosystem runs stock images. llm-d runs the upstream vLLM/SGLang container unmodified, and AIBRix runs stock vLLM for serving and patches it only for distributed KV offload. Both have a lower barrier to entry than Dynamo's in-process model. Dynamo cannot consume stock images today.
-
One boundary, cheap migration The sidecar and in-process paths share the same
LLMEngineboundary, so moving an engine to the sidecar is not a rewrite against a new interface. The same boundary also keeps in-process available for custom or experimental engines a developer wants to embed directly. -
Operator preference. We see signals that users prefer each engine's native
xxx serveexperience over a Dynamo-specificdynamo.xxxentrypoint, largely because CLI behavior and options diverge. -
Strategic positioning. A sidecar lets Dynamo offer zero-engine-change deployment while keeping its edge — the unified Rust request plane, per-iteration KV-aware routing, the SLA-driven predictive planner, framework-orchestrated disaggregation, KVBM multi-tier KV offload, and NIXL — in Dynamo's own fast plane rather than the engine's address space.
Proposal
Adopt a layered model on a single LLMEngine boundary. Today — in-process:
flowchart LR
C[Client<br/>OpenAI HTTP] --> F[Rust Frontend<br/>tokenize + chat template]
F --> R[KV-Aware Router]
R --> W[Unified Backend Worker<br/>one LLMEngine boundary]
subgraph img[Engine container — Dynamo embedded]
W --> T[LLMEngine trait]
T --> P[PyO3 shim]
P --> E[vLLM AsyncLLM / SGLang<br/>in-process · imports internals]
end
Proposed — sidecar: the Unified Backend becomes a sidecar driving a stock engine over an engine RPC. The Dynamo request plane is reused unchanged.
flowchart LR
C[Client] --> F[Rust Frontend<br/>tokenize + template]
F --> R[KV-Aware Router]
R --> S[Dynamo Sidecar<br/>remote LLMEngine impl · Rust]
subgraph pod[Kubernetes Pod / container-compose]
S <-->|engine RPC over UDS| E[Stock Engine container<br/>native vLLM serve / SGLang<br/>unmodified upstream image]
E -.->|KV events / load| S
end
The boundary supports both engine-location paths, with the sidecar as the target; the sidecar's contract is itself pluggable:
flowchart TD
B["Unified Backend Interface<br/>one LLMEngine boundary · landed in main"]
B --> SC["Remote LLMEngine — Sidecar<br/>target for vLLM · SGLang · TRT-LLM"]
B --> IP["In-process LLMEngine<br/>custom / experimental engines"]
SC --> OE["Contract option A: OpenEngine<br/>vendor-neutral, standardized RPC"]
SC --> ES["Contract option B: engine-specific RPC<br/>per-engine adapter · migration lever"]
Layer 1 — Unified Backend Interface (landed in main). Lifecycle, health, metrics, registration, cancellation, and shutdown solved once and inherited by every integrated engine, in-process or remote.
Layer 2 — Dynamo Sidecar (this DEP's new path). A remote LLMEngine implementation. The engine runs stock in its own container via its native serve; within a pod the transport is gRPC over a Unix domain socket. The Dynamo frontend, KV-aware router, SLA planner, NIXL KV transfer, KVBM, and migration/retry are all reused unchanged.
Layer 3 — the engine↔sidecar contract. Two realizations:
- OpenEngine (standardization track). A versioned, vendor-neutral gRPC contract each engine implements — generation, engine/model/load metadata, health/abort/drain, KV connector/event discovery and subscription, and a runtime-event stream. vLLM and SGLang prototypes show the engine keeps its own internals — Dynamo imports neither.
- Engine-specific RPC (autonomy track). A per-engine adapter against the engine's own native service surface, with no cross-engine standard to adhere to.
Prototype status — both contract options already exist in practice. The vLLM prototype (an OpenEngine gRPC service beside the native vLLM server path) is undergoing performance evaluation in private forks. For SGLang, a Rust sidecar bridge to its native gRPC server is open as ai-dynamo/dynamo#9402 — different terminology, but the same sidecar architecture via an engine-specific RPC. Together they exercise both realizations of the pluggable contract.
Child DEPs (deferred from this umbrella)
-
Engine RPC contract(s) (DEP: Sidecar — Engine RPC contract (OpenEngine + engine-specific) #10838) — the OpenEngine standardization option, and the engine-specific-adapter alternative; the decision framework for choosing per engine.
-
Functionality impact & feature-parity inventory (DEP: Sidecar — Functionality impact & feature-parity inventory #10839) — a per-engine audit mapping every current Python-wrapper responsibility to its destination (unified backend, sidecar, or upstream engine), and enumerating the upstream API gaps each engine (vLLM, SGLang, TRT-LLM) must solve before the sidecar can achieve parity.
-
Deployment (compose + Kubernetes operator) (DEP: Sidecar — Deployment (compose + Kubernetes operator) #10840) — how the sidecar discovers, launches, and monitors the stock engine; transport wiring; pod-termination ordering; the local/compose companion shape; and the operator pod/CRD spec, sidecar injection, readiness/drain, and Gateway API Inference Extension alignment.
-
KV & disaggregated serving over the sidecar (DEP: Sidecar — KV & disaggregated serving #10841) — KV-session handoff, KV-event sourcing and migration, and pluggable transfer backends (NIXL/UCX).
-
Sidecar evaluation criteria & benchmarks (DEP: Sidecar — Evaluation criteria & benchmarks #10842) — per-token RPC overhead budget, router-fidelity loss, ownership/engagement model with upstream engine teams, and the default-path decision.
Alternate Solutions
-
Status quo (in-process only). Continue embedding Dynamo in each engine image. Rejected as the direction for the main engines: dependency coupling, per-engine custom containers, inability to consume stock images, exposure to engine-internal churn. Retained only for custom or experimental engines that opt to embed in-process.
-
Frontend talks directly to the stock engine (no sidecar). A parallel effort under evaluation: the Dynamo frontend and router drive a stock upstream container's native server (e.g., its OpenAI/gRPC endpoint) directly, with no Dynamo sidecar process. It is the lowest-friction way to consume stock images, but forgoes the unified-backend worker boundary — lifecycle/health/metrics/registration and the
LLMEngineintegration point — and KV-aware routing and disaggregation depend on what the engine exposes directly. Architecturally close to the sidecar: both reuse the same Dynamo frontend and router and both consume stock images — they differ only in what sits between the router and the engine (a Dynamo worker vs. nothing). -
Adopt the llm-d / GIE model wholesale (Go control plane, no worker code, polled-Prometheus signal). Lowest barrier to entry, but loses Dynamo's per-iteration in-process KV-routing fidelity and the unified Rust request plane.
-
Fork/patch each engine (AIBRix-style per-version
.patchfor deep KV control). Brittle standardization treadmill; rejected.
Requirements
The engine↔sidecar contract — whether standardized (OpenEngine) or engine-specific — must:
- Generation — drive completion with pre-tokenized
token_idsor text, streamed. - Engine stays stock — require no fork of the upstream image; any KVBM / KV-event / disaggregation hooks ship as upstreamed engine connectors, not forks.
- Lifecycle & control — expose health/readiness, explicit abort, and streamed drain; support deterministic graceful shutdown and request cancellation.
- Disaggregated serving — support engine-owned KV sessions, prefill→decode handoff, and pluggable KV transfer backends (NIXL/UCX/etc.).
- Routing signal — provide a control-plane load signal sufficient for KV-aware routing and admission.
- Deployment — run both in Kubernetes (co-located sidecar container) and locally via container-compose.
- Observability — preserve standard Prometheus metrics plus an optional structured runtime-event stream.
Non-goals: making engines identical; replacing native engine APIs or internal execution paths; mandating a single cross-engine standard; committing the concrete RPC wire spec or operator design (deferred to child DEPs).
Activity
grahamking commentedon Jun 24, 2026
I'd like to explore this one further. It seems a more general solution that's easier for people to use. It's not tied to Kubernetes and it's as easy as
python -m dynamo.frontend+vllm serve --grpc <model>.We'd need to figure out registration / discovery but it doesn't seem too complex. sglang has Kubernetes discovery.
Today sglang's grpc server does not seem to work. I tried:
on these sglang versions:
mainbuilt from source.They all fail with a different error. I was usually able to find the error and fix in sglang repo, so each release they are fixing it and then breaking it a new way. This must mean their grpc server is not widely used.
This, however, works first try:
refactor(examples): drop the --unified launch flag
docs: fix stale references to removed unified backend
refactor(backends): remove experimental unified backend engines (python)
refactor(examples): drop the --unified launch flag
docs: fix stale references to removed unified backend
refactor(backends): remove experimental unified backend engines (python)
refactor(backends): remove experimental unified backend engines (python)
refactor(backends): remove experimental unified backend engines
refactor(backends): remove experimental unified backend engines
docs: fix stale references to removed unified backend
docs: fix stale references to removed unified backend
refactor(backends): remove experimental unified backend engines
refactor(backends): remove experimental unified backend engines
refactor(backends): remove experimental unified backend engines
docs: fix stale references to removed unified backend
refactor(backends): remove experimental unified backend implementations
docs: fix stale references to removed unified backend
Adapt the Dynamo serving stack design for this audience
Adapt the Dynamo serving stack design for this audience
Adapt the Dynamo serving stack design for this audience
Adapt the Dynamo serving stack design for this audience