Skip to content

DEP: Dynamo Sidecar — Remote Backend for stock engine images #10835

Open

Description

@tanmayv25

TL;DR

Today, Dynamo backends are designed as a wrapper on top of the inference engines (TRT-LLM, vLLM, SGLang and TokenSpeed). This introduces the following issues:

  • We have to maintain these wrappers for all of the inference frameworks.
  • Some of the features requires us to access private functions from the inference engines — which makes it fragile for version updates
  • We have to release Dynamo containers for each of these workers (vllm-runtime, sglang-runtime and trtllm-runtime)
  • The users need to learn a new worker CLI to migrate from stock upstream container to Dynamo.
  • There is also a performance issue with GIL contention across PyO3 boundary.

This DEP proposes a new sidecar architecture that aims to solve these issues by pulling Dynamo out of the engine process and driving communication over a gRPC API boundary.

Summary

Today, Dynamo is embedded into each engine's container: the stock vLLM/SGLang/TensorRT-LLM image has to include Dynamo's dynamo-backend-common crate and an engine-specific LLMEngine implementation. That entangles our dependencies with the engine's and means we cannot run the stock upstream images directly the way llm-d and AIBRix do.

This DEP proposes a Dynamo Sidecar: a remote LLMEngine on the unified-backend boundary (already in main) that runs alongside an unmodified engine and drives it over an engine RPC. The Dynamo request plane — frontend, KV-aware router, SLA planner, NIXL, KVBM, disaggregation — is reused unchanged, so we get stock-image deployment without giving up our differentiators.

This is an umbrella DEP: it sets the direction and requirements and defers the concrete designs (RPC contract, feature-parity inventory, deployment, KV/disaggregation) to the child DEPs listed under Proposal.

Motivation

  • Dependency entanglement. Installing Dynamo onto stock engine images mixes our dependencies with the engine's and requires a custom per-engine Dynamo container. Crossing the Rust/PyO3 boundary in-process also carries GIL-contention risk we are still validating.

  • The ecosystem runs stock images. llm-d runs the upstream vLLM/SGLang container unmodified, and AIBRix runs stock vLLM for serving and patches it only for distributed KV offload. Both have a lower barrier to entry than Dynamo's in-process model. Dynamo cannot consume stock images today.

  • One boundary, cheap migration The sidecar and in-process paths share the same LLMEngine boundary, so moving an engine to the sidecar is not a rewrite against a new interface. The same boundary also keeps in-process available for custom or experimental engines a developer wants to embed directly.

  • Operator preference. We see signals that users prefer each engine's native xxx serve experience over a Dynamo-specific dynamo.xxx entrypoint, largely because CLI behavior and options diverge.

  • Strategic positioning. A sidecar lets Dynamo offer zero-engine-change deployment while keeping its edge — the unified Rust request plane, per-iteration KV-aware routing, the SLA-driven predictive planner, framework-orchestrated disaggregation, KVBM multi-tier KV offload, and NIXL — in Dynamo's own fast plane rather than the engine's address space.

Proposal

Adopt a layered model on a single LLMEngine boundary. Today — in-process:

flowchart LR
  C[Client<br/>OpenAI HTTP] --> F[Rust Frontend<br/>tokenize + chat template]
  F --> R[KV-Aware Router]
  R --> W[Unified Backend Worker<br/>one LLMEngine boundary]
  subgraph img[Engine container — Dynamo embedded]
    W --> T[LLMEngine trait]
    T --> P[PyO3 shim]
    P --> E[vLLM AsyncLLM / SGLang<br/>in-process · imports internals]
  end
Loading

Proposed — sidecar: the Unified Backend becomes a sidecar driving a stock engine over an engine RPC. The Dynamo request plane is reused unchanged.

flowchart LR
  C[Client] --> F[Rust Frontend<br/>tokenize + template]
  F --> R[KV-Aware Router]
  R --> S[Dynamo Sidecar<br/>remote LLMEngine impl · Rust]
  subgraph pod[Kubernetes Pod / container-compose]
    S <-->|engine RPC over UDS| E[Stock Engine container<br/>native vLLM serve / SGLang<br/>unmodified upstream image]
    E -.->|KV events / load| S
  end
Loading

The boundary supports both engine-location paths, with the sidecar as the target; the sidecar's contract is itself pluggable:

flowchart TD
  B["Unified Backend Interface<br/>one LLMEngine boundary · landed in main"]
  B --> SC["Remote LLMEngine — Sidecar<br/>target for vLLM · SGLang · TRT-LLM"]
  B --> IP["In-process LLMEngine<br/>custom / experimental engines"]
  SC --> OE["Contract option A: OpenEngine<br/>vendor-neutral, standardized RPC"]
  SC --> ES["Contract option B: engine-specific RPC<br/>per-engine adapter · migration lever"]
Loading

Layer 1 — Unified Backend Interface (landed in main). Lifecycle, health, metrics, registration, cancellation, and shutdown solved once and inherited by every integrated engine, in-process or remote.

Layer 2 — Dynamo Sidecar (this DEP's new path). A remote LLMEngine implementation. The engine runs stock in its own container via its native serve; within a pod the transport is gRPC over a Unix domain socket. The Dynamo frontend, KV-aware router, SLA planner, NIXL KV transfer, KVBM, and migration/retry are all reused unchanged.

Layer 3 — the engine↔sidecar contract. Two realizations:

  • OpenEngine (standardization track). A versioned, vendor-neutral gRPC contract each engine implements — generation, engine/model/load metadata, health/abort/drain, KV connector/event discovery and subscription, and a runtime-event stream. vLLM and SGLang prototypes show the engine keeps its own internals — Dynamo imports neither.
  • Engine-specific RPC (autonomy track). A per-engine adapter against the engine's own native service surface, with no cross-engine standard to adhere to.

Prototype status — both contract options already exist in practice. The vLLM prototype (an OpenEngine gRPC service beside the native vLLM server path) is undergoing performance evaluation in private forks. For SGLang, a Rust sidecar bridge to its native gRPC server is open as ai-dynamo/dynamo#9402 — different terminology, but the same sidecar architecture via an engine-specific RPC. Together they exercise both realizations of the pluggable contract.

Child DEPs (deferred from this umbrella)

Alternate Solutions

  • Status quo (in-process only). Continue embedding Dynamo in each engine image. Rejected as the direction for the main engines: dependency coupling, per-engine custom containers, inability to consume stock images, exposure to engine-internal churn. Retained only for custom or experimental engines that opt to embed in-process.

  • Frontend talks directly to the stock engine (no sidecar). A parallel effort under evaluation: the Dynamo frontend and router drive a stock upstream container's native server (e.g., its OpenAI/gRPC endpoint) directly, with no Dynamo sidecar process. It is the lowest-friction way to consume stock images, but forgoes the unified-backend worker boundary — lifecycle/health/metrics/registration and the LLMEngine integration point — and KV-aware routing and disaggregation depend on what the engine exposes directly. Architecturally close to the sidecar: both reuse the same Dynamo frontend and router and both consume stock images — they differ only in what sits between the router and the engine (a Dynamo worker vs. nothing).

  • Adopt the llm-d / GIE model wholesale (Go control plane, no worker code, polled-Prometheus signal). Lowest barrier to entry, but loses Dynamo's per-iteration in-process KV-routing fidelity and the unified Rust request plane.

  • Fork/patch each engine (AIBRix-style per-version .patch for deep KV control). Brittle standardization treadmill; rejected.

Requirements

The engine↔sidecar contract — whether standardized (OpenEngine) or engine-specific — must:

  • Generation — drive completion with pre-tokenized token_ids or text, streamed.
  • Engine stays stock — require no fork of the upstream image; any KVBM / KV-event / disaggregation hooks ship as upstreamed engine connectors, not forks.
  • Lifecycle & control — expose health/readiness, explicit abort, and streamed drain; support deterministic graceful shutdown and request cancellation.
  • Disaggregated serving — support engine-owned KV sessions, prefill→decode handoff, and pluggable KV transfer backends (NIXL/UCX/etc.).
  • Routing signal — provide a control-plane load signal sufficient for KV-aware routing and admission.
  • Deployment — run both in Kubernetes (co-located sidecar container) and locally via container-compose.
  • Observability — preserve standard Prometheus metrics plus an optional structured runtime-event stream.

Non-goals: making engines identical; replacing native engine APIs or internal execution paths; mandating a single cross-engine standard; committing the concrete RPC wire spec or operator design (deferred to child DEPs).

Activity

  1. grahamking commented on Jun 24, 2026

    @grahamking
    Contributor

    Frontend talks directly to the stock engine (no sidecar).

    I'd like to explore this one further. It seems a more general solution that's easier for people to use. It's not tied to Kubernetes and it's as easy as python -m dynamo.frontend + vllm serve --grpc <model>.

    We'd need to figure out registration / discovery but it doesn't seem too complex. sglang has Kubernetes discovery.


    Today sglang's grpc server does not seem to work. I tried:

    sglang serve --grpc-mode --model-path ~/llms/Qwen3.5-0.8B/
    

    on these sglang versions:

    • 0.5.11
    • 0.5.12-post1
    • 0.5.13
    • 0.5.13-post1
    • and main built from source.

    They all fail with a different error. I was usually able to find the error and fix in sglang repo, so each release they are fixing it and then breaking it a new way. This must mean their grpc server is not widely used.

    This, however, works first try:

    vllm serve --grpc Qwen/Qwen3-4B-Instruct-2507
    
  2. added
    runtimeCODEOWNER area -> @ai-dynamo/dynamo-runtime-codeowners
    on Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backend::sglangRelates to the sglang backendbackend::trtllmRelates to the trtllm backendbackend::vllmRelates to the vllm backenddep:draftDEP in draft statusenhancementNew feature or requestrouterRelates to routing, KV-aware routing, etc.runtimeCODEOWNER area -> @ai-dynamo/dynamo-runtime-codeowners

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions