Repository navigation
Description
Parent: #10835
Status: Draft
Depends on: none (foundational).
Summary
This DEP defines the engine-to-sidecar RPC contract options for the Dynamo Sidecar architecture. The parent DEP establishes the sidecar as a second engine integration path on the existing LLMEngine boundary. This DEP focuses on the contract between the sidecar and the engine process.
There are two viable paths:
- OpenEngine path: a standardized, vendor-neutral gRPC contract implemented by engines and consumed by Dynamo sidecars.
- Engine-specific RPC path: a per-engine adapter that talks to the engine's native RPC or control surface directly.
Background
Dynamo currently integrates engines in-process: Dynamo code and engine code share the engine container and runtime. The parent sidecar DEP proposes a remote LLMEngine implementation where the engine runs through its native serve path and the Dynamo sidecar drives it over an RPC boundary.
The sidecar architecture does not require a single universal engine RPC. The parent DEP intentionally leaves the engine-side contract pluggable. OpenEngine is the standardization track. Engine-specific RPCs are the autonomy track.
This DEP fills in that decision space.
Goals
The engine-side RPC contract must support:
- streaming generation from text or token IDs
- engine and model discovery
- role discovery for aggregated, prefill, and decode engines
- capability discovery for partial feature support
- health, readiness, abort, and drain
- load and capacity reporting
- KV connector metadata and KV event source discovery
- disaggregated serving handoff from prefill to decode
- a typed error model
- enough extensibility for engine-specific runtime metadata
Non-goals
This DEP does not:
- require every engine to adopt OpenEngine
- replace the existing in-process integration path
- replace native engine APIs or CLIs
- standardize engine internals such as schedulers, KV transfer protocols, grammar backends, or multimodal preprocessing
- define Kubernetes/operator details
- settle feature-parity requirements for each engine
Path A: OpenEngine
OpenEngine is a standardized gRPC API between a sidecar or runtime controller and an inference engine. It defines a common runtime surface for:
GenerateGetEngineInfoGetModelInfoGetLoadHealthAbortDrainGetKvConnectorInfoGetKvEventSourcesSubscribeKvEventsSubscribeRuntimeEvents
OpenEngine is intended to standardize the boundary, not the engine implementation. Each engine maps OpenEngine requests into its own native request path.
How the OpenEngine path works
flowchart LR
F[Dynamo Frontend + Router] --> S[Dynamo Sidecar]
S -->|OpenEngine gRPC| N[Native engine runtime]
The engine owns its native runtime. The OpenEngine server is an adapter layer that exposes a common control and generation API.
Benefits
-
One stable runtime contract: Engines implement one external runtime API. Sidecars and future controllers target the same contract instead of depending on private engine internals.
-
Cross-engine consistency: Dynamo can drive vLLM, SGLang, TRT-LLM, and future engines with the same conceptual lifecycle: discover, wait until ready, generate, abort, drain, observe load, and discover KV event sources.
-
Engine-owned truth: The engine reports its own model identity, served model name, role, parallelism, KV block size, load, capacity, and supported features. This reduces duplicated configuration and prevents an external sidecar from incorrectly describing the engine.
-
Capability discovery and incremental adoption: An engine can start with a minimal OpenEngine implementation and advertise only the features it supports. The sidecar can adapt to partial support instead of assuming all engines implement the same feature set.
-
Portable disaggregated serving envelope: OpenEngine standardizes the outer disaggregated-serving shape: engine roles, prefill-ready events, KV session references, typed KV metadata, event source discovery, and DP-rank affinity. It does not require every engine to use the same KV transfer implementation.
-
Better ecosystem alignment: If engines converge on OpenEngine, other systems can integrate with the same runtime contract. The value of adoption compounds as more engines and platforms implement it.
Costs and risks
-
Standardization overhead: Every new cross-engine feature needs schema work, compatibility review, and versioning discipline. This can slow down fast-moving engine features.
-
Lowest-common-denominator pressure: If the API becomes too conservative, it may fail to express important engine-specific features. If it becomes too broad, it may become difficult to implement and maintain.
-
Engine-specific semantics still exist: OpenEngine can standardize the envelope, but it cannot make vLLM, SGLang, and TRT-LLM identical. Adapters still need engine-specific conversion logic.
-
Upstream adoption dependency: The OpenEngine path only works well if upstream engine teams are willing to implement, test, and maintain the OpenEngine server.
-
Contract maturity risk: If OpenEngine v1 lacks a feature required for parity, the engine either cannot support that sidecar mode yet, must use typed extension fields, or must wait for an additive proto update.
Path B: Engine-specific RPC
The engine-specific RPC path uses the engine's native RPC, control API, or purpose-built adapter instead of a shared standard contract.
This path still uses the Dynamo sidecar architecture. The sidecar remains a remote LLMEngine implementation, but the sidecar talks to the engine through an engine-specific API.
How the engine-specific path works
flowchart LR
F[Dynamo Frontend + Router] --> S[Dynamo Sidecar]
S -->|engine-specific RPC| N[Native engine runtime]
The RPC may be an existing engine gRPC service, a new engine-specific endpoint, or a thin adapter that exposes exactly the engine features Dynamo needs.
Benefits
-
Maximum engine autonomy: The engine can expose its natural request types, control methods, metadata, and disaggregation semantics without fitting them into a cross-engine schema.
-
Faster iteration: Engine teams can add features without waiting for OpenEngine schema changes or cross-engine agreement.
-
Better fit for immature or experimental features: If an engine feature is changing quickly, engine-specific RPCs can expose it directly while the feature stabilizes.
-
Potentially easier initial integration: If an engine already has a rich native RPC surface, the sidecar may be able to integrate against that surface faster than implementing a full OpenEngine server.
-
No standardization tax for one-off engines: For internal or experimental engines that are unlikely to be used by many orchestrators, a standard API may not be worth the maintenance cost.
Costs and risks
-
Per-engine adapter burden: Every engine requires custom client code, custom feature mapping, custom tests, and custom failure handling.
-
Less portability: A sidecar written for one engine-specific RPC cannot be reused for another engine.
-
Harder ecosystem alignment: External systems cannot target a shared contract. Each integration must learn the engine's native API.
-
More duplicated semantics: Discovery, health, load, drain, abort, KV metadata, and error handling may all be expressed differently per engine.
-
Higher long-term maintenance risk: If the native engine RPC changes, the sidecar must track those changes. This is especially risky if the sidecar depends on unstable or private APIs.
Comparison
| Dimension | OpenEngine | Engine-specific RPC |
|---|---|---|
| Integration model | Shared standard contract | Per-engine adapter |
| Engine autonomy | Medium-high: native internals preserved, boundary standardized | Highest: boundary can be engine-native |
| Initial implementation cost | Higher if engine lacks OpenEngine server | Lower if native RPC already exists |
| Long-term maintenance | Lower if multiple engines converge | Higher due to per-engine drift |
| Feature velocity | Slower for new standard fields | Faster for engine-specific features |
| Capability discovery | Standardized | Custom per engine |
| Disaggregated serving | Shared envelope, engine-specific payload | Fully engine-specific |
| Ecosystem portability | High | Low-medium |
| Compatibility discipline | Strong versioned contract | Depends on engine API stability |
| Best fit | Engines seeking broad runtime interoperability | Engines prioritizing autonomy or rapid iteration |
Decision framework
Choose OpenEngine when:
- the engine team is willing to maintain an OpenEngine server;
- the target feature set maps cleanly to OpenEngine v1;
- multiple orchestrators or platforms may benefit from the same runtime API;
- stable interoperability is more valuable than maximum API autonomy;
- the engine wants a standard runtime-facing API distinct from its user-facing API;
- partial adoption through capability discovery is acceptable.
Choose engine-specific RPC when:
- the engine already has a mature native RPC that exposes the required controls;
- required features are too engine-specific or too unstable for OpenEngine v1;
- the engine team wants to move faster than the standard can evolve;
- the engine is experimental, internal, or unlikely to need broad interoperability;
- OpenEngine would force too much tunneling through extension fields;
- parity would be significantly easier through the native engine API.
Retain in-process integration when:
- the engine needs deep in-address-space integration;
- critical signals are not available through any RPC;
- the sidecar path fails performance or feature-parity evaluation;
- the engine team does not want to maintain a remote runtime API.
Proposed recommendation
The sidecar architecture should remain contract-pluggable.
OpenEngine should be the preferred target for engines that want broad runtime interoperability and can express their serving model through the standard contract. Engine-specific RPC should remain a supported path for engines whose native API is already rich, whose required features do not fit OpenEngine yet, or whose development velocity would be hurt by standardization.
Conclusion
The sidecar architecture separates Dynamo's remote LLMEngine path from the specific engine RPC used to drive the engine. This DEP recommends preserving that separation.
OpenEngine is the right path when standard runtime interoperability matters and the engine can faithfully expose its behavior through a shared contract. Engine-specific RPC is the right path when autonomy, feature velocity, or native API fit matters more.
The goal is not to make engines identical. The goal is to define a clear engine-side RPC strategy that lets each engine choose the right boundary while still satisfying Dynamo's requirements for generation, lifecycle, observability, KV routing, and disaggregated serving.
Activity