Skip to content

DEP: Sidecar — Engine RPC contract (OpenEngine + engine-specific) #10838

Open

Description

@tanmayv25

Parent: #10835
Status: Draft
Depends on: none (foundational).

Summary

This DEP defines the engine-to-sidecar RPC contract options for the Dynamo Sidecar architecture. The parent DEP establishes the sidecar as a second engine integration path on the existing LLMEngine boundary. This DEP focuses on the contract between the sidecar and the engine process.

There are two viable paths:

  1. OpenEngine path: a standardized, vendor-neutral gRPC contract implemented by engines and consumed by Dynamo sidecars.
  2. Engine-specific RPC path: a per-engine adapter that talks to the engine's native RPC or control surface directly.

Background

Dynamo currently integrates engines in-process: Dynamo code and engine code share the engine container and runtime. The parent sidecar DEP proposes a remote LLMEngine implementation where the engine runs through its native serve path and the Dynamo sidecar drives it over an RPC boundary.

The sidecar architecture does not require a single universal engine RPC. The parent DEP intentionally leaves the engine-side contract pluggable. OpenEngine is the standardization track. Engine-specific RPCs are the autonomy track.

This DEP fills in that decision space.

Goals

The engine-side RPC contract must support:

  • streaming generation from text or token IDs
  • engine and model discovery
  • role discovery for aggregated, prefill, and decode engines
  • capability discovery for partial feature support
  • health, readiness, abort, and drain
  • load and capacity reporting
  • KV connector metadata and KV event source discovery
  • disaggregated serving handoff from prefill to decode
  • a typed error model
  • enough extensibility for engine-specific runtime metadata

Non-goals

This DEP does not:

  • require every engine to adopt OpenEngine
  • replace the existing in-process integration path
  • replace native engine APIs or CLIs
  • standardize engine internals such as schedulers, KV transfer protocols, grammar backends, or multimodal preprocessing
  • define Kubernetes/operator details
  • settle feature-parity requirements for each engine

Path A: OpenEngine

OpenEngine is a standardized gRPC API between a sidecar or runtime controller and an inference engine. It defines a common runtime surface for:

  • Generate
  • GetEngineInfo
  • GetModelInfo
  • GetLoad
  • Health
  • Abort
  • Drain
  • GetKvConnectorInfo
  • GetKvEventSources
  • SubscribeKvEvents
  • SubscribeRuntimeEvents

OpenEngine is intended to standardize the boundary, not the engine implementation. Each engine maps OpenEngine requests into its own native request path.

How the OpenEngine path works

flowchart LR
  F[Dynamo Frontend + Router] --> S[Dynamo Sidecar]
  S -->|OpenEngine gRPC| N[Native engine runtime]
Loading

The engine owns its native runtime. The OpenEngine server is an adapter layer that exposes a common control and generation API.

Benefits

  1. One stable runtime contract: Engines implement one external runtime API. Sidecars and future controllers target the same contract instead of depending on private engine internals.

  2. Cross-engine consistency: Dynamo can drive vLLM, SGLang, TRT-LLM, and future engines with the same conceptual lifecycle: discover, wait until ready, generate, abort, drain, observe load, and discover KV event sources.

  3. Engine-owned truth: The engine reports its own model identity, served model name, role, parallelism, KV block size, load, capacity, and supported features. This reduces duplicated configuration and prevents an external sidecar from incorrectly describing the engine.

  4. Capability discovery and incremental adoption: An engine can start with a minimal OpenEngine implementation and advertise only the features it supports. The sidecar can adapt to partial support instead of assuming all engines implement the same feature set.

  5. Portable disaggregated serving envelope: OpenEngine standardizes the outer disaggregated-serving shape: engine roles, prefill-ready events, KV session references, typed KV metadata, event source discovery, and DP-rank affinity. It does not require every engine to use the same KV transfer implementation.

  6. Better ecosystem alignment: If engines converge on OpenEngine, other systems can integrate with the same runtime contract. The value of adoption compounds as more engines and platforms implement it.

Costs and risks

  1. Standardization overhead: Every new cross-engine feature needs schema work, compatibility review, and versioning discipline. This can slow down fast-moving engine features.

  2. Lowest-common-denominator pressure: If the API becomes too conservative, it may fail to express important engine-specific features. If it becomes too broad, it may become difficult to implement and maintain.

  3. Engine-specific semantics still exist: OpenEngine can standardize the envelope, but it cannot make vLLM, SGLang, and TRT-LLM identical. Adapters still need engine-specific conversion logic.

  4. Upstream adoption dependency: The OpenEngine path only works well if upstream engine teams are willing to implement, test, and maintain the OpenEngine server.

  5. Contract maturity risk: If OpenEngine v1 lacks a feature required for parity, the engine either cannot support that sidecar mode yet, must use typed extension fields, or must wait for an additive proto update.

Path B: Engine-specific RPC

The engine-specific RPC path uses the engine's native RPC, control API, or purpose-built adapter instead of a shared standard contract.

This path still uses the Dynamo sidecar architecture. The sidecar remains a remote LLMEngine implementation, but the sidecar talks to the engine through an engine-specific API.

How the engine-specific path works

flowchart LR
  F[Dynamo Frontend + Router] --> S[Dynamo Sidecar]
  S -->|engine-specific RPC| N[Native engine runtime]
Loading

The RPC may be an existing engine gRPC service, a new engine-specific endpoint, or a thin adapter that exposes exactly the engine features Dynamo needs.

Benefits

  1. Maximum engine autonomy: The engine can expose its natural request types, control methods, metadata, and disaggregation semantics without fitting them into a cross-engine schema.

  2. Faster iteration: Engine teams can add features without waiting for OpenEngine schema changes or cross-engine agreement.

  3. Better fit for immature or experimental features: If an engine feature is changing quickly, engine-specific RPCs can expose it directly while the feature stabilizes.

  4. Potentially easier initial integration: If an engine already has a rich native RPC surface, the sidecar may be able to integrate against that surface faster than implementing a full OpenEngine server.

  5. No standardization tax for one-off engines: For internal or experimental engines that are unlikely to be used by many orchestrators, a standard API may not be worth the maintenance cost.

Costs and risks

  1. Per-engine adapter burden: Every engine requires custom client code, custom feature mapping, custom tests, and custom failure handling.

  2. Less portability: A sidecar written for one engine-specific RPC cannot be reused for another engine.

  3. Harder ecosystem alignment: External systems cannot target a shared contract. Each integration must learn the engine's native API.

  4. More duplicated semantics: Discovery, health, load, drain, abort, KV metadata, and error handling may all be expressed differently per engine.

  5. Higher long-term maintenance risk: If the native engine RPC changes, the sidecar must track those changes. This is especially risky if the sidecar depends on unstable or private APIs.

Comparison

Dimension OpenEngine Engine-specific RPC
Integration model Shared standard contract Per-engine adapter
Engine autonomy Medium-high: native internals preserved, boundary standardized Highest: boundary can be engine-native
Initial implementation cost Higher if engine lacks OpenEngine server Lower if native RPC already exists
Long-term maintenance Lower if multiple engines converge Higher due to per-engine drift
Feature velocity Slower for new standard fields Faster for engine-specific features
Capability discovery Standardized Custom per engine
Disaggregated serving Shared envelope, engine-specific payload Fully engine-specific
Ecosystem portability High Low-medium
Compatibility discipline Strong versioned contract Depends on engine API stability
Best fit Engines seeking broad runtime interoperability Engines prioritizing autonomy or rapid iteration

Decision framework

Choose OpenEngine when:

  • the engine team is willing to maintain an OpenEngine server;
  • the target feature set maps cleanly to OpenEngine v1;
  • multiple orchestrators or platforms may benefit from the same runtime API;
  • stable interoperability is more valuable than maximum API autonomy;
  • the engine wants a standard runtime-facing API distinct from its user-facing API;
  • partial adoption through capability discovery is acceptable.

Choose engine-specific RPC when:

  • the engine already has a mature native RPC that exposes the required controls;
  • required features are too engine-specific or too unstable for OpenEngine v1;
  • the engine team wants to move faster than the standard can evolve;
  • the engine is experimental, internal, or unlikely to need broad interoperability;
  • OpenEngine would force too much tunneling through extension fields;
  • parity would be significantly easier through the native engine API.

Retain in-process integration when:

  • the engine needs deep in-address-space integration;
  • critical signals are not available through any RPC;
  • the sidecar path fails performance or feature-parity evaluation;
  • the engine team does not want to maintain a remote runtime API.

Proposed recommendation

The sidecar architecture should remain contract-pluggable.

OpenEngine should be the preferred target for engines that want broad runtime interoperability and can express their serving model through the standard contract. Engine-specific RPC should remain a supported path for engines whose native API is already rich, whose required features do not fit OpenEngine yet, or whose development velocity would be hurt by standardization.

Conclusion

The sidecar architecture separates Dynamo's remote LLMEngine path from the specific engine RPC used to drive the engine. This DEP recommends preserving that separation.

OpenEngine is the right path when standard runtime interoperability matters and the engine can faithfully expose its behavior through a shared contract. Engine-specific RPC is the right path when autonomy, feature velocity, or native API fit matters more.

The goal is not to make engines identical. The goal is to define a clear engine-side RPC strategy that lets each engine choose the right boundary while still satisfying Dynamo's requirements for generation, lifecycle, observability, KV routing, and disaggregated serving.

Activity

  1. added
    runtimeCODEOWNER area -> @ai-dynamo/dynamo-runtime-codeowners
    on Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    backend::sglangRelates to the sglang backendbackend::trtllmRelates to the trtllm backendbackend::vllmRelates to the vllm backenddep:draftDEP in draft statusenhancementNew feature or requestruntimeCODEOWNER area -> @ai-dynamo/dynamo-runtime-codeowners

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions