Skip to content
Merged
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
d6a0744
feat(sidecar): speak TensorRT-LLM's OpenEngine gRPC API
tanmayv25 Sep 4, 2026
512c6e2
fix(trtllm-sidecar): route by the leg's own dp_rank and bound the dec…
tanmayv25 Sep 8, 2026
fec628f
fix(trtllm-sidecar): service a context request that ends without a ha…
tanmayv25 Sep 9, 2026
d8fca9b
fix(trtllm-sidecar): report the context phase's cache hits and harden…
tanmayv25 Sep 15, 2026
a9c25a0
refactor(sidecar): keep one Struct/JSON codec across protobuf versions
tanmayv25 Sep 15, 2026
566c68e
fix(trtllm-sidecar): reject dp-rank targeting and stop migrating on O…
tanmayv25 Sep 16, 2026
69e7c8b
docs(trtllm-sidecar): name the measured build by commit, not by versi…
tanmayv25 Sep 16, 2026
5f3cef9
docs(trtllm-sidecar): point the instructions at the published OpenEng…
tanmayv25 Sep 17, 2026
7172542
fix(trtllm-sidecar): give the quick-start manifest a context length
tanmayv25 Sep 17, 2026
bfd71c9
docs(trtllm-sidecar): name the max_seq_len pin as an upstream workaround
tanmayv25 Sep 17, 2026
b640c65
feat(trtllm-sidecar): discover KV events and per-rank routing load
tanmayv25 Sep 19, 2026
e419430
fix(trtllm-sidecar): supervise node-local KV routing publishers
tanmayv25 Sep 23, 2026
a71b0d4
feat(trtllm-sidecar): forward signed subagent affinity to OpenEngine
tanmayv25 Oct 14, 2026
88dc4d9
feat(trtllm-sidecar): discover KV events and per-rank routing load
tanmayv25 Sep 19, 2026
a10ad8f
fix(trtllm-sidecar): supervise node-local KV routing publishers
tanmayv25 Sep 23, 2026
6c732b9
feat(trtllm-sidecar): forward signed subagent affinity to OpenEngine
tanmayv25 Oct 14, 2026
abe1075
fix(trtllm-sidecar): validate KV routing port on current main
tanmayv25 Oct 14, 2026
5ba5090
chore(trtllm-sidecar): retain PR history after main port
tanmayv25 Oct 14, 2026
76c0dc8
fix(sidecar): initialize KV liveness in shared callers
tanmayv25 Oct 14, 2026
e87cfbc
fix(trtllm-sidecar): validate CLI before node discovery
tanmayv25 Oct 14, 2026
7ccdf2a
chore(trtllm-sidecar): merge latest Dynamo main
tanmayv25 Oct 14, 2026
2ed3f6a
fix(trtllm-sidecar): simplify optional control RPC fallbacks
tanmayv25 Oct 14, 2026
a0ae832
fix(trtllm-sidecar): align mock load and rank telemetry with routing
tanmayv25 Oct 14, 2026
0b84229
docs(trtllm): add KV routing launch and DGD examples
tanmayv25 Oct 15, 2026
d10df3f
chore(trtllm): move KV examples to stacked PR
tanmayv25 Oct 15, 2026
536d58a
docs(trtllm): add KV routing launch and DGD examples
tanmayv25 Oct 15, 2026
73d43ef
chore(trtllm): keep aggregate deployment out of feature PR
tanmayv25 Oct 15, 2026
8ff7ae1
chore(trtllm): sync examples stack with implementation
tanmayv25 Oct 15, 2026
c7443ea
fix(trtllm): use engine-primary KV routing DGDs
tanmayv25 Oct 15, 2026
829ccda
docs(trtllm): identify KV-capable release image
tanmayv25 Oct 19, 2026
9fe0baa
docs(trtllm): use KV-capable release image in examples
tanmayv25 Oct 19, 2026
733c1a9
chore(trtllm): sync examples with parent PR
tanmayv25 Oct 19, 2026
622d1f7
fix(trtllm): preserve routing compatibility and fence engine restarts
tanmayv25 Oct 19, 2026
6718628
chore(trtllm): sync examples with routing fixes
tanmayv25 Oct 19, 2026
3aa7fb7
fix(examples): merge nested TRTLLM KV memory settings
tanmayv25 Oct 22, 2026
59a7d31
chore(trtllm): merge main into KV routing examples
tanmayv25 Oct 22, 2026
770d268
fix(trtllm): bind example KV event publishers to loopback
tanmayv25 Oct 23, 2026
acccd92
docs(trtllm): pin published KV routing images
tanmayv25 Oct 23, 2026
e4f1886
fix(trtllm): move KV routing example gRPC ports below ephemeral range
tanmayv25 Oct 23, 2026
63dffb5
docs(trtllm): simplify KV routing example comments
tanmayv25 Oct 23, 2026
59a3c22
refactor(examples): simplify TensorRT-LLM memory config merging
tanmayv25 Oct 23, 2026
c2a839e
refactor(trtllm): use readable JSON heredocs in sidecar launchers
tanmayv25 Oct 23, 2026
7a007a7
fix(trtllm): honor KV launcher resource assignments and review feedback
tanmayv25 Oct 26, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ for the architecture.
|---|---|
| Aggregated | Yes |
| Disaggregated | Yes |
| KV routing | No |
| KV routing | Yes, with a compatible OpenEngine server and KVCM2 |

## Launch Locally

Expand Down Expand Up @@ -66,8 +66,8 @@ kubectl port-forward -n <namespace> svc/trtllm-sidecar-agg-frontend 8000:8000
## Topologies

The frontend reaches each sidecar over Dynamo's request, discovery, and event
planes; the sidecar reaches the engine over its native gRPC API. The
TensorRT-LLM sidecar does not publish KV cache events.
planes; the sidecar reaches the engine over its native gRPC API. The sidecar
discovers per-rank KV event sources and routing load from OpenEngine.

### Single-Node TP

Expand All @@ -84,7 +84,8 @@ node holds the remaining TP ranks.

### Multi-Node DP

Not supported yet. The TensorRT-LLM sidecar does not target DP ranks or
publish KV events.
The sidecar forwards the selected attention-DP rank to OpenEngine when the
server advertises rank targeting. KV routing requires a TensorRT-LLM build
with native KVCM2 event streaming and routing load support.

![TensorRT-LLM data parallelism across two nodes. Only the leader sidecar serves requests: the Dynamo Frontend router picks a DP rank and sends requests over the request plane to the node 0 Dynamo Sidecar, which registers DP ranks 0-3 and calls its local TensorRT-LLM over native gRPC.](../../../../../../assets/img/sidecar-trtllm-multinode-dp.svg)
2 changes: 2 additions & 0 deletions lib/backend-common/src/engine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -507,6 +507,8 @@ pub enum KvEventSource {
dp_rank: u32,
/// Model image-placeholder token used to normalize multimodal events.
image_token_id: Option<u32>,
/// Opt-in idle heartbeat deadline; absent preserves legacy publishers.
heartbeat_timeout: Option<std::time::Duration>,
},
Push {
on_ready: OnPublisherReady,
Expand Down
3 changes: 3 additions & 0 deletions lib/backend-common/src/publisher.rs
Original file line number Diff line number Diff line change
Expand Up @@ -75,13 +75,16 @@ fn setup_kv_publishers(
endpoint,
topic,
image_token_id,
heartbeat_timeout,
..
} => (
Some(KvEventSourceConfig::Zmq {
endpoint,
topic,
image_token_id,
video_token_id: None,
expected_rank: Some(dp_rank),
liveness: heartbeat_timeout.map(|timeout| (dp_rank, timeout)),
}),
None,
),
Expand Down
18 changes: 16 additions & 2 deletions lib/bindings/python/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 8 additions & 5 deletions lib/bindings/python/rust/backend.rs
Original file line number Diff line number Diff line change
Expand Up @@ -152,11 +152,13 @@ fn trtllm_sidecar_argv(argv: Vec<String>) -> Vec<String> {
#[pyo3(signature = (argv=None))]
fn _run_trtllm_sidecar(py: Python<'_>, argv: Option<Vec<String>>) -> PyResult<()> {
let cli_argv = trtllm_sidecar_argv(argv.unwrap_or_default());
let bootstrap = dynamo_trtllm_sidecar::TrtllmSidecarEngine::try_from_args_async(cli_argv)
.map_err(sidecar_startup_to_pyerr)?;

py.allow_threads(move || dynamo_sidecar_common::run(bootstrap))
.map_err(sidecar_run_to_pyerr)
py.allow_threads(move || dynamo_trtllm_sidecar::run(cli_argv))
.map_err(
|err| match err.downcast::<dynamo_sidecar_common::SidecarStartupError>() {
Ok(startup) => sidecar_startup_to_pyerr(startup),
Err(error) => pyo3::exceptions::PyRuntimeError::new_err(error.to_string()),
},
)
}

// ---------------------------------------------------------------------------
Expand Down Expand Up @@ -1599,6 +1601,7 @@ fn depythonize_kv_source(item: &Bound<'_, PyAny>) -> PyResult<RsKvEventSource> {
let dp_rank: u32 = item.getattr("dp_rank")?.extract()?;
match cls.as_str() {
"ZmqSource" => Ok(RsKvEventSource::Zmq {
heartbeat_timeout: None,
endpoint: item.getattr("endpoint")?.extract()?,
topic: item.getattr("topic")?.extract()?,
dp_rank,
Expand Down
2 changes: 2 additions & 0 deletions lib/bindings/python/rust/llm/kv.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1294,6 +1294,8 @@ impl KvEventPublisher {
video_token_id: Option<u32>,
) -> PyResult<Self> {
let source_config = zmq_endpoint.map(|ep| KvEventSourceConfig::Zmq {
liveness: None,
expected_rank: None,
endpoint: ep,
topic: zmq_topic.unwrap_or_default(),
image_token_id,
Expand Down
64 changes: 56 additions & 8 deletions lib/llm/src/kv_router/prefill_router/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,10 @@ use crate::{
discovery::{ModelManager, WorkerSetTarget, WorkerSetTargetId},
kv_router::{RoutingHost, SelectionPolicySource},
protocols::common::{
extensions::{SESSION_AFFINITY_CONTEXT_KEY, SessionAffinityId},
extensions::{
AGENT_CONTEXT_CONTEXT_KEY, AgentContext, SESSION_AFFINITY_CONTEXT_KEY,
SessionAffinityId,
},
llm_backend::{LLMEngineOutput, PreprocessedRequest},
preprocessor::{BootstrapInfo, PrefillResult, TraceLink},
timing::{RequestPhase, RequestTracker},
Expand All @@ -53,6 +56,28 @@ enum PrefillLifecycleState {
Unavailable = 2,
}

fn prefill_affinity_id<T: Send + Sync + 'static>(
context: &Context<T>,
) -> Result<Option<SessionAffinityId>> {
let child = context
.get_optional::<SessionAffinityId>(SESSION_AFFINITY_CONTEXT_KEY)
.map_err(|message| anyhow::anyhow!("invalid session affinity context: {message}"))?;
let parent = context
.get_optional::<AgentContext>(AGENT_CONTEXT_CONTEXT_KEY)
.map_err(|message| anyhow::anyhow!("invalid agent context: {message}"))?
.and_then(|agent| {
agent
.parent_session_id
.as_deref()
.map(str::trim)
.filter(|parent| !parent.is_empty() && *parent != agent.session_id.as_str())
.map(str::to_owned)
});
Ok(parent
.map(SessionAffinityId::new)
.or_else(|| child.map(|id| id.as_ref().clone())))
}

impl TryFrom<u8> for PrefillLifecycleState {
type Error = u8;

Expand Down Expand Up @@ -375,9 +400,7 @@ impl
return next.generate(context.map(|_| req)).await;
}

let session_affinity = context
.get_optional::<SessionAffinityId>(SESSION_AFFINITY_CONTEXT_KEY)
.map_err(|message| anyhow::anyhow!("invalid session affinity context: {message}"))?;
let session_affinity = prefill_affinity_id(&context)?;

if self.conditional_disagg_policy.is_enabled() {
let conditional_request = context.map(|_| req);
Expand Down Expand Up @@ -468,10 +491,7 @@ impl
let tracker = prefill_req.tracker.clone();
let mut prefill_context = independent_prefill_context(prefill_req, &context)?;
if let Some(session_affinity) = session_affinity {
prefill_context.insert(
SESSION_AFFINITY_CONTEXT_KEY,
session_affinity.as_ref().clone(),
);
prefill_context.insert(SESSION_AFFINITY_CONTEXT_KEY, session_affinity);
}
let Some(binding) = self.binding.load_full() else {
return next.generate(context.map(|_| req)).await;
Expand Down Expand Up @@ -784,6 +804,34 @@ mod tests {

const MAX_ROOM: u64 = i64::MAX as u64;

#[test]
fn child_context_uses_parent_affinity_while_decode_retains_child() {
let (_, mut context) = SingleIn::new(query_only_request()).into_parts();
context.insert(
SESSION_AFFINITY_CONTEXT_KEY,
SessionAffinityId::new("child-a"),
);
context.insert(
AGENT_CONTEXT_CONTEXT_KEY,
serde_json::from_value::<AgentContext>(serde_json::json!({
"session_id": "child-a",
"parent_session_id": "parent-1"
}))
.unwrap(),
);
assert_eq!(
prefill_affinity_id(&context).unwrap().unwrap().as_str(),
"parent-1"
);
assert_eq!(
context
.get::<SessionAffinityId>(SESSION_AFFINITY_CONTEXT_KEY)
.unwrap()
.as_str(),
"child-a"
);
}

#[derive(Default)]
struct QueryOnlyDecodeHost {
requests: AtomicUsize,
Expand Down
42 changes: 40 additions & 2 deletions lib/llm/src/kv_router/publisher/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};

use anyhow::Result;
use tokio::sync::mpsc;
use tokio::sync::{mpsc, oneshot};
use tokio_util::sync::CancellationToken;

use dynamo_kv_router::indexer::{KvIndexerMetrics, LocalKvIndexer};
Expand Down Expand Up @@ -77,6 +77,10 @@ pub enum KvEventSourceConfig {
/// Model video-placeholder token id. `None` leaves video runs on the
/// engine's native hashing path.
video_token_id: Option<u32>,
/// Source rank when this socket is dedicated to one DP rank.
expected_rank: Option<u32>,
/// Expected source rank and negotiated heartbeat timeout.
liveness: Option<(u32, std::time::Duration)>,
},
}

Expand Down Expand Up @@ -135,6 +139,8 @@ impl KvEventSource {
topic,
image_token_id,
video_token_id,
expected_rank,
liveness,
} => {
let listener_handle =
component
Expand All @@ -151,6 +157,8 @@ impl KvEventSource {
next_event_id,
image_token_id,
video_token_id,
expected_rank,
liveness,
));
let listener_abort_handle = listener_handle.abort_handle();
let supervisor_handle =
Expand Down Expand Up @@ -207,6 +215,9 @@ pub struct KvEventPublisher {
tx: mpsc::UnboundedSender<Vec<PlacementEvent>>,
/// Internal monotonic event ID counter. Shared with the ZMQ listener if present.
next_event_id: Arc<AtomicU64>,
/// Local pipeline initialization, independent of remote source activity.
startup: Option<oneshot::Receiver<()>>,
processor_task: tokio::task::JoinHandle<()>,
}

impl KvEventPublisher {
Expand Down Expand Up @@ -363,7 +374,8 @@ impl KvEventPublisher {

tracing::info!("Using event plane for KV event publishing");
let endpoint_clone = endpoint.clone();
component.drt().runtime().secondary().spawn(async move {
let (startup_tx, startup) = oneshot::channel();
let processor_task = component.drt().runtime().secondary().spawn(async move {
let event_publisher =
match dynamo_runtime::transports::event_plane::EventPublisher::for_endpoint_id(
endpoint_clone.drt(),
Expand Down Expand Up @@ -449,6 +461,7 @@ impl KvEventPublisher {
}
};

let _ = startup_tx.send(());
start_event_processor(
EventPlanePublisher(event_publisher),
worker_id,
Expand Down Expand Up @@ -481,9 +494,34 @@ impl KvEventPublisher {
worker_id,
tx,
next_event_id,
startup: Some(startup),
processor_task,
})
}

/// Wait for local publication and discovery initialization, not a remote event.
pub async fn ready(&mut self) -> Result<()> {
if let Some(startup) = self.startup.take() {
tokio::select! {
result = startup => result.map_err(|_| anyhow::anyhow!("KV event publisher initialization failed"))?,
_ = self.cancellation_token.cancelled() => anyhow::bail!("KV event listener stopped during initialization"),
}
}
anyhow::ensure!(
!self.cancellation_token.is_cancelled() && !self.processor_task.is_finished(),
"KV event publisher stopped during initialization"
);
Ok(())
}

/// Observe fatal termination. Ordinary gaps and heartbeat expiry keep listening.
pub async fn terminated(&mut self) {
tokio::select! {
_ = &mut self.processor_task => {},
_ = self.cancellation_token.cancelled() => {},
}
}

pub fn publish(&self, event: KvCacheEvent) -> Result<(), mpsc::error::SendError<KvCacheEvent>> {
self.send_singleton(PlacementEvent::local_gpu(self.worker_id, event))
}
Expand Down
Loading
Loading