Skip to content
Merged
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
d6a0744
feat(sidecar): speak TensorRT-LLM's OpenEngine gRPC API
tanmayv25 Sep 4, 2026
512c6e2
fix(trtllm-sidecar): route by the leg's own dp_rank and bound the dec…
tanmayv25 Sep 8, 2026
fec628f
fix(trtllm-sidecar): service a context request that ends without a ha…
tanmayv25 Sep 9, 2026
d8fca9b
fix(trtllm-sidecar): report the context phase's cache hits and harden…
tanmayv25 Sep 15, 2026
a9c25a0
refactor(sidecar): keep one Struct/JSON codec across protobuf versions
tanmayv25 Sep 15, 2026
566c68e
fix(trtllm-sidecar): reject dp-rank targeting and stop migrating on O…
tanmayv25 Sep 16, 2026
69e7c8b
docs(trtllm-sidecar): name the measured build by commit, not by versi…
tanmayv25 Sep 16, 2026
5f3cef9
docs(trtllm-sidecar): point the instructions at the published OpenEng…
tanmayv25 Sep 17, 2026
7172542
fix(trtllm-sidecar): give the quick-start manifest a context length
tanmayv25 Sep 17, 2026
bfd71c9
docs(trtllm-sidecar): name the max_seq_len pin as an upstream workaround
tanmayv25 Sep 17, 2026
b640c65
feat(trtllm-sidecar): discover KV events and per-rank routing load
tanmayv25 Sep 19, 2026
e419430
fix(trtllm-sidecar): supervise node-local KV routing publishers
tanmayv25 Sep 23, 2026
a71b0d4
feat(trtllm-sidecar): forward signed subagent affinity to OpenEngine
tanmayv25 Oct 14, 2026
88dc4d9
feat(trtllm-sidecar): discover KV events and per-rank routing load
tanmayv25 Sep 19, 2026
a10ad8f
fix(trtllm-sidecar): supervise node-local KV routing publishers
tanmayv25 Sep 23, 2026
6c732b9
feat(trtllm-sidecar): forward signed subagent affinity to OpenEngine
tanmayv25 Oct 14, 2026
abe1075
fix(trtllm-sidecar): validate KV routing port on current main
tanmayv25 Oct 14, 2026
5ba5090
chore(trtllm-sidecar): retain PR history after main port
tanmayv25 Oct 14, 2026
76c0dc8
fix(sidecar): initialize KV liveness in shared callers
tanmayv25 Oct 14, 2026
e87cfbc
fix(trtllm-sidecar): validate CLI before node discovery
tanmayv25 Oct 14, 2026
7ccdf2a
chore(trtllm-sidecar): merge latest Dynamo main
tanmayv25 Oct 14, 2026
2ed3f6a
fix(trtllm-sidecar): simplify optional control RPC fallbacks
tanmayv25 Oct 14, 2026
a0ae832
fix(trtllm-sidecar): align mock load and rank telemetry with routing
tanmayv25 Oct 14, 2026
0b84229
docs(trtllm): add KV routing launch and DGD examples
tanmayv25 Oct 15, 2026
d10df3f
chore(trtllm): move KV examples to stacked PR
tanmayv25 Oct 15, 2026
73d43ef
chore(trtllm): keep aggregate deployment out of feature PR
tanmayv25 Oct 15, 2026
829ccda
docs(trtllm): identify KV-capable release image
tanmayv25 Oct 19, 2026
622d1f7
fix(trtllm): preserve routing compatibility and fence engine restarts
tanmayv25 Oct 19, 2026
1e40c40
fix(trtllm): validate only the current worker rank
tanmayv25 Oct 20, 2026
742ec2d
docs(trtllm): simplify KV routing support entry
tanmayv25 Oct 20, 2026
9bef178
docs(trtllm): capitalize KV routing support entry
tanmayv25 Oct 20, 2026
e02454e
docs(trtllm): simplify rank routing description
tanmayv25 Oct 20, 2026
f8dc07f
docs(trtllm): correct affinity setup and sidecar roles
tanmayv25 Oct 20, 2026
73e19bd
fix(trtllm): preserve prefill affinity and bound routing health failures
tanmayv25 Oct 20, 2026
e9e9427
fix(trtllm): merge main and preserve shared sidecar conformance
tanmayv25 Oct 20, 2026
ba0c8da
fix(trtllm): reconcile publisher tests with latest main
tanmayv25 Oct 20, 2026
517e9ee
fix(trtllm): address routing lifecycle and compatibility review
tanmayv25 Oct 21, 2026
bbb8fa4
Potential fix for pull request finding 'CodeQL / Cleartext logging of…
tanmayv25 Oct 21, 2026
cca76eb
fix(trtllm): omit single-rank hints and protect affinity secrets
tanmayv25 Oct 21, 2026
f8a3646
style(trtllm): format executable help regression test
tanmayv25 Oct 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,17 @@ for the architecture.
> which carries the latest gRPC server updates. Nightly builds are the
> `nvcr.io/nvidia/tensorrt-llm/release:<version>.dev<date>` tags on
> [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/tensorrt-llm/containers/release/-).
> KV routing is available in
> `nvcr.io/nvidia/tensorrt-llm/release:1.4.0a0.dev202610180820`, which includes
> native KVCM2 event streaming and OpenEngine routing load support.

## Support Matrix

| Feature | Supported |
|---|---|
| Aggregated | Yes |
| Disaggregated | Yes |
| KV routing | No |
| KV routing | Yes, with a compatible OpenEngine server and KVCM2 |

## Launch Locally

Expand Down Expand Up @@ -66,8 +69,8 @@ kubectl port-forward -n <namespace> svc/trtllm-sidecar-agg-frontend 8000:8000
## Topologies

The frontend reaches each sidecar over Dynamo's request, discovery, and event
planes; the sidecar reaches the engine over its native gRPC API. The
TensorRT-LLM sidecar does not publish KV cache events.
planes; the sidecar reaches the engine over its native gRPC API. The sidecar
discovers per-rank KV event sources and routing load from OpenEngine.

### Single-Node TP

Expand All @@ -84,7 +87,8 @@ node holds the remaining TP ranks.

### Multi-Node DP

Not supported yet. The TensorRT-LLM sidecar does not target DP ranks or
publish KV events.
The sidecar forwards the selected attention-DP rank to OpenEngine when the
server advertises rank targeting. KV routing requires a TensorRT-LLM build
with native KVCM2 event streaming and routing load support.

![TensorRT-LLM data parallelism across two nodes. Only the leader sidecar serves requests: the Dynamo Frontend router picks a DP rank and sends requests over the request plane to the node 0 Dynamo Sidecar, which registers DP ranks 0-3 and calls its local TensorRT-LLM over native gRPC.](../../../../../../assets/img/sidecar-trtllm-multinode-dp.svg)
2 changes: 2 additions & 0 deletions lib/backend-common/src/engine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -507,6 +507,8 @@ pub enum KvEventSource {
dp_rank: u32,
/// Model image-placeholder token used to normalize multimodal events.
image_token_id: Option<u32>,
/// Opt-in idle heartbeat deadline; absent preserves legacy publishers.
heartbeat_timeout: Option<std::time::Duration>,
},
Push {
on_ready: OnPublisherReady,
Expand Down
3 changes: 3 additions & 0 deletions lib/backend-common/src/publisher.rs
Original file line number Diff line number Diff line change
Expand Up @@ -75,13 +75,16 @@ fn setup_kv_publishers(
endpoint,
topic,
image_token_id,
heartbeat_timeout,
..
} => (
Some(KvEventSourceConfig::Zmq {
endpoint,
topic,
image_token_id,
video_token_id: None,
expected_rank: Some(dp_rank),
liveness: heartbeat_timeout.map(|timeout| (dp_rank, timeout)),
}),
None,
),
Expand Down
18 changes: 16 additions & 2 deletions lib/bindings/python/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 8 additions & 5 deletions lib/bindings/python/rust/backend.rs
Original file line number Diff line number Diff line change
Expand Up @@ -152,11 +152,13 @@ fn trtllm_sidecar_argv(argv: Vec<String>) -> Vec<String> {
#[pyo3(signature = (argv=None))]
fn _run_trtllm_sidecar(py: Python<'_>, argv: Option<Vec<String>>) -> PyResult<()> {
let cli_argv = trtllm_sidecar_argv(argv.unwrap_or_default());
let bootstrap = dynamo_trtllm_sidecar::TrtllmSidecarEngine::try_from_args_async(cli_argv)
.map_err(sidecar_startup_to_pyerr)?;

py.allow_threads(move || dynamo_sidecar_common::run(bootstrap))
.map_err(sidecar_run_to_pyerr)
py.allow_threads(move || dynamo_trtllm_sidecar::run(cli_argv))
.map_err(
|err| match err.downcast::<dynamo_sidecar_common::SidecarStartupError>() {
Ok(startup) => sidecar_startup_to_pyerr(startup),
Err(error) => pyo3::exceptions::PyRuntimeError::new_err(error.to_string()),
},
)
}

// ---------------------------------------------------------------------------
Expand Down Expand Up @@ -1599,6 +1601,7 @@ fn depythonize_kv_source(item: &Bound<'_, PyAny>) -> PyResult<RsKvEventSource> {
let dp_rank: u32 = item.getattr("dp_rank")?.extract()?;
match cls.as_str() {
"ZmqSource" => Ok(RsKvEventSource::Zmq {
heartbeat_timeout: None,
endpoint: item.getattr("endpoint")?.extract()?,
topic: item.getattr("topic")?.extract()?,
dp_rank,
Expand Down
2 changes: 2 additions & 0 deletions lib/bindings/python/rust/llm/kv.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1294,6 +1294,8 @@ impl KvEventPublisher {
video_token_id: Option<u32>,
) -> PyResult<Self> {
let source_config = zmq_endpoint.map(|ep| KvEventSourceConfig::Zmq {
liveness: None,
expected_rank: None,
endpoint: ep,
topic: zmq_topic.unwrap_or_default(),
image_token_id,
Expand Down
64 changes: 56 additions & 8 deletions lib/llm/src/kv_router/prefill_router/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,10 @@ use crate::{
discovery::{ModelManager, WorkerSetTarget, WorkerSetTargetId},
kv_router::{RoutingHost, SelectionPolicySource},
protocols::common::{
extensions::{SESSION_AFFINITY_CONTEXT_KEY, SessionAffinityId},
extensions::{
AGENT_CONTEXT_CONTEXT_KEY, AgentContext, SESSION_AFFINITY_CONTEXT_KEY,
SessionAffinityId,
},
llm_backend::{LLMEngineOutput, PreprocessedRequest},
preprocessor::{BootstrapInfo, PrefillResult, TraceLink},
timing::{RequestPhase, RequestTracker},
Expand All @@ -53,6 +56,28 @@ enum PrefillLifecycleState {
Unavailable = 2,
}

fn prefill_affinity_id<T: Send + Sync + 'static>(
context: &Context<T>,
) -> Result<Option<SessionAffinityId>> {
let child = context
.get_optional::<SessionAffinityId>(SESSION_AFFINITY_CONTEXT_KEY)
.map_err(|message| anyhow::anyhow!("invalid session affinity context: {message}"))?;
let parent = context
.get_optional::<AgentContext>(AGENT_CONTEXT_CONTEXT_KEY)
.map_err(|message| anyhow::anyhow!("invalid agent context: {message}"))?
.and_then(|agent| {
agent
.parent_session_id
.as_deref()
.map(str::trim)
.filter(|parent| !parent.is_empty() && *parent != agent.session_id.as_str())
.map(str::to_owned)
});
Ok(parent
.map(SessionAffinityId::new)
.or_else(|| child.map(|id| id.as_ref().clone())))
}

impl TryFrom<u8> for PrefillLifecycleState {
type Error = u8;

Expand Down Expand Up @@ -375,9 +400,7 @@ impl
return next.generate(context.map(|_| req)).await;
}

let session_affinity = context
.get_optional::<SessionAffinityId>(SESSION_AFFINITY_CONTEXT_KEY)
.map_err(|message| anyhow::anyhow!("invalid session affinity context: {message}"))?;
let session_affinity = prefill_affinity_id(&context)?;

if self.conditional_disagg_policy.is_enabled() {
let conditional_request = context.map(|_| req);
Expand Down Expand Up @@ -468,10 +491,7 @@ impl
let tracker = prefill_req.tracker.clone();
let mut prefill_context = independent_prefill_context(prefill_req, &context)?;
if let Some(session_affinity) = session_affinity {
prefill_context.insert(
SESSION_AFFINITY_CONTEXT_KEY,
session_affinity.as_ref().clone(),
);
prefill_context.insert(SESSION_AFFINITY_CONTEXT_KEY, session_affinity);
}
let Some(binding) = self.binding.load_full() else {
return next.generate(context.map(|_| req)).await;
Expand Down Expand Up @@ -784,6 +804,34 @@ mod tests {

const MAX_ROOM: u64 = i64::MAX as u64;

#[test]
fn child_context_uses_parent_affinity_while_decode_retains_child() {
let (_, mut context) = SingleIn::new(query_only_request()).into_parts();
context.insert(
SESSION_AFFINITY_CONTEXT_KEY,
SessionAffinityId::new("child-a"),
);
context.insert(
AGENT_CONTEXT_CONTEXT_KEY,
serde_json::from_value::<AgentContext>(serde_json::json!({
"session_id": "child-a",
"parent_session_id": "parent-1"
}))
.unwrap(),
);
assert_eq!(
prefill_affinity_id(&context).unwrap().unwrap().as_str(),
"parent-1"
);
assert_eq!(
context
.get::<SessionAffinityId>(SESSION_AFFINITY_CONTEXT_KEY)
.unwrap()
.as_str(),
"child-a"
);
}

#[derive(Default)]
struct QueryOnlyDecodeHost {
requests: AtomicUsize,
Expand Down
42 changes: 40 additions & 2 deletions lib/llm/src/kv_router/publisher/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};

use anyhow::Result;
use tokio::sync::mpsc;
use tokio::sync::{mpsc, oneshot};
use tokio_util::sync::CancellationToken;

use dynamo_kv_router::indexer::{KvIndexerMetrics, LocalKvIndexer};
Expand Down Expand Up @@ -77,6 +77,10 @@ pub enum KvEventSourceConfig {
/// Model video-placeholder token id. `None` leaves video runs on the
/// engine's native hashing path.
video_token_id: Option<u32>,
/// Source rank when this socket is dedicated to one DP rank.
expected_rank: Option<u32>,
/// Expected source rank and negotiated heartbeat timeout.
liveness: Option<(u32, std::time::Duration)>,
},
}

Expand Down Expand Up @@ -135,6 +139,8 @@ impl KvEventSource {
topic,
image_token_id,
video_token_id,
expected_rank,
liveness,
} => {
let listener_handle =
component
Expand All @@ -151,6 +157,8 @@ impl KvEventSource {
next_event_id,
image_token_id,
video_token_id,
expected_rank,
liveness,
));
let listener_abort_handle = listener_handle.abort_handle();
let supervisor_handle =
Expand Down Expand Up @@ -207,6 +215,9 @@ pub struct KvEventPublisher {
tx: mpsc::UnboundedSender<Vec<PlacementEvent>>,
/// Internal monotonic event ID counter. Shared with the ZMQ listener if present.
next_event_id: Arc<AtomicU64>,
/// Local pipeline initialization, independent of remote source activity.
startup: Option<oneshot::Receiver<()>>,
processor_task: tokio::task::JoinHandle<()>,
}

impl KvEventPublisher {
Expand Down Expand Up @@ -363,7 +374,8 @@ impl KvEventPublisher {

tracing::info!("Using event plane for KV event publishing");
let endpoint_clone = endpoint.clone();
component.drt().runtime().secondary().spawn(async move {
let (startup_tx, startup) = oneshot::channel();
let processor_task = component.drt().runtime().secondary().spawn(async move {
let event_publisher =
match dynamo_runtime::transports::event_plane::EventPublisher::for_endpoint_id(
endpoint_clone.drt(),
Expand Down Expand Up @@ -449,6 +461,7 @@ impl KvEventPublisher {
}
};

let _ = startup_tx.send(());
start_event_processor(
EventPlanePublisher(event_publisher),
worker_id,
Expand Down Expand Up @@ -481,9 +494,34 @@ impl KvEventPublisher {
worker_id,
tx,
next_event_id,
startup: Some(startup),
processor_task,
})
}

/// Wait for local publication and discovery initialization, not a remote event.
pub async fn ready(&mut self) -> Result<()> {
if let Some(startup) = self.startup.take() {
tokio::select! {
result = startup => result.map_err(|_| anyhow::anyhow!("KV event publisher initialization failed"))?,
_ = self.cancellation_token.cancelled() => anyhow::bail!("KV event listener stopped during initialization"),
}
}
anyhow::ensure!(
!self.cancellation_token.is_cancelled() && !self.processor_task.is_finished(),
"KV event publisher stopped during initialization"
);
Ok(())
}

/// Observe fatal termination. Ordinary gaps and heartbeat expiry keep listening.
pub async fn terminated(&mut self) {
tokio::select! {
_ = &mut self.processor_task => {},
_ = self.cancellation_token.cancelled() => {},
}
}

pub fn publish(&self, event: KvCacheEvent) -> Result<(), mpsc::error::SendError<KvCacheEvent>> {
self.send_singleton(PlacementEvent::local_gpu(self.worker_id, event))
}
Expand Down
Loading
Loading