Repository navigation
feat(trtllm-sidecar): enable KV routing and subagent affinity through OpenEngine - #15087
Conversation
TensorRT-LLM is standardizing on OpenEngine (`openengine.v1`) as its gRPC surface. Migrate the sidecar off the bespoke `TrtllmService` protocol, and add disaggregated serving, which the old protocol had no representation for. The protos are vendored rather than pulled from the Buf Schema Registry: the BSR Rust SDK requires a per-developer token to build, and it forced the crate onto a tonic version island. `proto/README.md` records the upstream commit and a SHA-256 per file so the copies are verifiable. Disaggregation follows the protocol's own shape. A prefill worker marks its request `context_only` through `extra` and terminates with `PrefillReady` instead of `finished`; the decode worker replays the returned `KvSessionRef` verbatim. A `finished` on a prefill worker means the context phase ended without transmitting a handoff, which the frontend's prefill router would treat as an already-complete request and answer with an empty response, so it is rejected instead. A decode request that arrives without a handoff is the conditional-disaggregation bypass and runs the whole request locally. Routing targets travel as the protocol's `openengine-target-dp-rank` metadata rather than being rejected, so disaggregation composes with KV-aware routing. Decode cancellation is deferred to the terminal event, since the transceiver releases transferred KV when the engine finishes the request and not when the client goes away. `PrefillReady` is the final response for a context request, so its usage is authoritative and is forwarded, including the cached-token count the frontend relays to the decode leg. Engine errors keep the codes the router acts on: an overload sheds and migrates instead of surfacing as an opaque failure. `json.rs` moves from the vLLM sidecar into `dynamo-sidecar-common`, since both sidecars now need Struct/JSON conversion. Behaviour is unchanged. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…ode deferral RoutingHints.dp_rank names the decode worker and prefill_dp_rank the prefill one. Falling back from one to the other sent the prefill worker's rank to a decode engine whenever the decode router chose no rank, and made the prefill fallback dead. Select by mode, as the vLLM and SGLang sidecars already do. The decode cancellation deferral had no upper bound, so a cancelled decode request generated its whole budget with no consumer. Defer only until the first token proves the KV transfer landed, and only for a request that actually took a handoff. Also carry cached prompt tokens through to the terminal on both the aggregated and decode paths -- they were collected on PrefillReady and then dropped -- clamped to the prompt the client sent; restore the consolidated lib/sidecar/Dockerfile references the rebase reverted; and fix a mislabelled codec field name in the vLLM tests. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…ndoff The OpenEngine servicer suppresses `finished` on a context request only when it already sent a `PrefillReady` (streaming.py:174-184). A context phase that hits a stop condition on its one token, or is cancelled before transmission, deliberately gets the real terminal instead -- STOP, LENGTH or CANCELLED -- because there is no decode leg to answer the caller. The frontend supports exactly this (kv_router/prefill_router/admission.rs:95-108). Rejecting every prefill `finished` as a protocol error failed those requests. Surface the terminal instead, and carry the context phase's tokens on it: they are held back rather than streamed so the decode leg's replay cannot duplicate them, which leaves this terminal as the only place they can appear. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
… startup Findings from a deep review of the OpenEngine path, each confirmed against a live TensorRT-LLM 1.3.0rc26 server. A decode engine counts the blocks transferred into it as cache hits, which is a different quantity from the context phase's prefix-cache hit: the prefill leg reported 5 of 6 prompt tokens while the decode leg reported 6, so every disaggregated request told the client -- and the planner -- it had a ~100% cache hit rate. Once a leg takes a handoff, the context phase is now the only source of that count, including when it reported none. A context request that ends at its one-token budget never transmitted its KV. The servicer emits `Finished(LENGTH)` there, which the frontend's prefill router rejects with a misleading "missing disaggregated_params"; returning the single token instead would silently truncate the completion. It now fails naming the cause. Cancellation was deferred before the request was dispatched, where no KV can be stranded, so a caller that gave up waited out the dispatch anyway. `GetModelInfo` was a single un-retried call bounded by a hardcoded 30s, so an engine that binds its port before the model finishes loading crash-looped the worker and `--grpc-startup-deadline` did nothing. It now retries until that deadline and fails fast only on answers waiting cannot fix. `--context-length` returns as an override for an engine that reports a wrong window -- TensorRT-LLM falls back to `max_input_len` and reads as 1024 when `max_seq_len` is unset -- and the derived `max_tokens` is capped by `ModelInfo.max_output_tokens`. `ResourceExhausted` no longer maps to a migratable WorkerOverloaded in the shared helper, where it silently changed vLLM's behaviour and could burn the migration budget on a deterministic per-request rejection; the TensorRT-LLM Generate path maps it locally instead, where the code's meaning is known. The handoff codec becomes a serde mirror struct with `deny_unknown_fields`, so the decode worker requires exactly what the prefill worker wrote; a dropped `dp_rank` previously read as rank 0 and pulled KV from the wrong shard. That removes eight hand-written error branches. `ResponseState` becomes a typed phase so a prefill worker cannot reach a decode worker's accounting, the per-token path makes one pass over the deltas instead of three, and the 1965-line test file splits by surface. Validation: 41 unit + 1 executable test green; new tests are mutation-checked; aggregated and disaggregated end-to-end re-run against live TensorRT-LLM engines with identical tokens and usage. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
The sidecars no longer agree on a protobuf version -- vLLM generates against prost 0.14 while TensorRT-LLM and SGLang are on 0.13 -- and the two `prost_types::Struct` types are unrelated, so moving the codec into the shared crate left a second copy behind in vLLM. 130 lines of number, null and NaN handling existed twice, which is 130 lines where a fix can land in one sidecar and not the other. The codec is now emitted once per `prost_types` path by a macro, the same shape `error::status_to_dynamo_v14` already uses for this problem, and vLLM keeps only the labels that name its payload and peer in error messages -- 138 lines down to 27, with no call site changed. A test asserts both instantiations agree, so a divergence fails here rather than in one sidecar's own suite. `startup_deadline` was likewise implemented twice, identically; vLLM now uses the shared one. In the tests, four copies of the same nine-line engine construction collapse into one `engine_with` helper that takes what they actually varied. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…VERLOADED The OpenEngine servicer rejects `openengine-target-dp-rank` in the same `_trace_headers` branch that rejects `openengine-priority`, so every request carrying a routing rank failed with UNIMPLEMENTED -- a non-migratable 5xx. Mirror the priority guard and drop the metadata plumbing entirely; a session's KV affinity still travels in `KvSessionRef.dp_rank`, inside the request body. `ERROR_CODE_OVERLOADED` has one emitter, the 30-second consumer-stall watchdog, which fires when this sidecar stopped draining the stream. Mapping it to `WorkerOverloaded` re-dispatched to a second worker that stalls the same way, so map it to a plain engine error. `generate_status` scoped RESOURCE_EXHAUSTED to the Generate path, but no OpenEngine site emits that code; drop it. Also: apply the decode deferral to the dispatch itself, since `generate` sends before it awaits headers; hold whole `TokenInfo`s on a prefill leg so a handoff-less terminal keeps its logprobs; escape the peer's status text in the abort warning; and correct the docs, which claimed an engine started without `--max_seq_len` reports a `max_input_len` placeholder -- it leaves `max_context_length` unset. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…on string The local engine image self-reports `1.3.0rc26`, but it is a build of TensorRT-LLM main at 8bbaf66bd5; the published v1.3.0rc26 tree does not carry the OpenEngine servicer at all. Cite the commit instead. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…ine release TensorRT-LLM now ships the OpenEngine servicer in a public tag, so the docs no longer have to tell readers to build their own engine image. Name 1.3.0rc27.dev202609170000 as the floor in the README, the launch script, and the deployment manifest. The release ships the servicer but not the `openengine.v1` Python bindings, and the servicer imports them, so a stock image exits with "Failed to import OpenEngine support" instead of binding the listener. The instructions say so and keep the pinned pip step rather than implying the image is ready as-is. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Started without `--max_seq_len`, TensorRT-LLM answers `GetModelInfo` with `max_context_length` unset, and `wait_for_model_limits` then polls for the whole startup deadline before giving up. `deploy/agg.yaml` shipped exactly that: no `--max_seq_len` on the engine and no `--context-length` on the sidecar, so the worker took the default 1800 seconds to register and then rejected every request that omitted `max_tokens`. Measured against release:1.3.0rc27.dev202609170000 with a 20s deadline: 19.07s of polling, then `context_length=None`, then `InvalidRequest: TensorRT-LLM requires max_tokens`. The failure is quiet from outside -- the frontend still advertises a context window from the model card, so `/v1/models` looks healthy while every such request 500s. Pin 4096 on both sides, the way `launch/agg.sh` already does. Also correct `--grpc-startup-deadline` to `--grpc-startup-deadline-secs` in the four places that named it, one of them the remediation line of the error an operator hits at the end of that wait, and point the docs site at the published OpenEngine release like the README already does. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
GetModelInfo reads llm.args.max_seq_len, which the PyTorch path never writes back, so the servicer answers UNSET while the executor is already running at the window it resolved from the model config -- measured on release:1.3.0rc27.dev202609170000 with TinyLlama-1.1B: executor max_seq_len=2048, GetModelInfo UNSET. The pin is a workaround for that, not a property of the OpenEngine contract; say so where it is, so it goes away when the servicer reports the resolved window. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
jh-nv
left a comment
There was a problem hiding this comment.
One blocking issue inline.
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
No P0 or P1 found. This review adds one P2 and two P3 findings, and it measured one open P2 from another reviewer. Reviewed head 517e9ee4a5f4f49e95cdbf802eb5ddf1a020dae4.
- [P2]
--helpprints the value ofTRTLLM_SUBAGENT_AFFINITY_AUTH_KEY.lib/sidecar/trtllm/src/args.rs:29(inline, with a tested suggestion). - [P2] Open thread from another reviewer, measured here: with KV routing on, a single-rank engine still gets a synthetic
conversation_idon every anonymous request.lib/sidecar/trtllm/src/convert.rs:161(reply with a tested fix and a test that fails at this head). - [P3] No test covers the new
expected_rankfilter in the shared ZMQ listener.lib/llm/src/kv_router/publisher/zmq_listener.rs:150(inline, with a tested test). - [P3] The PR description does not match the code in five places (details below).
The five description mismatches.
- The Validation section names
child_context_uses_parent_affinity_while_decode_retains_child. No test has that name at this head, so that command runs 0 tests. The new test ischild_prefill_uses_parent_worker_while_decode_keeps_group_binding. - The test counts are older than the code. At this head:
dynamo-trtllm-sidecar --lib64 passed and 2 ignored (description: 59),dynamo-trtllm-mocker --test sidecar5 passed (description: 7),dynamo-sidecar-common --lib18 passed (description: 17),dynamo-llm --lib kv_router::publisher105 passed (description: 103). - "use the same private
TRTLLM_SUBAGENT_AFFINITY_AUTH_KEYin the prefill sidecar and OpenEngine server": the server does not read that variable. It takesinternal_request_auth_keyfrom its YAML configuration, aslib/sidecar/trtllm/README.mdsays. - "Negotiate heartbeat expiry": no TensorRT-LLM server advertises
kv_event_heartbeat_interval_ms. It is absent in1.4.0rc1.dev202610201007(a liveGetServerInfoand a source search) and on TensorRT-LLMmain. Solivenessis alwaysNoneagainst current servers, and the heartbeat clear and the clear on a sequence gap do not run. Please name the server change that will add it, or say that the path is for a future server. - The known limitation says that the last load snapshot stays stale after a failed poll. That is true for 30 seconds only. Then the worker withdraws, and the sidecar exits with
Routing load remained unavailable for 30 seconds; withdrawing worker(measured below). Please say so.
Earlier review threads, checked at this head.
| thread | status at this head | evidence |
|---|---|---|
dynamo-llm default features (Cargo.toml) |
fixed | cargo tree -p dynamo-trtllm-sidecar -f "{p} [{f}]" shows dynamo-llm with no features. Control: -p dynamo-llm shows [block-manager,default]. |
startup error wrapper (lib.rs) |
fixed in the code | bootstrap errors go through SidecarStartupError::from. The Python ValueError path was not run. |
RESOURCE_EXHAUSTED on metadata RPCs (headless.rs) |
fixed | A deliberate break that maps it back to the generic error fails sustained_metadata_outage_withdraws_worker and leader_monitor_rejects_restarted_native_engine. |
synthetic conversation_id (convert.rs) |
not fixed for KV routing | Still open in the newer thread on convert.rs:161. See the reply there. |
clear on gap for every ZMQ source (zmq_listener.rs) |
fixed | A deliberate break that clears on every gap fails test_start_zmq_listener_pushes_to_channel::case_1_legacy. |
rank 0 rejected without targeting (convert.rs) |
fixed | A deliberate break that restores the old check of both ranks fails prefill_ready_is_the_terminal_handoff and decode_request_replays_the_prefill_session. |
What this review measured.
Rust tests in a container on Linux x86_64, toolchain 1.96.1, --locked, with the merge base 77e76c1bf3 as the control:
| command | merge base | head |
|---|---|---|
cargo test -p dynamo-trtllm-sidecar --lib |
51 passed, 2 ignored | 64 passed, 2 ignored |
cargo test -p dynamo-trtllm-sidecar --test executable |
2 passed | 2 passed |
cargo test -p dynamo-trtllm-mocker --test sidecar |
5 passed | 5 passed |
cargo test -p dynamo-sidecar-common --lib |
18 passed | 18 passed |
cargo test -p dynamo-llm --lib kv_router |
431 passed | 434 passed |
cargo test -p dynamo-backend-common --lib |
161 passed | 161 passed |
cargo test -p dynamo-vllm-sidecar |
135 passed | 135 passed |
cargo test -p dynamo-sglang-sidecar |
126 passed | 126 passed |
cargo test -p dynamo-sidecar-testkit |
65 passed | 65 passed |
These commands pass at head:
cargo check -p dynamo-trtllm-sidecar --all-targetscargo check --manifest-path lib/bindings/python/Cargo.tomlcargo clippy --all-targets -- -D warningson the TensorRT-LLM, vLLM and SGLang sidecars, the common and testkit crates,dynamo-backend-commonanddynamo-trtllm-mockercargo fmt --all --check
Each of these deliberate breaks of the new code fails at least one new test. Each file was restored by checksum.
- Remove the
livenesscondition on the gap clear. - Change
request_typein the signed payload. - Map
RESOURCE_EXHAUSTEDto the generic error. - Never fail the load watchdog.
- Use the sibling group key for child prefill.
- Restore the old check of both ranks.
Removing the expected_rank filter fails no test (inline P3).
Signed affinity: the head test vector (sha256=e98db0ec…) passes validate_parent_affinity in TensorRT-LLM 1.4.0rc1.dev202610201007. A changed signature, a different key, generation_only, and a server without a key each raise ValueError. The signature covers purpose, model, request_id, conversation_id, the parent ID and request_type, not the target rank. An empty key fails at startup. If the server does not advertise trtllm_supports_subagent_affinity, a configured key also fails startup. By the code, a one-byte key is accepted on both sides.
End to end on one RTX 6000 Ada: two trtllm-serve --grpc --grpc-protocol openengine --openengine-enable-load-metrics servers (1.4.0rc1.dev202610201007, Qwen3-0.6B, ZMQ KV events), a Dynamo frontend with --router-mode kv, and two head sidecars.
- KV events: the router applied
storedevents. Each repeat of a prefix of 1,063 to 1,097 tokens (12 of 12) went to the worker that held it. These requests showedoverlap_blocks=33or34and about 1,060cached_tokens. With merge-base sidecars against the same servers, all 18 requests showedoverlap_blocks=0. - Load: the per-rank
gpu_cache_usage_percentof the busy worker went from 0 to 0.0123 during a long request, and the frontend subscribed to bothkv_metricsstreams. - Engine stop: 30 seconds after one server stopped, its sidecar exited with
Routing load remained unavailable for 30 seconds; withdrawing worker, and the router loggedRemoving worker. - A request started after an idle period. Then one
GetLoadpoll of the sidecar failed withKV occupancy sample is stale while requests are active. During a later run of long requests, directGetLoadcalls once per second all returned OK, and the sidecar did not withdraw.
The Pre Merge job rust-tests (.) on this head ran the new tests::kv_routing tests (71 sidecar library tests on the merge ref).
This server has one attention-DP rank. So this review did not run rank targeting with more than one rank, the follower relay, or signed affinity on a live server. Only the unit tests and the source reading cover them.
A merge of this head with main at 7929f5370eef7f15095b16f0dd41012b920c334c has no conflicts. The checks ran on a merge with the earlier tip ebfcce7891a760950ca7f88b700d584d59010fba, and main changed no file of this PR between the two. On that merge, cargo check --all-targets on the changed crates, cargo check of the Python bindings, dynamo-trtllm-sidecar --lib (69 passed, 2 ignored) and dynamo-llm --lib kv_router (444 passed) pass.
Open items:
- Shared code that changes behavior without the TensorRT-LLM sidecar:
lib/backend-common/src/publisher.rsnow setsexpected_rankfor every ZMQ source, so a batch for another rank or with no rank is dropped.routing_host.rsnow places a child's prefill on the parent's own binding for every backend with session affinity.
… sensitive information' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: Tanmay Verma <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
|
/ok to test f8a3646 |
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
No P0 or P1 found. One P3 from the earlier review on 2026-10-21 is still open: the PR description. Reviewed head f8a3646cf9013af8dd84daec470162e12f4c7ad3, the three commits after 517e9ee4a5.
- Fixed:
--helpno longer printsTRTLLM_SUBAGENT_AFFINITY_AUTH_KEY(args.rs). - Fixed: a single-rank engine no longer gets a rank hint or a synthetic
conversation_id(engine.rs). - Fixed: a test now covers the
expected_rankfilter (publisher/tests.rs). - [P3] Still open: the PR description is unchanged. The five mismatches from the earlier review remain. For example, the Validation section still names
child_context_uses_parent_affinity_while_decode_retains_child, and no test has that name.
What this review measured at the new head.
Tests in a Linux container, toolchain 1.96.1, --locked:
| command | result |
|---|---|
cargo test -p dynamo-trtllm-sidecar --lib |
65 passed, 2 ignored |
cargo test -p dynamo-trtllm-sidecar --test executable |
2 passed |
cargo test -p dynamo-llm --lib kv_router |
434 passed |
cargo clippy -p dynamo-trtllm-sidecar -p dynamo-llm --all-targets -- -D warnings |
pass |
cargo fmt --all --check |
pass |
Each fix was reverted in turn, and each new test then failed:
| change | failing test |
|---|---|
engine.rs of 517e9ee4a5 |
single_rank_kv_routing_preserves_only_real_conversation_ids (kv_routing.rs:164) |
args.rs of 517e9ee4a5 |
executable_exposes_native_grpc_configuration (tests/executable.rs:36) |
expected_rank filter removed |
both cases of test_start_zmq_listener_pushes_to_channel (publisher/tests.rs:1521) |
The CodeQL commit bbb8fa440d removes session={session_id:?} from an assert message in convert/request_tests.rs. The value is a fixed test string, so the alert was a false positive. The change touches only the failure text of a test and changes no behavior.
The new commits need no new GPU run. For a single-rank engine, the sidecar now sends no rank hint and no synthetic ID. The merge-base sidecar sent neither, and in the earlier end-to-end run it served all 18 requests on the same TensorRT-LLM server.
A merge of this head with main at b1e4b765d3779cc821267a2d7a59106220441e32 has no conflicts.
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
Reviewed head f8a3646cf9013af8dd84daec470162e12f4c7ad3. No P0 or P1 found.
The three new commits fix both P2 findings from the earlier review and add the missing rank-filter test. If its fix is reverted, each new test fails. At this head, the sidecar library tests, the executable tests, the kv_router tests, clippy and fmt pass. A merge with main at b1e4b765d3 has no conflicts.
Open item: [P3] the PR description still does not match the code in five places, for example a test name that does not exist.
Not run: rank targeting with more than one rank, the follower relay, signed affinity on a live server, and the Python ValueError path of the bindings.
✅ Dynamo PR CI passed — run 39664698826 (attempt 1) on
|
… OpenEngine (ai-dynamo#15087) Signed-off-by: tanmayv25 <tanmay2592@gmail.com> Signed-off-by: Tanmay Verma <tanmay2592@gmail.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Main added rank filtering, heartbeat liveness and publisher ready()/terminated() for the TRT-LLM node-local sidecar (#15087). Keep them alongside replay recovery: - KvEventSourceConfig::Zmq and start_zmq_listener carry expected_rank, liveness and recovery; recovery still hands off to the recovery listener. - The live listener applies main's rank filter, heartbeat and gap handling, then the shared normalize_batch, and sends through PublisherInput. - The processor JoinHandle stays in processor_handle. ready() and terminated() read it through Mutex::get_mut, and terminated() takes it once the task ends, so stop_and_wait finds an empty slot. - The TRT-LLM headless source sets recovery: None. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
The sidecar needs OpenEngine routing discovery and the node-local KV relay from #15087, which merged after the 1.6.x releases were cut. The frontend is pinned to the same build because only that pairing was tested. Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
Fixes #16453. Part of #10841.
Summary
mainwithout rewriting the existing PR branch history.Deployment requirements
nvcr.io/nvidia/tensorrt-llm/release:1.4.0a0.dev202610180820for TensorRT-LLM. This released nightly includes the native OpenEngine KV-event and routing-load support from NVIDIA/TensorRT-LLM#19440, including KVCM2.TRTLLM_SUBAGENT_AFFINITY_AUTH_KEYin the prefill sidecar and OpenEngine server. The sidecar checks the server capability before serving traffic.Validation
cargo test -p dynamo-trtllm-sidecar --lib: 59 passed, 2 existing ignored.cargo test -p dynamo-trtllm-sidecar --test executable: 2 passed.cargo test -p dynamo-trtllm-mocker --test sidecar: 7 passed;cargo test -p dynamo-sidecar-common --lib: 17 passed.cargo test -p dynamo-llm --lib kv_router::publisher: 103 passed.cargo test -p dynamo-llm --lib child_context_uses_parent_affinity_while_decode_retains_child: passed.cargo check -p dynamo-trtllm-sidecar --all-targetsandcargo check --manifest-path lib/bindings/python/Cargo.toml: passed.cargo clippyon the affected Rust crates with--all-targets -- -D warnings,cargo fmt --all --check, docs lint, and pre-commit on changed files: passed.Known limitation
The last published load snapshot remains stale if polling fails. Heartbeat expiry clears KV residency but does not quarantine inference traffic.
Summary by CodeRabbit