Repository navigation
docs(trtllm): add KV routing launch and DGD examples - #16561
Conversation
TensorRT-LLM is standardizing on OpenEngine (`openengine.v1`) as its gRPC surface. Migrate the sidecar off the bespoke `TrtllmService` protocol, and add disaggregated serving, which the old protocol had no representation for. The protos are vendored rather than pulled from the Buf Schema Registry: the BSR Rust SDK requires a per-developer token to build, and it forced the crate onto a tonic version island. `proto/README.md` records the upstream commit and a SHA-256 per file so the copies are verifiable. Disaggregation follows the protocol's own shape. A prefill worker marks its request `context_only` through `extra` and terminates with `PrefillReady` instead of `finished`; the decode worker replays the returned `KvSessionRef` verbatim. A `finished` on a prefill worker means the context phase ended without transmitting a handoff, which the frontend's prefill router would treat as an already-complete request and answer with an empty response, so it is rejected instead. A decode request that arrives without a handoff is the conditional-disaggregation bypass and runs the whole request locally. Routing targets travel as the protocol's `openengine-target-dp-rank` metadata rather than being rejected, so disaggregation composes with KV-aware routing. Decode cancellation is deferred to the terminal event, since the transceiver releases transferred KV when the engine finishes the request and not when the client goes away. `PrefillReady` is the final response for a context request, so its usage is authoritative and is forwarded, including the cached-token count the frontend relays to the decode leg. Engine errors keep the codes the router acts on: an overload sheds and migrates instead of surfacing as an opaque failure. `json.rs` moves from the vLLM sidecar into `dynamo-sidecar-common`, since both sidecars now need Struct/JSON conversion. Behaviour is unchanged. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…ode deferral RoutingHints.dp_rank names the decode worker and prefill_dp_rank the prefill one. Falling back from one to the other sent the prefill worker's rank to a decode engine whenever the decode router chose no rank, and made the prefill fallback dead. Select by mode, as the vLLM and SGLang sidecars already do. The decode cancellation deferral had no upper bound, so a cancelled decode request generated its whole budget with no consumer. Defer only until the first token proves the KV transfer landed, and only for a request that actually took a handoff. Also carry cached prompt tokens through to the terminal on both the aggregated and decode paths -- they were collected on PrefillReady and then dropped -- clamped to the prompt the client sent; restore the consolidated lib/sidecar/Dockerfile references the rebase reverted; and fix a mislabelled codec field name in the vLLM tests. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…ndoff The OpenEngine servicer suppresses `finished` on a context request only when it already sent a `PrefillReady` (streaming.py:174-184). A context phase that hits a stop condition on its one token, or is cancelled before transmission, deliberately gets the real terminal instead -- STOP, LENGTH or CANCELLED -- because there is no decode leg to answer the caller. The frontend supports exactly this (kv_router/prefill_router/admission.rs:95-108). Rejecting every prefill `finished` as a protocol error failed those requests. Surface the terminal instead, and carry the context phase's tokens on it: they are held back rather than streamed so the decode leg's replay cannot duplicate them, which leaves this terminal as the only place they can appear. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
… startup Findings from a deep review of the OpenEngine path, each confirmed against a live TensorRT-LLM 1.3.0rc26 server. A decode engine counts the blocks transferred into it as cache hits, which is a different quantity from the context phase's prefix-cache hit: the prefill leg reported 5 of 6 prompt tokens while the decode leg reported 6, so every disaggregated request told the client -- and the planner -- it had a ~100% cache hit rate. Once a leg takes a handoff, the context phase is now the only source of that count, including when it reported none. A context request that ends at its one-token budget never transmitted its KV. The servicer emits `Finished(LENGTH)` there, which the frontend's prefill router rejects with a misleading "missing disaggregated_params"; returning the single token instead would silently truncate the completion. It now fails naming the cause. Cancellation was deferred before the request was dispatched, where no KV can be stranded, so a caller that gave up waited out the dispatch anyway. `GetModelInfo` was a single un-retried call bounded by a hardcoded 30s, so an engine that binds its port before the model finishes loading crash-looped the worker and `--grpc-startup-deadline` did nothing. It now retries until that deadline and fails fast only on answers waiting cannot fix. `--context-length` returns as an override for an engine that reports a wrong window -- TensorRT-LLM falls back to `max_input_len` and reads as 1024 when `max_seq_len` is unset -- and the derived `max_tokens` is capped by `ModelInfo.max_output_tokens`. `ResourceExhausted` no longer maps to a migratable WorkerOverloaded in the shared helper, where it silently changed vLLM's behaviour and could burn the migration budget on a deterministic per-request rejection; the TensorRT-LLM Generate path maps it locally instead, where the code's meaning is known. The handoff codec becomes a serde mirror struct with `deny_unknown_fields`, so the decode worker requires exactly what the prefill worker wrote; a dropped `dp_rank` previously read as rank 0 and pulled KV from the wrong shard. That removes eight hand-written error branches. `ResponseState` becomes a typed phase so a prefill worker cannot reach a decode worker's accounting, the per-token path makes one pass over the deltas instead of three, and the 1965-line test file splits by surface. Validation: 41 unit + 1 executable test green; new tests are mutation-checked; aggregated and disaggregated end-to-end re-run against live TensorRT-LLM engines with identical tokens and usage. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
The sidecars no longer agree on a protobuf version -- vLLM generates against prost 0.14 while TensorRT-LLM and SGLang are on 0.13 -- and the two `prost_types::Struct` types are unrelated, so moving the codec into the shared crate left a second copy behind in vLLM. 130 lines of number, null and NaN handling existed twice, which is 130 lines where a fix can land in one sidecar and not the other. The codec is now emitted once per `prost_types` path by a macro, the same shape `error::status_to_dynamo_v14` already uses for this problem, and vLLM keeps only the labels that name its payload and peer in error messages -- 138 lines down to 27, with no call site changed. A test asserts both instantiations agree, so a divergence fails here rather than in one sidecar's own suite. `startup_deadline` was likewise implemented twice, identically; vLLM now uses the shared one. In the tests, four copies of the same nine-line engine construction collapse into one `engine_with` helper that takes what they actually varied. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…VERLOADED The OpenEngine servicer rejects `openengine-target-dp-rank` in the same `_trace_headers` branch that rejects `openengine-priority`, so every request carrying a routing rank failed with UNIMPLEMENTED -- a non-migratable 5xx. Mirror the priority guard and drop the metadata plumbing entirely; a session's KV affinity still travels in `KvSessionRef.dp_rank`, inside the request body. `ERROR_CODE_OVERLOADED` has one emitter, the 30-second consumer-stall watchdog, which fires when this sidecar stopped draining the stream. Mapping it to `WorkerOverloaded` re-dispatched to a second worker that stalls the same way, so map it to a plain engine error. `generate_status` scoped RESOURCE_EXHAUSTED to the Generate path, but no OpenEngine site emits that code; drop it. Also: apply the decode deferral to the dispatch itself, since `generate` sends before it awaits headers; hold whole `TokenInfo`s on a prefill leg so a handoff-less terminal keeps its logprobs; escape the peer's status text in the abort warning; and correct the docs, which claimed an engine started without `--max_seq_len` reports a `max_input_len` placeholder -- it leaves `max_context_length` unset. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…on string The local engine image self-reports `1.3.0rc26`, but it is a build of TensorRT-LLM main at 8bbaf66bd5; the published v1.3.0rc26 tree does not carry the OpenEngine servicer at all. Cite the commit instead. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…ine release TensorRT-LLM now ships the OpenEngine servicer in a public tag, so the docs no longer have to tell readers to build their own engine image. Name 1.3.0rc27.dev202609170000 as the floor in the README, the launch script, and the deployment manifest. The release ships the servicer but not the `openengine.v1` Python bindings, and the servicer imports them, so a stock image exits with "Failed to import OpenEngine support" instead of binding the listener. The instructions say so and keep the pinned pip step rather than implying the image is ready as-is. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Started without `--max_seq_len`, TensorRT-LLM answers `GetModelInfo` with `max_context_length` unset, and `wait_for_model_limits` then polls for the whole startup deadline before giving up. `deploy/agg.yaml` shipped exactly that: no `--max_seq_len` on the engine and no `--context-length` on the sidecar, so the worker took the default 1800 seconds to register and then rejected every request that omitted `max_tokens`. Measured against release:1.3.0rc27.dev202609170000 with a 20s deadline: 19.07s of polling, then `context_length=None`, then `InvalidRequest: TensorRT-LLM requires max_tokens`. The failure is quiet from outside -- the frontend still advertises a context window from the model card, so `/v1/models` looks healthy while every such request 500s. Pin 4096 on both sides, the way `launch/agg.sh` already does. Also correct `--grpc-startup-deadline` to `--grpc-startup-deadline-secs` in the four places that named it, one of them the remediation line of the error an operator hits at the end of that wait, and point the docs site at the published OpenEngine release like the README already does. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
GetModelInfo reads llm.args.max_seq_len, which the PyTorch path never writes back, so the servicer answers UNSET while the executor is already running at the window it resolved from the model config -- measured on release:1.3.0rc27.dev202609170000 with TinyLlama-1.1B: executor max_seq_len=2048, GetModelInfo UNSET. The pin is a workaround for that, not a property of the OpenEngine contract; say so where it is, so it goes away when the servicer reports the resolved window. Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
krishung5
left a comment
There was a problem hiding this comment.
Do we plan to add CI coverage for these launch scripts?
nv-yna
left a comment
There was a problem hiding this comment.
Reviewed c2a839e. The helper's 15 self-tests, shell syntax checks, YAML parsing, and six launcher checks with inert process substitutes passed. The launcher checks verified memory/event config preservation and temporary-file cleanup and confirmed the GPU-selection concern noted inline. No GPU inference or multi-node validation was rerun in this review.
jthomson04
left a comment
There was a problem hiding this comment.
Reviewed c2a839e. The KV event port comment matters most if these scripts get CI coverage. The inherited CUDA_VISIBLE_DEVICES issue is already tracked in nv-yna's thread, so I haven't repeated it.
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
|
/ok to test 7a007a7 |
Yes. I will add that soon. |
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
No P0, P1 or P2 found. This review read head 7a007a7f. The PR merged during the review as squash 6d80985a on main b5313e78, and the 6 files in that commit match 7a007a7f byte for byte.
- [P3]
lib/sidecar/trtllm/launch/disagg_kv_router.sh:109: a missing managed port (DYN_MANAGED_PORTS=1) for worker 3 or 4 fails only after the frontend and earlier workers started. Inline, with a tested diff. - [P3] PR description: the Summary does not name the change to
build_trtllm_override_args_with_meminexamples/common/gpu_utils.sh. 12 launchers inexamples/backends/trtllm/launch/, 2 sidecar launchers andtests/utils/gpu_args.pycall that function. The squash commitdocs(trtllm): add KV routing launch and DGD examples (#16561)does not name it either. Please add one Summary line for the helper change. Also remove "review and merge that implementation PR first", because #15087 merged on 2026-10-22. - Open: CI coverage for the two new launchers. krishung5 asked about it, and the author plans to add it. A note for that work: the TensorRT-LLM image that CI pins on
main(release:1.4.0rc0) has no--openengine-enable-load-metricsintensorrt_llm/commands/serve.py, andv1.4.0rc1has it. A test needs that pin to move, or a skip guard like_trtllm_serves_openengine(). - Open: no CI job ran this diff before the merge. The
changed-filesjob reportedcore: []andtrtllm: [], so run 40413423615 skipped every TensorRT-LLM job and ran only the sidecar image build. No job runsexamples/common/gpu_utils.sh --self-test. - Not run here:
disagg_kv_router.shon GPUs, and the two DGDs on a cluster.
What this review ran, with controls.
The helper build_trtllm_override_args_with_mem ran with 31 inputs at base d66fc3f4 and at the head:
- For every input that a caller on
mainpasses, the output and the exit code match base. These inputs are no cap, a token cap, a byte cap, and both caps. They also include the merge JSON fromexamples/backends/trtllm/launch/agg.sh,disagg_same_gpu.sh, and the sidecaragg.shanddisagg.sh. Every setter intests/passesstr(int(...)). - With the JSON of the new launchers and a cap, base prints two
kv_cache_configkeys. The head prints one merged object. - Input that is not an integer or not a JSON object now stops the helper with a Python error, where base printed it through. Examples are
4096.0,1e9,abc,{"kv_cache_config": null}, a JSON list, text that is not JSON, and only spaces. All 16 launchers that call the helper useset -e, so they stop and do not start an engine without its cap. - Self-test: base 13/13, head 15/15, and the head merged with
maine3b7b76215/15, under bash 5.3 and 3.2. The merge has no conflict. Two deliberate breaks make the two new cases fail (13 passed, 2 failed). One break puts the old function back. The other applies the cap before the JSON of the caller.
The launchers:
bash -nandshellcheck0.10.0 report nothing on both scripts. A planted SC2086 is caught.- Stub cases on Linux: each worker gets its own gRPC, KV-event and system port, with default ports and with managed ports. Each KV publisher uses
tcp://127.0.0.1:<port>. - Inherited
CUDA_VISIBLE_DEVICESentries, numbers and UUIDs, map to the workers in order. A short list fails before any child starts. --configand--extra_llm_api_optionsstop both launchers before the bindings step. No temporary config file stays after exit.
A GPU run on one RTX 6000 Ada:
agg_kv_router.shfrom the head ran in a container fromnvcr.io/nvidia/tensorrt-llm/release@sha256:b8c862fea7c4306043dd2dbda06175af0fc8af4f336435597c07ec92f6fff3bd. The frontend venv anddynamo-trtllm-sidecarcame from the20261023-75d0686nightly images that the DGDs pin.- The run put both workers on GPU 0 and served the model
Qwen/Qwen3-0.6B. It set_PROFILE_OVERRIDE_TRTLLM_MAX_TOTAL_TOKENS=4096,DYN_DISCOVERY_BACKEND=file,PRTE_ALLOW_RUN_AS_ROOT=1andPRTE_ALLOW_RUN_AS_ROOT_CONFIRM=1. - Both engines logged
max_tokens 4096 is providedand started with the merged config. Both sidecars loggedevents_enabled=true load_enabled=true. - Streamed requests with
nvext.extra_fieldsset to["worker_id", "timing"]: prompt A went to one worker and prompt B to the other, every time.kv_hit_ratewas 0.0 on the first request of each prompt, and 0.954 and 0.947 on each of 3 repeats. Thestoredevent counter of each worker went from none to 1. - Listeners:
127.0.0.1:5557and127.0.0.1:5558for KV events, and gRPC ports 30001 and 30011 on loopback only. - The startup probe script from the DGDs exits 0 against ports 30001 and 30011, and 1 against the closed port 30002.
The manifests:
- Both DGDs pass the v1beta1 schema of the CRD on
main, with unknown fields rejected. The existingagg.yamlanddisagg.yamlalso pass. - Six deliberate breaks fail: an unknown field, a bad
type, a badruntimeVersionOverride, an unknown container field, a badbackendFramework, and a number forrestartPolicy. The CEL rules that apply to these DGDs hold by reading. - Each pinned digest is the digest of its tag on nvcr.io, as a multi-arch index with amd64 and arm64. A bogus tag and a bogus digest return 404.
- The ports, the ZMQ endpoints and the sidecar flags (
--grpc-endpoint,--model-path,--context-length,--disaggregation-mode) match the launchers andmain. Nightly75d0686contains #15087 (f7deb693dd).
The head keeps the fixes from the 12 resolved threads. This review probed six of them: the cap merge, the loopback KV bind, inherited GPUs, managed KV ports, the --config order, and temporary file cleanup.
| kv_var="TRTLLM_WORKER${worker}_KV_EVENT_PORT" | ||
| system_var="DYN_SYSTEM_PORT${worker}" | ||
| gpu="${!gpu_var}" | ||
| grpc_port=$(dyn_port DYN_ENGINE_GRPC_PORT "$worker" "${!grpc_var:-$((30001 + 10 * (worker - 1)))}") |
There was a problem hiding this comment.
[P3] A missing managed port for worker 3 or 4 fails only after the frontend and earlier workers started. lib/sidecar/trtllm/launch/disagg_kv_router.sh:109. Managed means DYN_MANAGED_PORTS=1, which the tests set. Please resolve all four workers' ports before the frontend starts, as agg_kv_router.sh does. The PR is merged, so here is a tested diff for a follow-up:
--- a/lib/sidecar/trtllm/launch/disagg_kv_router.sh
+++ b/lib/sidecar/trtllm/launch/disagg_kv_router.sh
@@ -96,18 +96,24 @@
HTTP_PORT="${DYN_HTTP_PORT:-8000}"
print_launch_banner "TensorRT-LLM OpenEngine KV-aware sidecars (2 prefill, 2 decode)" "$MODEL" "$HTTP_PORT" \
"KV transfer: ${TRTLLM_CACHE_TRANSCEIVER_BACKEND} cache transceiver"
+# Resolve every worker's ports before starting any process.
+GRPC_PORTS=() KV_PORTS=()
+for worker in 1 2 3 4; do
+ grpc_var="TRTLLM_WORKER${worker}_GRPC_PORT"
+ kv_var="TRTLLM_WORKER${worker}_KV_EVENT_PORT"
+ GRPC_PORTS[worker]=$(dyn_port DYN_ENGINE_GRPC_PORT "$worker" "${!grpc_var:-$((30001 + 10 * (worker - 1)))}")
+ KV_PORTS[worker]=$(dyn_port DYN_VLLM_KV_EVENT_PORT "$worker" "${!kv_var:-$((5556 + worker))}")
+done
python3 -m dynamo.frontend --router-mode kv &
for worker in 1 2 3 4; do
role=prefill
if (( worker > 2 )); then role=decode; fi
gpu_var="TRTLLM_WORKER${worker}_GPU"
- grpc_var="TRTLLM_WORKER${worker}_GRPC_PORT"
- kv_var="TRTLLM_WORKER${worker}_KV_EVENT_PORT"
system_var="DYN_SYSTEM_PORT${worker}"
gpu="${!gpu_var}"
- grpc_port=$(dyn_port DYN_ENGINE_GRPC_PORT "$worker" "${!grpc_var:-$((30001 + 10 * (worker - 1)))}")
- kv_port=$(dyn_port DYN_VLLM_KV_EVENT_PORT "$worker" "${!kv_var:-$((5556 + worker))}")
+ grpc_port="${GRPC_PORTS[worker]}"
+ kv_port="${KV_PORTS[worker]}"
system_port="${!system_var:-$((8080 + worker))}"
config=$(mktemp "${TMPDIR:-/tmp}/dynamo-trtllm-disagg-kv.XXXXXX.yaml")
TRTLLM_CONFIG_FILES+=("$config")Measured on the head and on the fix, with stub commands.
The launcher ran on Linux (bash 5) with stub frontend, engine and sidecar commands. Each stub writes its arguments and environment to a log, then exits after a few seconds. The fix passed bash -n, shellcheck 0.10.0 and these cases:
| case | head: exit, children started | with the fix |
|---|---|---|
DYN_ENGINE_GRPC_PORT3 missing |
1, 5 | 1, 0 |
DYN_VLLM_KV_EVENT_PORT4 missing |
1, 7 | 1, 0 |
| all managed ports set | 0, 9 | 0, 9, same arguments |
| default ports | 0, 9 | 0, 9, same arguments |
inherited GPUs 4,5,6,7 |
0, 9 | 0, 9, same arguments |
| byte cap | 0, 9 | 0, 9, same arguments |
| only 2 GPUs visible | 1, 0 | 1, 0 |
Both versions print Missing or invalid managed port DYN_ENGINE_GRPC_PORT3: <unset>. agg_kv_router.sh with DYN_ENGINE_GRPC_PORT2 missing exits 1 before it starts any child.
Part of #16453. Stacked on #15087; review and merge that implementation PR first.
Summary
Architecture
The disaggregated example places two prefill and two decode worker pods across GPU nodes. Each worker pod pairs a Dynamo sidecar with a TensorRT-LLM
maincontainer. Node placement below is illustrative; the DGD does not require these roles to occupy separate nodes.flowchart LR client[Client] --> frontend subgraph frontend_node[Frontend node] frontend[Dynamo frontend<br/>KV router] end subgraph prefill_node[GPU node A - prefill worker pods x2] prefill_sidecar[Dynamo sidecar] prefill_engine[TensorRT-LLM main container<br/>OpenEngine + KVCM2] prefill_sidecar -->|loopback OpenEngine gRPC| prefill_engine prefill_engine -.->|ZMQ KV events| prefill_sidecar end subgraph decode_node[GPU node B - decode worker pods x2] decode_sidecar[Dynamo sidecar] decode_engine[TensorRT-LLM main container<br/>OpenEngine + KVCM2] decode_sidecar -->|loopback OpenEngine gRPC| decode_engine decode_engine -.->|ZMQ KV events| decode_sidecar end frontend -->|prefill request| prefill_sidecar prefill_sidecar -->|KV handoff reference| frontend frontend -->|decode request| decode_sidecar prefill_engine -->|NIXL KV transfer| decode_engine prefill_sidecar -.->|KV events + load for routing| frontend decode_sidecar -.->|KV events + load for routing| frontendThe aggregate example uses the same sidecar/engine pod layout with two aggregate workers; it has no prefill-to-decode NIXL transfer.
Deployment requirements
20261023-75d0686by digest. This revision includes the merged KV-routing implementation from feat(trtllm-sidecar): enable KV routing and subagent affinity through OpenEngine #15087; no custom Dynamo image build is required.nvcr.io/nvidia/tensorrt-llm/release:1.4.0a0.dev202610180820to its verified multi-architecture OCI digest. No GHCR pull secret is required for the TensorRT-LLM engine image.Validation
20261023-75d0686images and the pinned TensorRT-LLM release: aggregate DGD 12/12 HTTP requests passed; disaggregate DGD 24/24 passed. Streaming completed successfully in both modes.UNAVAILABLEresponses reporting stale occupancy recovered. There were no HTTP failures or container restarts. All test resources were removed after logs and metrics were saved.Summary by CodeRabbit