Skip to content

docs(trtllm): add KV routing launch and DGD examples - #16561

Merged
tanmayv25 merged 43 commits into
mainfrom
docs/trtllm-kv-routing-examples
Oct 26, 2026
Merged

tanmayv25 merged 43 commits into
mainfrom
docs/trtllm-kv-routing-examples

Conversation

@tanmayv25

@tanmayv25 tanmayv25 commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Part of #16453. Stacked on #15087; review and merge that implementation PR first.

Summary

  • Add aggregate (two workers) and disaggregate (two prefill and two decode workers) KV-router launch scripts and DynamoGraphDeployment examples for TensorRT-LLM OpenEngine.
  • Configure KVCM2 ZMQ events, routing load metrics, separate event ports for prefill and decode, and the released KV-enabled TensorRT-LLM image.
  • Keep KV routing in dedicated examples; the existing aggregate DGD stays unchanged.

Architecture

The disaggregated example places two prefill and two decode worker pods across GPU nodes. Each worker pod pairs a Dynamo sidecar with a TensorRT-LLM main container. Node placement below is illustrative; the DGD does not require these roles to occupy separate nodes.

flowchart LR
    client[Client] --> frontend
    subgraph frontend_node[Frontend node]
        frontend[Dynamo frontend<br/>KV router]
    end
    subgraph prefill_node[GPU node A - prefill worker pods x2]
        prefill_sidecar[Dynamo sidecar]
        prefill_engine[TensorRT-LLM main container<br/>OpenEngine + KVCM2]
        prefill_sidecar -->|loopback OpenEngine gRPC| prefill_engine
        prefill_engine -.->|ZMQ KV events| prefill_sidecar
    end
    subgraph decode_node[GPU node B - decode worker pods x2]
        decode_sidecar[Dynamo sidecar]
        decode_engine[TensorRT-LLM main container<br/>OpenEngine + KVCM2]
        decode_sidecar -->|loopback OpenEngine gRPC| decode_engine
        decode_engine -.->|ZMQ KV events| decode_sidecar
    end
    frontend -->|prefill request| prefill_sidecar
    prefill_sidecar -->|KV handoff reference| frontend
    frontend -->|decode request| decode_sidecar
    prefill_engine -->|NIXL KV transfer| decode_engine
    prefill_sidecar -.->|KV events + load for routing| frontend
    decode_sidecar -.->|KV events + load for routing| frontend
Loading

The aggregate example uses the same sidecar/engine pod layout with two aggregate workers; it has no prefill-to-decode NIXL transfer.

Deployment requirements

  • The DGDs pin matching published Dynamo frontend and sidecar images from 20261023-75d0686 by digest. This revision includes the merged KV-routing implementation from feat(trtllm-sidecar): enable KV routing and subagent affinity through OpenEngine #15087; no custom Dynamo image build is required.
  • The DGDs pin nvcr.io/nvidia/tensorrt-llm/release:1.4.0a0.dev202610180820 to its verified multi-architecture OCI digest. No GHCR pull secret is required for the TensorRT-LLM engine image.

Validation

  • GPU E2E with the unmodified, digest-pinned official Dynamo frontend and sidecar 20261023-75d0686 images and the pinned TensorRT-LLM release: aggregate DGD 12/12 HTTP requests passed; disaggregate DGD 24/24 passed. Streaming completed successfully in both modes.
  • Repeated-prefix routing changed from 0 to 20 cached blocks. Native C++ ZMQ KV events and changing KV occupancy were observed. The disaggregate run exercised both prefill replicas, both decode replicas, and all four prefill/decode worker pairs; all four engines emitted events.
  • Two isolated native GetLoad UNAVAILABLE responses reporting stale occupancy recovered. There were no HTTP failures or container restarts. All test resources were removed after logs and metrics were saved.
  • Scope: same-node A100, Qwen3-0.6B, TP1 per engine. The cluster's older operator required copying discovery environment variables into the runtime container. Other test-only adjustments were public-model secret omission, node placement, and a 4096-token KV-cache limit. No binary or wheel overlays were used. This does not establish multi-rank/multi-node or large-model benchmark reliability.
  • Both manifests parse, launcher shell syntax checks pass, all 15 GPU configuration helper checks pass, and applicable pre-commit checks pass. The launch scripts were previously validated separately with real GPU traffic; the current official-image rerun covers the two DGDs.

Summary by CodeRabbit

  • New Features
    • Added local launch options and Kubernetes deployment examples for TensorRT-LLM KV-aware routing, including aggregated and experimental disaggregated configurations.
    • The examples support streaming responses and KV-cache reuse, with separate prefill and decode workers in the disaggregated setup.
  • Documentation
    • Expanded setup guidance with build requirements, image configuration, GPU counts, and instructions for running the KV-router examples.

TensorRT-LLM is standardizing on OpenEngine (`openengine.v1`) as its gRPC
surface. Migrate the sidecar off the bespoke `TrtllmService` protocol, and add
disaggregated serving, which the old protocol had no representation for.

The protos are vendored rather than pulled from the Buf Schema Registry: the
BSR Rust SDK requires a per-developer token to build, and it forced the crate
onto a tonic version island. `proto/README.md` records the upstream commit and
a SHA-256 per file so the copies are verifiable.

Disaggregation follows the protocol's own shape. A prefill worker marks its
request `context_only` through `extra` and terminates with `PrefillReady`
instead of `finished`; the decode worker replays the returned `KvSessionRef`
verbatim. A `finished` on a prefill worker means the context phase ended
without transmitting a handoff, which the frontend's prefill router would treat
as an already-complete request and answer with an empty response, so it is
rejected instead. A decode request that arrives without a handoff is the
conditional-disaggregation bypass and runs the whole request locally.

Routing targets travel as the protocol's `openengine-target-dp-rank` metadata
rather than being rejected, so disaggregation composes with KV-aware routing.
Decode cancellation is deferred to the terminal event, since the transceiver
releases transferred KV when the engine finishes the request and not when the
client goes away.

`PrefillReady` is the final response for a context request, so its usage is
authoritative and is forwarded, including the cached-token count the frontend
relays to the decode leg. Engine errors keep the codes the router acts on:
an overload sheds and migrates instead of surfacing as an opaque failure.

`json.rs` moves from the vLLM sidecar into `dynamo-sidecar-common`, since both
sidecars now need Struct/JSON conversion. Behaviour is unchanged.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…ode deferral

RoutingHints.dp_rank names the decode worker and prefill_dp_rank the
prefill one. Falling back from one to the other sent the prefill worker's
rank to a decode engine whenever the decode router chose no rank, and made
the prefill fallback dead. Select by mode, as the vLLM and SGLang sidecars
already do.

The decode cancellation deferral had no upper bound, so a cancelled decode
request generated its whole budget with no consumer. Defer only until the
first token proves the KV transfer landed, and only for a request that
actually took a handoff.

Also carry cached prompt tokens through to the terminal on both the
aggregated and decode paths -- they were collected on PrefillReady and
then dropped -- clamped to the prompt the client sent; restore the
consolidated lib/sidecar/Dockerfile references the rebase reverted; and
fix a mislabelled codec field name in the vLLM tests.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…ndoff

The OpenEngine servicer suppresses `finished` on a context request only when
it already sent a `PrefillReady` (streaming.py:174-184). A context phase that
hits a stop condition on its one token, or is cancelled before transmission,
deliberately gets the real terminal instead -- STOP, LENGTH or CANCELLED --
because there is no decode leg to answer the caller. The frontend supports
exactly this (kv_router/prefill_router/admission.rs:95-108).

Rejecting every prefill `finished` as a protocol error failed those requests.
Surface the terminal instead, and carry the context phase's tokens on it:
they are held back rather than streamed so the decode leg's replay cannot
duplicate them, which leaves this terminal as the only place they can appear.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
… startup

Findings from a deep review of the OpenEngine path, each confirmed against a
live TensorRT-LLM 1.3.0rc26 server.

A decode engine counts the blocks transferred into it as cache hits, which is a
different quantity from the context phase's prefix-cache hit: the prefill leg
reported 5 of 6 prompt tokens while the decode leg reported 6, so every
disaggregated request told the client -- and the planner -- it had a ~100%
cache hit rate. Once a leg takes a handoff, the context phase is now the only
source of that count, including when it reported none.

A context request that ends at its one-token budget never transmitted its KV.
The servicer emits `Finished(LENGTH)` there, which the frontend's prefill
router rejects with a misleading "missing disaggregated_params"; returning the
single token instead would silently truncate the completion. It now fails
naming the cause.

Cancellation was deferred before the request was dispatched, where no KV can be
stranded, so a caller that gave up waited out the dispatch anyway.

`GetModelInfo` was a single un-retried call bounded by a hardcoded 30s, so an
engine that binds its port before the model finishes loading crash-looped the
worker and `--grpc-startup-deadline` did nothing. It now retries until that
deadline and fails fast only on answers waiting cannot fix. `--context-length`
returns as an override for an engine that reports a wrong window -- TensorRT-LLM
falls back to `max_input_len` and reads as 1024 when `max_seq_len` is unset --
and the derived `max_tokens` is capped by `ModelInfo.max_output_tokens`.

`ResourceExhausted` no longer maps to a migratable WorkerOverloaded in the
shared helper, where it silently changed vLLM's behaviour and could burn the
migration budget on a deterministic per-request rejection; the TensorRT-LLM
Generate path maps it locally instead, where the code's meaning is known.

The handoff codec becomes a serde mirror struct with `deny_unknown_fields`, so
the decode worker requires exactly what the prefill worker wrote; a dropped
`dp_rank` previously read as rank 0 and pulled KV from the wrong shard. That
removes eight hand-written error branches. `ResponseState` becomes a typed
phase so a prefill worker cannot reach a decode worker's accounting, the
per-token path makes one pass over the deltas instead of three, and the
1965-line test file splits by surface.

Validation: 41 unit + 1 executable test green; new tests are mutation-checked;
aggregated and disaggregated end-to-end re-run against live TensorRT-LLM
engines with identical tokens and usage.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
The sidecars no longer agree on a protobuf version -- vLLM generates against
prost 0.14 while TensorRT-LLM and SGLang are on 0.13 -- and the two
`prost_types::Struct` types are unrelated, so moving the codec into the shared
crate left a second copy behind in vLLM. 130 lines of number, null and NaN
handling existed twice, which is 130 lines where a fix can land in one sidecar
and not the other.

The codec is now emitted once per `prost_types` path by a macro, the same shape
`error::status_to_dynamo_v14` already uses for this problem, and vLLM keeps only
the labels that name its payload and peer in error messages -- 138 lines down to
27, with no call site changed. A test asserts both instantiations agree, so a
divergence fails here rather than in one sidecar's own suite.

`startup_deadline` was likewise implemented twice, identically; vLLM now uses
the shared one. In the tests, four copies of the same nine-line engine
construction collapse into one `engine_with` helper that takes what they
actually varied.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…VERLOADED

The OpenEngine servicer rejects `openengine-target-dp-rank` in the same
`_trace_headers` branch that rejects `openengine-priority`, so every request
carrying a routing rank failed with UNIMPLEMENTED -- a non-migratable 5xx.
Mirror the priority guard and drop the metadata plumbing entirely; a session's
KV affinity still travels in `KvSessionRef.dp_rank`, inside the request body.

`ERROR_CODE_OVERLOADED` has one emitter, the 30-second consumer-stall watchdog,
which fires when this sidecar stopped draining the stream. Mapping it to
`WorkerOverloaded` re-dispatched to a second worker that stalls the same way, so
map it to a plain engine error. `generate_status` scoped RESOURCE_EXHAUSTED to
the Generate path, but no OpenEngine site emits that code; drop it.

Also: apply the decode deferral to the dispatch itself, since `generate` sends
before it awaits headers; hold whole `TokenInfo`s on a prefill leg so a
handoff-less terminal keeps its logprobs; escape the peer's status text in the
abort warning; and correct the docs, which claimed an engine started without
`--max_seq_len` reports a `max_input_len` placeholder -- it leaves
`max_context_length` unset.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…on string

The local engine image self-reports `1.3.0rc26`, but it is a build of
TensorRT-LLM main at 8bbaf66bd5; the published v1.3.0rc26 tree does not
carry the OpenEngine servicer at all. Cite the commit instead.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…ine release

TensorRT-LLM now ships the OpenEngine servicer in a public tag, so the docs no
longer have to tell readers to build their own engine image. Name
1.3.0rc27.dev202609170000 as the floor in the README, the launch script, and the
deployment manifest.

The release ships the servicer but not the `openengine.v1` Python bindings, and
the servicer imports them, so a stock image exits with "Failed to import
OpenEngine support" instead of binding the listener. The instructions say so and
keep the pinned pip step rather than implying the image is ready as-is.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Started without `--max_seq_len`, TensorRT-LLM answers `GetModelInfo` with
`max_context_length` unset, and `wait_for_model_limits` then polls for the whole
startup deadline before giving up. `deploy/agg.yaml` shipped exactly that: no
`--max_seq_len` on the engine and no `--context-length` on the sidecar, so the
worker took the default 1800 seconds to register and then rejected every request
that omitted `max_tokens`. Measured against
release:1.3.0rc27.dev202609170000 with a 20s deadline: 19.07s of polling, then
`context_length=None`, then `InvalidRequest: TensorRT-LLM requires max_tokens`.

The failure is quiet from outside -- the frontend still advertises a context
window from the model card, so `/v1/models` looks healthy while every such
request 500s. Pin 4096 on both sides, the way `launch/agg.sh` already does.

Also correct `--grpc-startup-deadline` to `--grpc-startup-deadline-secs` in the
four places that named it, one of them the remediation line of the error an
operator hits at the end of that wait, and point the docs site at the published
OpenEngine release like the README already does.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
GetModelInfo reads llm.args.max_seq_len, which the PyTorch path never writes
back, so the servicer answers UNSET while the executor is already running at the
window it resolved from the model config -- measured on
release:1.3.0rc27.dev202609170000 with TinyLlama-1.1B: executor max_seq_len=2048,
GetModelInfo UNSET. The pin is a workaround for that, not a property of the
OpenEngine contract; say so where it is, so it goes away when the servicer
reports the resolved window.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@tanmayv25
tanmayv25 requested a review from a team as a code owner October 15, 2026 19:26
@github-actions github-actions Bot added docs documentation Improvements or additions to documentation labels Oct 15, 2026
@tedzhouhk
tedzhouhk requested review from a team as code owners October 22, 2026 16:26
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the router Relates to routing, KV-aware routing, etc. label Oct 22, 2026
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>

@krishung5 krishung5 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we plan to add CI coverage for these launch scripts?

@nv-yna nv-yna left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed c2a839e. The helper's 15 self-tests, shell syntax checks, YAML parsing, and six launcher checks with inert process substitutes passed. The launcher checks verified memory/event config preservation and temporary-file cleanup and confirmed the GPU-selection concern noted inline. No GPU inference or multi-node validation was rerun in this review.

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed c2a839e. The KV event port comment matters most if these scripts get CI coverage. The inherited CUDA_VISIBLE_DEVICES issue is already tracked in nv-yna's thread, so I haven't repeated it.

Comment thread examples/common/gpu_utils.sh Resolved
Comment thread lib/sidecar/trtllm/launch/disagg.sh Resolved
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@tanmayv25
tanmayv25 requested a review from jthomson04 October 26, 2026 21:41
@tanmayv25
tanmayv25 enabled stacked auto-merge (squash) October 26, 2026 22:27
@tanmayv25

Copy link
Copy Markdown
Contributor Author

/ok to test 7a007a7

@tanmayv25

tanmayv25 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Do we plan to add CI coverage for these launch scripts?

Yes. I will add that soon.

@tanmayv25
tanmayv25 merged commit 6d80985 into main Oct 26, 2026
108 checks passed
@tanmayv25
tanmayv25 deleted the docs/trtllm-kv-routing-examples branch October 26, 2026 23:09

@dmitry-tokarev-nv dmitry-tokarev-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No P0, P1 or P2 found. This review read head 7a007a7f. The PR merged during the review as squash 6d80985a on main b5313e78, and the 6 files in that commit match 7a007a7f byte for byte.

  • [P3] lib/sidecar/trtllm/launch/disagg_kv_router.sh:109: a missing managed port (DYN_MANAGED_PORTS=1) for worker 3 or 4 fails only after the frontend and earlier workers started. Inline, with a tested diff.
  • [P3] PR description: the Summary does not name the change to build_trtllm_override_args_with_mem in examples/common/gpu_utils.sh. 12 launchers in examples/backends/trtllm/launch/, 2 sidecar launchers and tests/utils/gpu_args.py call that function. The squash commit docs(trtllm): add KV routing launch and DGD examples (#16561) does not name it either. Please add one Summary line for the helper change. Also remove "review and merge that implementation PR first", because #15087 merged on 2026-10-22.
  • Open: CI coverage for the two new launchers. krishung5 asked about it, and the author plans to add it. A note for that work: the TensorRT-LLM image that CI pins on main (release:1.4.0rc0) has no --openengine-enable-load-metrics in tensorrt_llm/commands/serve.py, and v1.4.0rc1 has it. A test needs that pin to move, or a skip guard like _trtllm_serves_openengine().
  • Open: no CI job ran this diff before the merge. The changed-files job reported core: [] and trtllm: [], so run 40413423615 skipped every TensorRT-LLM job and ran only the sidecar image build. No job runs examples/common/gpu_utils.sh --self-test.
  • Not run here: disagg_kv_router.sh on GPUs, and the two DGDs on a cluster.
What this review ran, with controls.

The helper build_trtllm_override_args_with_mem ran with 31 inputs at base d66fc3f4 and at the head:

  • For every input that a caller on main passes, the output and the exit code match base. These inputs are no cap, a token cap, a byte cap, and both caps. They also include the merge JSON from examples/backends/trtllm/launch/agg.sh, disagg_same_gpu.sh, and the sidecar agg.sh and disagg.sh. Every setter in tests/ passes str(int(...)).
  • With the JSON of the new launchers and a cap, base prints two kv_cache_config keys. The head prints one merged object.
  • Input that is not an integer or not a JSON object now stops the helper with a Python error, where base printed it through. Examples are 4096.0, 1e9, abc, {"kv_cache_config": null}, a JSON list, text that is not JSON, and only spaces. All 16 launchers that call the helper use set -e, so they stop and do not start an engine without its cap.
  • Self-test: base 13/13, head 15/15, and the head merged with main e3b7b762 15/15, under bash 5.3 and 3.2. The merge has no conflict. Two deliberate breaks make the two new cases fail (13 passed, 2 failed). One break puts the old function back. The other applies the cap before the JSON of the caller.

The launchers:

  • bash -n and shellcheck 0.10.0 report nothing on both scripts. A planted SC2086 is caught.
  • Stub cases on Linux: each worker gets its own gRPC, KV-event and system port, with default ports and with managed ports. Each KV publisher uses tcp://127.0.0.1:<port>.
  • Inherited CUDA_VISIBLE_DEVICES entries, numbers and UUIDs, map to the workers in order. A short list fails before any child starts.
  • --config and --extra_llm_api_options stop both launchers before the bindings step. No temporary config file stays after exit.

A GPU run on one RTX 6000 Ada:

  • agg_kv_router.sh from the head ran in a container from nvcr.io/nvidia/tensorrt-llm/release@sha256:b8c862fea7c4306043dd2dbda06175af0fc8af4f336435597c07ec92f6fff3bd. The frontend venv and dynamo-trtllm-sidecar came from the 20261023-75d0686 nightly images that the DGDs pin.
  • The run put both workers on GPU 0 and served the model Qwen/Qwen3-0.6B. It set _PROFILE_OVERRIDE_TRTLLM_MAX_TOTAL_TOKENS=4096, DYN_DISCOVERY_BACKEND=file, PRTE_ALLOW_RUN_AS_ROOT=1 and PRTE_ALLOW_RUN_AS_ROOT_CONFIRM=1.
  • Both engines logged max_tokens 4096 is provided and started with the merged config. Both sidecars logged events_enabled=true load_enabled=true.
  • Streamed requests with nvext.extra_fields set to ["worker_id", "timing"]: prompt A went to one worker and prompt B to the other, every time. kv_hit_rate was 0.0 on the first request of each prompt, and 0.954 and 0.947 on each of 3 repeats. The stored event counter of each worker went from none to 1.
  • Listeners: 127.0.0.1:5557 and 127.0.0.1:5558 for KV events, and gRPC ports 30001 and 30011 on loopback only.
  • The startup probe script from the DGDs exits 0 against ports 30001 and 30011, and 1 against the closed port 30002.

The manifests:

  • Both DGDs pass the v1beta1 schema of the CRD on main, with unknown fields rejected. The existing agg.yaml and disagg.yaml also pass.
  • Six deliberate breaks fail: an unknown field, a bad type, a bad runtimeVersionOverride, an unknown container field, a bad backendFramework, and a number for restartPolicy. The CEL rules that apply to these DGDs hold by reading.
  • Each pinned digest is the digest of its tag on nvcr.io, as a multi-arch index with amd64 and arm64. A bogus tag and a bogus digest return 404.
  • The ports, the ZMQ endpoints and the sidecar flags (--grpc-endpoint, --model-path, --context-length, --disaggregation-mode) match the launchers and main. Nightly 75d0686 contains #15087 (f7deb693dd).

The head keeps the fixes from the 12 resolved threads. This review probed six of them: the cap merge, the loopback KV bind, inherited GPUs, managed KV ports, the --config order, and temporary file cleanup.

kv_var="TRTLLM_WORKER${worker}_KV_EVENT_PORT"
system_var="DYN_SYSTEM_PORT${worker}"
gpu="${!gpu_var}"
grpc_port=$(dyn_port DYN_ENGINE_GRPC_PORT "$worker" "${!grpc_var:-$((30001 + 10 * (worker - 1)))}")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] A missing managed port for worker 3 or 4 fails only after the frontend and earlier workers started. lib/sidecar/trtllm/launch/disagg_kv_router.sh:109. Managed means DYN_MANAGED_PORTS=1, which the tests set. Please resolve all four workers' ports before the frontend starts, as agg_kv_router.sh does. The PR is merged, so here is a tested diff for a follow-up:

--- a/lib/sidecar/trtllm/launch/disagg_kv_router.sh
+++ b/lib/sidecar/trtllm/launch/disagg_kv_router.sh
@@ -96,18 +96,24 @@
 HTTP_PORT="${DYN_HTTP_PORT:-8000}"
 print_launch_banner "TensorRT-LLM OpenEngine KV-aware sidecars (2 prefill, 2 decode)" "$MODEL" "$HTTP_PORT" \
     "KV transfer: ${TRTLLM_CACHE_TRANSCEIVER_BACKEND} cache transceiver"
+# Resolve every worker's ports before starting any process.
+GRPC_PORTS=() KV_PORTS=()
+for worker in 1 2 3 4; do
+    grpc_var="TRTLLM_WORKER${worker}_GRPC_PORT"
+    kv_var="TRTLLM_WORKER${worker}_KV_EVENT_PORT"
+    GRPC_PORTS[worker]=$(dyn_port DYN_ENGINE_GRPC_PORT "$worker" "${!grpc_var:-$((30001 + 10 * (worker - 1)))}")
+    KV_PORTS[worker]=$(dyn_port DYN_VLLM_KV_EVENT_PORT "$worker" "${!kv_var:-$((5556 + worker))}")
+done
 python3 -m dynamo.frontend --router-mode kv &
 
 for worker in 1 2 3 4; do
     role=prefill
     if (( worker > 2 )); then role=decode; fi
     gpu_var="TRTLLM_WORKER${worker}_GPU"
-    grpc_var="TRTLLM_WORKER${worker}_GRPC_PORT"
-    kv_var="TRTLLM_WORKER${worker}_KV_EVENT_PORT"
     system_var="DYN_SYSTEM_PORT${worker}"
     gpu="${!gpu_var}"
-    grpc_port=$(dyn_port DYN_ENGINE_GRPC_PORT "$worker" "${!grpc_var:-$((30001 + 10 * (worker - 1)))}")
-    kv_port=$(dyn_port DYN_VLLM_KV_EVENT_PORT "$worker" "${!kv_var:-$((5556 + worker))}")
+    grpc_port="${GRPC_PORTS[worker]}"
+    kv_port="${KV_PORTS[worker]}"
     system_port="${!system_var:-$((8080 + worker))}"
     config=$(mktemp "${TMPDIR:-/tmp}/dynamo-trtllm-disagg-kv.XXXXXX.yaml")
     TRTLLM_CONFIG_FILES+=("$config")
Measured on the head and on the fix, with stub commands.

The launcher ran on Linux (bash 5) with stub frontend, engine and sidecar commands. Each stub writes its arguments and environment to a log, then exits after a few seconds. The fix passed bash -n, shellcheck 0.10.0 and these cases:

case head: exit, children started with the fix
DYN_ENGINE_GRPC_PORT3 missing 1, 5 1, 0
DYN_VLLM_KV_EVENT_PORT4 missing 1, 7 1, 0
all managed ports set 0, 9 0, 9, same arguments
default ports 0, 9 0, 9, same arguments
inherited GPUs 4,5,6,7 0, 9 0, 9, same arguments
byte cap 0, 9 0, 9, same arguments
only 2 GPUs visible 1, 0 1, 0

Both versions print Missing or invalid managed port DYN_ENGINE_GRPC_PORT3: <unset>. agg_kv_router.sh with DYN_ENGINE_GRPC_PORT2 missing exits 1 before it starts any child.

@biswapanda biswapanda left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs documentation Improvements or additions to documentation router Relates to routing, KV-aware routing, etc. size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants