Skip to content

feat(trtllm-sidecar): enable KV routing and subagent affinity through OpenEngine - #15087

Merged
tanmayv25 merged 40 commits into
mainfrom
feat/trtllm-openengine-kv-routing
Oct 22, 2026
Merged

tanmayv25 merged 40 commits into
mainfrom
feat/trtllm-openengine-kv-routing

Conversation

@tanmayv25

@tanmayv25 tanmayv25 commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #16453. Part of #10841.

Summary

  • Discover TensorRT-LLM native per-rank KV event sources and routing load through OpenEngine, publish ZMQ events and per-rank load snapshots, and forward Dynamo's selected attention-DP rank.
  • Negotiate heartbeat expiry and supervise node-local KV publishers for multi-rank deployments.
  • Carry the child's stable conversation ID through prefill and decode, and send a signed parent-affinity hint on child context requests when both the sidecar and OpenEngine server enable it.
  • Port the implementation onto current Dynamo main without rewriting the existing PR branch history.
  • The launch scripts and DGDs are split into the stacked examples PR #16561, which targets this branch.

Deployment requirements

  • Use nvcr.io/nvidia/tensorrt-llm/release:1.4.0a0.dev202610180820 for TensorRT-LLM. This released nightly includes the native OpenEngine KV-event and routing-load support from NVIDIA/TensorRT-LLM#19440, including KVCM2.
  • For parent-child rank affinity, enable TensorRT-LLM conversation affinity and use the same private TRTLLM_SUBAGENT_AFFINITY_AUTH_KEY in the prefill sidecar and OpenEngine server. The sidecar checks the server capability before serving traffic.

Validation

  • cargo test -p dynamo-trtllm-sidecar --lib: 59 passed, 2 existing ignored.
  • cargo test -p dynamo-trtllm-sidecar --test executable: 2 passed.
  • cargo test -p dynamo-trtllm-mocker --test sidecar: 7 passed; cargo test -p dynamo-sidecar-common --lib: 17 passed.
  • cargo test -p dynamo-llm --lib kv_router::publisher: 103 passed.
  • cargo test -p dynamo-llm --lib child_context_uses_parent_affinity_while_decode_retains_child: passed.
  • cargo check -p dynamo-trtllm-sidecar --all-targets and cargo check --manifest-path lib/bindings/python/Cargo.toml: passed.
  • cargo clippy on the affected Rust crates with --all-targets -- -D warnings, cargo fmt --all --check, docs lint, and pre-commit on changed files: passed.
  • Earlier remote multi-rank smoke covered cache-affinity selection, load reporting, and worker removal on the pre-port implementation. Current-main GPU validation and the four-arm GLM comparison are pending cluster scheduling.

Known limitation

The last published load snapshot remains stale if polling fails. Heartbeat expiry clears KV residency but does not quarantine inference traffic.

Summary by CodeRabbit

  • New Features
    • TensorRT-LLM now supports KV-cache routing with compatible servers, including per-rank event discovery and load reporting.
    • Multi-node deployments can route requests to selected data-parallel ranks when the server supports rank targeting.
    • Prefill requests can use signed parent-session affinity hints when the required server and gateway capabilities are available.
    • KV event monitoring detects missing sequence updates and inactive sources, clearing stale cache state when needed.
  • Bug Fixes
    • Prefill routing now handles child and parent session affinity more reliably.

TensorRT-LLM is standardizing on OpenEngine (`openengine.v1`) as its gRPC
surface. Migrate the sidecar off the bespoke `TrtllmService` protocol, and add
disaggregated serving, which the old protocol had no representation for.

The protos are vendored rather than pulled from the Buf Schema Registry: the
BSR Rust SDK requires a per-developer token to build, and it forced the crate
onto a tonic version island. `proto/README.md` records the upstream commit and
a SHA-256 per file so the copies are verifiable.

Disaggregation follows the protocol's own shape. A prefill worker marks its
request `context_only` through `extra` and terminates with `PrefillReady`
instead of `finished`; the decode worker replays the returned `KvSessionRef`
verbatim. A `finished` on a prefill worker means the context phase ended
without transmitting a handoff, which the frontend's prefill router would treat
as an already-complete request and answer with an empty response, so it is
rejected instead. A decode request that arrives without a handoff is the
conditional-disaggregation bypass and runs the whole request locally.

Routing targets travel as the protocol's `openengine-target-dp-rank` metadata
rather than being rejected, so disaggregation composes with KV-aware routing.
Decode cancellation is deferred to the terminal event, since the transceiver
releases transferred KV when the engine finishes the request and not when the
client goes away.

`PrefillReady` is the final response for a context request, so its usage is
authoritative and is forwarded, including the cached-token count the frontend
relays to the decode leg. Engine errors keep the codes the router acts on:
an overload sheds and migrates instead of surfacing as an opaque failure.

`json.rs` moves from the vLLM sidecar into `dynamo-sidecar-common`, since both
sidecars now need Struct/JSON conversion. Behaviour is unchanged.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…ode deferral

RoutingHints.dp_rank names the decode worker and prefill_dp_rank the
prefill one. Falling back from one to the other sent the prefill worker's
rank to a decode engine whenever the decode router chose no rank, and made
the prefill fallback dead. Select by mode, as the vLLM and SGLang sidecars
already do.

The decode cancellation deferral had no upper bound, so a cancelled decode
request generated its whole budget with no consumer. Defer only until the
first token proves the KV transfer landed, and only for a request that
actually took a handoff.

Also carry cached prompt tokens through to the terminal on both the
aggregated and decode paths -- they were collected on PrefillReady and
then dropped -- clamped to the prompt the client sent; restore the
consolidated lib/sidecar/Dockerfile references the rebase reverted; and
fix a mislabelled codec field name in the vLLM tests.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…ndoff

The OpenEngine servicer suppresses `finished` on a context request only when
it already sent a `PrefillReady` (streaming.py:174-184). A context phase that
hits a stop condition on its one token, or is cancelled before transmission,
deliberately gets the real terminal instead -- STOP, LENGTH or CANCELLED --
because there is no decode leg to answer the caller. The frontend supports
exactly this (kv_router/prefill_router/admission.rs:95-108).

Rejecting every prefill `finished` as a protocol error failed those requests.
Surface the terminal instead, and carry the context phase's tokens on it:
they are held back rather than streamed so the decode leg's replay cannot
duplicate them, which leaves this terminal as the only place they can appear.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
… startup

Findings from a deep review of the OpenEngine path, each confirmed against a
live TensorRT-LLM 1.3.0rc26 server.

A decode engine counts the blocks transferred into it as cache hits, which is a
different quantity from the context phase's prefix-cache hit: the prefill leg
reported 5 of 6 prompt tokens while the decode leg reported 6, so every
disaggregated request told the client -- and the planner -- it had a ~100%
cache hit rate. Once a leg takes a handoff, the context phase is now the only
source of that count, including when it reported none.

A context request that ends at its one-token budget never transmitted its KV.
The servicer emits `Finished(LENGTH)` there, which the frontend's prefill
router rejects with a misleading "missing disaggregated_params"; returning the
single token instead would silently truncate the completion. It now fails
naming the cause.

Cancellation was deferred before the request was dispatched, where no KV can be
stranded, so a caller that gave up waited out the dispatch anyway.

`GetModelInfo` was a single un-retried call bounded by a hardcoded 30s, so an
engine that binds its port before the model finishes loading crash-looped the
worker and `--grpc-startup-deadline` did nothing. It now retries until that
deadline and fails fast only on answers waiting cannot fix. `--context-length`
returns as an override for an engine that reports a wrong window -- TensorRT-LLM
falls back to `max_input_len` and reads as 1024 when `max_seq_len` is unset --
and the derived `max_tokens` is capped by `ModelInfo.max_output_tokens`.

`ResourceExhausted` no longer maps to a migratable WorkerOverloaded in the
shared helper, where it silently changed vLLM's behaviour and could burn the
migration budget on a deterministic per-request rejection; the TensorRT-LLM
Generate path maps it locally instead, where the code's meaning is known.

The handoff codec becomes a serde mirror struct with `deny_unknown_fields`, so
the decode worker requires exactly what the prefill worker wrote; a dropped
`dp_rank` previously read as rank 0 and pulled KV from the wrong shard. That
removes eight hand-written error branches. `ResponseState` becomes a typed
phase so a prefill worker cannot reach a decode worker's accounting, the
per-token path makes one pass over the deltas instead of three, and the
1965-line test file splits by surface.

Validation: 41 unit + 1 executable test green; new tests are mutation-checked;
aggregated and disaggregated end-to-end re-run against live TensorRT-LLM
engines with identical tokens and usage.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
The sidecars no longer agree on a protobuf version -- vLLM generates against
prost 0.14 while TensorRT-LLM and SGLang are on 0.13 -- and the two
`prost_types::Struct` types are unrelated, so moving the codec into the shared
crate left a second copy behind in vLLM. 130 lines of number, null and NaN
handling existed twice, which is 130 lines where a fix can land in one sidecar
and not the other.

The codec is now emitted once per `prost_types` path by a macro, the same shape
`error::status_to_dynamo_v14` already uses for this problem, and vLLM keeps only
the labels that name its payload and peer in error messages -- 138 lines down to
27, with no call site changed. A test asserts both instantiations agree, so a
divergence fails here rather than in one sidecar's own suite.

`startup_deadline` was likewise implemented twice, identically; vLLM now uses
the shared one. In the tests, four copies of the same nine-line engine
construction collapse into one `engine_with` helper that takes what they
actually varied.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…VERLOADED

The OpenEngine servicer rejects `openengine-target-dp-rank` in the same
`_trace_headers` branch that rejects `openengine-priority`, so every request
carrying a routing rank failed with UNIMPLEMENTED -- a non-migratable 5xx.
Mirror the priority guard and drop the metadata plumbing entirely; a session's
KV affinity still travels in `KvSessionRef.dp_rank`, inside the request body.

`ERROR_CODE_OVERLOADED` has one emitter, the 30-second consumer-stall watchdog,
which fires when this sidecar stopped draining the stream. Mapping it to
`WorkerOverloaded` re-dispatched to a second worker that stalls the same way, so
map it to a plain engine error. `generate_status` scoped RESOURCE_EXHAUSTED to
the Generate path, but no OpenEngine site emits that code; drop it.

Also: apply the decode deferral to the dispatch itself, since `generate` sends
before it awaits headers; hold whole `TokenInfo`s on a prefill leg so a
handoff-less terminal keeps its logprobs; escape the peer's status text in the
abort warning; and correct the docs, which claimed an engine started without
`--max_seq_len` reports a `max_input_len` placeholder -- it leaves
`max_context_length` unset.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…on string

The local engine image self-reports `1.3.0rc26`, but it is a build of
TensorRT-LLM main at 8bbaf66bd5; the published v1.3.0rc26 tree does not
carry the OpenEngine servicer at all. Cite the commit instead.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
…ine release

TensorRT-LLM now ships the OpenEngine servicer in a public tag, so the docs no
longer have to tell readers to build their own engine image. Name
1.3.0rc27.dev202609170000 as the floor in the README, the launch script, and the
deployment manifest.

The release ships the servicer but not the `openengine.v1` Python bindings, and
the servicer imports them, so a stock image exits with "Failed to import
OpenEngine support" instead of binding the listener. The instructions say so and
keep the pinned pip step rather than implying the image is ready as-is.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Started without `--max_seq_len`, TensorRT-LLM answers `GetModelInfo` with
`max_context_length` unset, and `wait_for_model_limits` then polls for the whole
startup deadline before giving up. `deploy/agg.yaml` shipped exactly that: no
`--max_seq_len` on the engine and no `--context-length` on the sidecar, so the
worker took the default 1800 seconds to register and then rejected every request
that omitted `max_tokens`. Measured against
release:1.3.0rc27.dev202609170000 with a 20s deadline: 19.07s of polling, then
`context_length=None`, then `InvalidRequest: TensorRT-LLM requires max_tokens`.

The failure is quiet from outside -- the frontend still advertises a context
window from the model card, so `/v1/models` looks healthy while every such
request 500s. Pin 4096 on both sides, the way `launch/agg.sh` already does.

Also correct `--grpc-startup-deadline` to `--grpc-startup-deadline-secs` in the
four places that named it, one of them the remediation line of the error an
operator hits at the end of that wait, and point the docs site at the published
OpenEngine release like the README already does.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
GetModelInfo reads llm.args.max_seq_len, which the PyTorch path never writes
back, so the servicer answers UNSET while the executor is already running at the
window it resolved from the model config -- measured on
release:1.3.0rc27.dev202609170000 with TinyLlama-1.1B: executor max_seq_len=2048,
GetModelInfo UNSET. The pin is a workaround for that, not a property of the
OpenEngine contract; say so where it is, so it goes away when the servicer
reports the resolved window.

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation router Relates to routing, KV-aware routing, etc. labels Sep 19, 2026
@github-actions github-actions Bot added the feat label Sep 19, 2026
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Base automatically changed from feat/trtllm-openengine-sidecar to main September 30, 2026 20:16
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>

@jh-nv jh-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One blocking issue inline.

Comment thread lib/sidecar/trtllm/src/convert.rs Resolved

@dmitry-tokarev-nv dmitry-tokarev-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No P0 or P1 found. This review adds one P2 and two P3 findings, and it measured one open P2 from another reviewer. Reviewed head 517e9ee4a5f4f49e95cdbf802eb5ddf1a020dae4.

  • [P2] --help prints the value of TRTLLM_SUBAGENT_AFFINITY_AUTH_KEY. lib/sidecar/trtllm/src/args.rs:29 (inline, with a tested suggestion).
  • [P2] Open thread from another reviewer, measured here: with KV routing on, a single-rank engine still gets a synthetic conversation_id on every anonymous request. lib/sidecar/trtllm/src/convert.rs:161 (reply with a tested fix and a test that fails at this head).
  • [P3] No test covers the new expected_rank filter in the shared ZMQ listener. lib/llm/src/kv_router/publisher/zmq_listener.rs:150 (inline, with a tested test).
  • [P3] The PR description does not match the code in five places (details below).
The five description mismatches.
  1. The Validation section names child_context_uses_parent_affinity_while_decode_retains_child. No test has that name at this head, so that command runs 0 tests. The new test is child_prefill_uses_parent_worker_while_decode_keeps_group_binding.
  2. The test counts are older than the code. At this head: dynamo-trtllm-sidecar --lib 64 passed and 2 ignored (description: 59), dynamo-trtllm-mocker --test sidecar 5 passed (description: 7), dynamo-sidecar-common --lib 18 passed (description: 17), dynamo-llm --lib kv_router::publisher 105 passed (description: 103).
  3. "use the same private TRTLLM_SUBAGENT_AFFINITY_AUTH_KEY in the prefill sidecar and OpenEngine server": the server does not read that variable. It takes internal_request_auth_key from its YAML configuration, as lib/sidecar/trtllm/README.md says.
  4. "Negotiate heartbeat expiry": no TensorRT-LLM server advertises kv_event_heartbeat_interval_ms. It is absent in 1.4.0rc1.dev202610201007 (a live GetServerInfo and a source search) and on TensorRT-LLM main. So liveness is always None against current servers, and the heartbeat clear and the clear on a sequence gap do not run. Please name the server change that will add it, or say that the path is for a future server.
  5. The known limitation says that the last load snapshot stays stale after a failed poll. That is true for 30 seconds only. Then the worker withdraws, and the sidecar exits with Routing load remained unavailable for 30 seconds; withdrawing worker (measured below). Please say so.
Earlier review threads, checked at this head.
thread status at this head evidence
dynamo-llm default features (Cargo.toml) fixed cargo tree -p dynamo-trtllm-sidecar -f "{p} [{f}]" shows dynamo-llm with no features. Control: -p dynamo-llm shows [block-manager,default].
startup error wrapper (lib.rs) fixed in the code bootstrap errors go through SidecarStartupError::from. The Python ValueError path was not run.
RESOURCE_EXHAUSTED on metadata RPCs (headless.rs) fixed A deliberate break that maps it back to the generic error fails sustained_metadata_outage_withdraws_worker and leader_monitor_rejects_restarted_native_engine.
synthetic conversation_id (convert.rs) not fixed for KV routing Still open in the newer thread on convert.rs:161. See the reply there.
clear on gap for every ZMQ source (zmq_listener.rs) fixed A deliberate break that clears on every gap fails test_start_zmq_listener_pushes_to_channel::case_1_legacy.
rank 0 rejected without targeting (convert.rs) fixed A deliberate break that restores the old check of both ranks fails prefill_ready_is_the_terminal_handoff and decode_request_replays_the_prefill_session.
What this review measured.

Rust tests in a container on Linux x86_64, toolchain 1.96.1, --locked, with the merge base 77e76c1bf3 as the control:

command merge base head
cargo test -p dynamo-trtllm-sidecar --lib 51 passed, 2 ignored 64 passed, 2 ignored
cargo test -p dynamo-trtllm-sidecar --test executable 2 passed 2 passed
cargo test -p dynamo-trtllm-mocker --test sidecar 5 passed 5 passed
cargo test -p dynamo-sidecar-common --lib 18 passed 18 passed
cargo test -p dynamo-llm --lib kv_router 431 passed 434 passed
cargo test -p dynamo-backend-common --lib 161 passed 161 passed
cargo test -p dynamo-vllm-sidecar 135 passed 135 passed
cargo test -p dynamo-sglang-sidecar 126 passed 126 passed
cargo test -p dynamo-sidecar-testkit 65 passed 65 passed

These commands pass at head:

  • cargo check -p dynamo-trtllm-sidecar --all-targets
  • cargo check --manifest-path lib/bindings/python/Cargo.toml
  • cargo clippy --all-targets -- -D warnings on the TensorRT-LLM, vLLM and SGLang sidecars, the common and testkit crates, dynamo-backend-common and dynamo-trtllm-mocker
  • cargo fmt --all --check

Each of these deliberate breaks of the new code fails at least one new test. Each file was restored by checksum.

  • Remove the liveness condition on the gap clear.
  • Change request_type in the signed payload.
  • Map RESOURCE_EXHAUSTED to the generic error.
  • Never fail the load watchdog.
  • Use the sibling group key for child prefill.
  • Restore the old check of both ranks.

Removing the expected_rank filter fails no test (inline P3).

Signed affinity: the head test vector (sha256=e98db0ec…) passes validate_parent_affinity in TensorRT-LLM 1.4.0rc1.dev202610201007. A changed signature, a different key, generation_only, and a server without a key each raise ValueError. The signature covers purpose, model, request_id, conversation_id, the parent ID and request_type, not the target rank. An empty key fails at startup. If the server does not advertise trtllm_supports_subagent_affinity, a configured key also fails startup. By the code, a one-byte key is accepted on both sides.

End to end on one RTX 6000 Ada: two trtllm-serve --grpc --grpc-protocol openengine --openengine-enable-load-metrics servers (1.4.0rc1.dev202610201007, Qwen3-0.6B, ZMQ KV events), a Dynamo frontend with --router-mode kv, and two head sidecars.

  • KV events: the router applied stored events. Each repeat of a prefix of 1,063 to 1,097 tokens (12 of 12) went to the worker that held it. These requests showed overlap_blocks=33 or 34 and about 1,060 cached_tokens. With merge-base sidecars against the same servers, all 18 requests showed overlap_blocks=0.
  • Load: the per-rank gpu_cache_usage_percent of the busy worker went from 0 to 0.0123 during a long request, and the frontend subscribed to both kv_metrics streams.
  • Engine stop: 30 seconds after one server stopped, its sidecar exited with Routing load remained unavailable for 30 seconds; withdrawing worker, and the router logged Removing worker.
  • A request started after an idle period. Then one GetLoad poll of the sidecar failed with KV occupancy sample is stale while requests are active. During a later run of long requests, direct GetLoad calls once per second all returned OK, and the sidecar did not withdraw.

The Pre Merge job rust-tests (.) on this head ran the new tests::kv_routing tests (71 sidecar library tests on the merge ref).

This server has one attention-DP rank. So this review did not run rank targeting with more than one rank, the follower relay, or signed affinity on a live server. Only the unit tests and the source reading cover them.

A merge of this head with main at 7929f5370eef7f15095b16f0dd41012b920c334c has no conflicts. The checks ran on a merge with the earlier tip ebfcce7891a760950ca7f88b700d584d59010fba, and main changed no file of this PR between the two. On that merge, cargo check --all-targets on the changed crates, cargo check of the Python bindings, dynamo-trtllm-sidecar --lib (69 passed, 2 ignored) and dynamo-llm --lib kv_router (444 passed) pass.

Open items:

  • Shared code that changes behavior without the TensorRT-LLM sidecar: lib/backend-common/src/publisher.rs now sets expected_rank for every ZMQ source, so a batch for another rank or with no rank is dropped. routing_host.rs now places a child's prefill on the parent's own binding for every backend with session affinity.

Comment thread lib/sidecar/trtllm/src/args.rs Resolved
tanmayv25 and others added 3 commits October 21, 2026 11:43
… sensitive information'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Signed-off-by: Tanmay Verma <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@dmitry-tokarev-nv

Copy link
Copy Markdown
Contributor

/ok to test f8a3646

@dmitry-tokarev-nv dmitry-tokarev-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No P0 or P1 found. One P3 from the earlier review on 2026-10-21 is still open: the PR description. Reviewed head f8a3646cf9013af8dd84daec470162e12f4c7ad3, the three commits after 517e9ee4a5.

  • Fixed: --help no longer prints TRTLLM_SUBAGENT_AFFINITY_AUTH_KEY (args.rs).
  • Fixed: a single-rank engine no longer gets a rank hint or a synthetic conversation_id (engine.rs).
  • Fixed: a test now covers the expected_rank filter (publisher/tests.rs).
  • [P3] Still open: the PR description is unchanged. The five mismatches from the earlier review remain. For example, the Validation section still names child_context_uses_parent_affinity_while_decode_retains_child, and no test has that name.
What this review measured at the new head.

Tests in a Linux container, toolchain 1.96.1, --locked:

command result
cargo test -p dynamo-trtllm-sidecar --lib 65 passed, 2 ignored
cargo test -p dynamo-trtllm-sidecar --test executable 2 passed
cargo test -p dynamo-llm --lib kv_router 434 passed
cargo clippy -p dynamo-trtllm-sidecar -p dynamo-llm --all-targets -- -D warnings pass
cargo fmt --all --check pass

Each fix was reverted in turn, and each new test then failed:

change failing test
engine.rs of 517e9ee4a5 single_rank_kv_routing_preserves_only_real_conversation_ids (kv_routing.rs:164)
args.rs of 517e9ee4a5 executable_exposes_native_grpc_configuration (tests/executable.rs:36)
expected_rank filter removed both cases of test_start_zmq_listener_pushes_to_channel (publisher/tests.rs:1521)

The CodeQL commit bbb8fa440d removes session={session_id:?} from an assert message in convert/request_tests.rs. The value is a fixed test string, so the alert was a false positive. The change touches only the failure text of a test and changes no behavior.

The new commits need no new GPU run. For a single-rank engine, the sidecar now sends no rank hint and no synthetic ID. The merge-base sidecar sent neither, and in the earlier end-to-end run it served all 18 requests on the same TensorRT-LLM server.

A merge of this head with main at b1e4b765d3779cc821267a2d7a59106220441e32 has no conflicts.

@github-actions

Copy link
Copy Markdown
Contributor

@dmitry-tokarev-nv dmitry-tokarev-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head f8a3646cf9013af8dd84daec470162e12f4c7ad3. No P0 or P1 found.

The three new commits fix both P2 findings from the earlier review and add the missing rank-filter test. If its fix is reverted, each new test fails. At this head, the sidecar library tests, the executable tests, the kv_router tests, clippy and fmt pass. A merge with main at b1e4b765d3 has no conflicts.

Open item: [P3] the PR description still does not match the code in five places, for example a test name that does not exist.

Not run: rank targeting with more than one rank, the follower relay, signed affinity on a live server, and the Python ValueError path of the bindings.

@devin-ai-integration

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 39664698826 (attempt 1) on f8a3646cf9

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 39664698826. Updated on every full-CI run of this PR.

@biswapanda biswapanda left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Comment thread lib/sidecar/trtllm/src/engine.rs Resolved
@tanmayv25
tanmayv25 enabled stacked auto-merge (squash) October 22, 2026 02:07
@tanmayv25
tanmayv25 merged commit f7deb69 into main Oct 22, 2026
125 checks passed
@tanmayv25
tanmayv25 deleted the feat/trtllm-openengine-kv-routing branch October 22, 2026 16:26
erezzarum pushed a commit to erezzarum/dynamo that referenced this pull request Oct 23, 2026
… OpenEngine (ai-dynamo#15087)

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Signed-off-by: Tanmay Verma <tanmay2592@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
JulienDarve added a commit that referenced this pull request Oct 26, 2026
Main added rank filtering, heartbeat liveness and publisher
ready()/terminated() for the TRT-LLM node-local sidecar (#15087).
Keep them alongside replay recovery:

- KvEventSourceConfig::Zmq and start_zmq_listener carry expected_rank,
  liveness and recovery; recovery still hands off to the recovery
  listener.
- The live listener applies main's rank filter, heartbeat and gap
  handling, then the shared normalize_batch, and sends through
  PublisherInput.
- The processor JoinHandle stays in processor_handle. ready() and
  terminated() read it through Mutex::get_mut, and terminated() takes
  it once the task ends, so stop_and_wait finds an empty slot.
- The TRT-LLM headless source sets recovery: None.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
tmonty12 added a commit that referenced this pull request Oct 29, 2026
The sidecar needs OpenEngine routing discovery and the node-local KV
relay from #15087, which merged after the 1.6.x releases were cut. The
frontend is pinned to the same build because only that pairing was tested.

Signed-off-by: Thomas Montfort <tjmontfort12@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat human-review router Relates to routing, KV-aware routing, etc. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(trtllm-sidecar): track OpenEngine KV and subagent routing

9 participants