Skip to content

feat(trtllm): add image-aware KV routing to OpenEngine sidecars - #17437

Merged
krishung5 merged 69 commits into
mainfrom
openengine-image-routing-dynamo
Nov 4, 2026
Merged

krishung5 merged 69 commits into
mainfrom
openengine-image-routing-dynamo

Conversation

@krishung5

@krishung5 krishung5 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Enable image-aware KV routing for the TensorRT-LLM OpenEngine sidecar. Forward frontend image identities through the negotiated engine contract and normalize native KV events to the routing hashes Dynamo expects.

Key Changes

239 added Rust production lines relative to main, excluding tests, CI, comments, and blank lines.

  • Adapter and discovery: share the UUID namespace and version-1 capability check. Generate image hints only for aggregated requests with exact frontend routing metadata; preserve caller UUIDs and reject caller use of the reserved namespace.
  • Ordered media conversion: retain main's image/video/audio passthrough and bind image hints to modality-local indices before producing the globally ordered media list. Automatic image hints in mixed requests use grouped hashes; ambiguous flat hashes are ignored. Video-aware routing remains out of scope.
  • Events and CI: isolate TRT-LLM event normalization, fail closed on invalid or future metadata without losing subsequent removals, and retain vLLM/SGLang compatibility. The official-nightly matrix runs image, video, audio, and image-routing scenarios, with video decoder setup and strict executed-test checks.

The shared UUID constant makes dynamo-kv-router a normal dependency; it was already linked through dynamo-llm, so this introduces no new package or release feature. The root and Python bindings lockfiles both record the dependency. Flat Serde wire decoding avoids the extra buffered-sequence clone from nested untagged derives.

Automatic UUIDs include the full media URL, including its query string. TensorRT-LLM combines UUID and content identity, so identical images behind rotating signed URLs get separate native KV chains. Stable caller image UUIDs restore native reuse for unchanged content across those URLs, but disable automatic image-routing hints. The sidecar rejects decoded media, so --frontend-decoding is unsupported here. Routing URLs must reference stable content.

Validation

Behavior Evidence
Media passthrough, capability gates, caller identities, and startup lifecycle 392 sidecar/testkit unit tests and 11 multimodal mock gRPC tests passed after merging main. The new mixed-media regression verifies reordered image URLs and their UUIDs, preserves the video UUID, rejects forged reserved UUIDs, and suppresses ambiguous flat hashes.
Native event normalization and other producers All 78 KV-event wire tests passed, including digest-first images, cross-block continuations, malformed metadata, removals, and positional/named-map bigrams.
CI scenario selection and failure reporting All 25 Python harness tests passed. Both --media and --scenario select image, video, and audio correctly; zero executed tests or ignored tests fail validation.
Decoder allocation cost The unchanged flat-wire wrapper retained the original allocation counts and bytes in all 12 release-mode benchmark cases (32/64-token MessagePack and JSON events); timings were within 5% of the original parser. Nested derives were 11–44% slower.
Real-weight native coverage Earlier image-routing and image GPU jobs passed. The current merged head and its stricter full-chain checks require a fresh GPU run; no GPU rerun is claimed for this update.

The native routing scenario uses real Qwen3-VL weights, checks answers and native cache reuse/isolation, captures ZMQ events, and compares the complete converted hash chain with the real Rust frontend preprocessor. CPU fixture checks of that comparison accepted the exact chain and rejected extra or missing image tokens. CPU unit runs intentionally ignore seven external-runtime tests. These nightly scenarios explicitly select the image, video, audio, and routing-event tests.

Issues and Ongoing Work

Closes DIS-2972.

Pull Request Checklist
  • Checked design, performance, tests, and code quality with the repository PR checklist and Graham review guidance.

Summary by CodeRabbit

  • New Features

    • Added image input support for TensorRT-LLM inference, including image URLs and data URIs, with capability and context-window checks.
    • Added support for routing image-related tokens in stored KV events.
    • Added automated pull-request and nightly integration coverage for image inference and routing.
  • Improvements

    • Updated TensorRT-LLM deployment examples and configurations to use a newer pinned engine image.

Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
…nto openengine-image-media-dynamo

Signed-off-by: krishung5 <krish@nvidia.com>

# Conflicts:
#	.github/scripts/test-filters.js
#	.github/workflows/pr.yaml
#	lib/sidecar/trtllm/src/tests/disagg.rs
#	lib/sidecar/trtllm/src/tests/e2e.rs
Signed-off-by: krishung5 <krish@nvidia.com>
…a-dynamo

Signed-off-by: krishung5 <krish@nvidia.com>

# Conflicts:
#	lib/sidecar/trtllm/src/engine.rs
#	lib/sidecar/trtllm/src/tests.rs
#	lib/sidecar/trtllm/src/tests/disagg.rs
#	lib/sidecar/trtllm/src/tests/engine.rs
Signed-off-by: krishung5 <krish@nvidia.com>
…a-dynamo

Signed-off-by: krishung5 <krish@nvidia.com>
…a-dynamo

Signed-off-by: krishung5 <krish@nvidia.com>

# Conflicts:
#	tests/serve/test_sidecar.py
…a-dynamo

Signed-off-by: krishung5 <krish@nvidia.com>

# Conflicts:
#	lib/sidecar/trtllm/src/convert/request_tests.rs
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Comment thread lib/sidecar/trtllm/src/tests/e2e.rs Resolved
Comment thread lib/kv-router/src/zmq_wire/trtllm.rs Resolved
Comment thread lib/sidecar/trtllm/src/convert.rs Resolved
Comment thread lib/sidecar/trtllm/src/convert.rs Resolved
Signed-off-by: krishung5 <krish@nvidia.com>

@tanmayv25 tanmayv25 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two focused items for freeze: reserve the internal routing UUID namespace, and preserve the existing media behavior and coverage when integrating these PRs.

Comment thread lib/kv-router/src/zmq_wire/trtllm.rs Resolved
Comment thread lib/sidecar/trtllm/src/convert.rs Resolved
Signed-off-by: krishung5 <krish@nvidia.com>

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Non-blocking notes inline.

Comment thread lib/sidecar/trtllm/src/convert.rs Resolved
Comment thread lib/sidecar/trtllm/src/convert.rs Resolved
…ing-dynamo

Signed-off-by: krishung5 <krish@nvidia.com>

# Conflicts:
#	.github/workflows/shared-test-trtllm-nightly-image.yml
#	lib/sidecar/trtllm/ci/run_image_test.py
#	lib/sidecar/trtllm/src/client.rs
#	lib/sidecar/trtllm/src/convert.rs
#	lib/sidecar/trtllm/src/convert/request_tests.rs
#	lib/sidecar/trtllm/src/tests/e2e.rs
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Preserve the image, video, audio, and image-routing nightly scenarios and synchronize the Python bindings lockfile.

Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions backend::sglang Relates to the sglang backend backend::trtllm Relates to the trtllm backend backend::vllm Relates to the vllm backend container deployment::k8s Relates to dynamo deployment in kubernetes documentation Improvements or additions to documentation feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` multimodal planner router Relates to routing, KV-aware routing, etc. size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants