Repository navigation
feat(trtllm): add image-aware KV routing to OpenEngine sidecars - #17437
Merged
Merged
Conversation
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
…nto openengine-image-media-dynamo Signed-off-by: krishung5 <krish@nvidia.com> # Conflicts: # .github/scripts/test-filters.js # .github/workflows/pr.yaml # lib/sidecar/trtllm/src/tests/disagg.rs # lib/sidecar/trtllm/src/tests/e2e.rs
Signed-off-by: krishung5 <krish@nvidia.com>
…a-dynamo Signed-off-by: krishung5 <krish@nvidia.com> # Conflicts: # lib/sidecar/trtllm/src/engine.rs # lib/sidecar/trtllm/src/tests.rs # lib/sidecar/trtllm/src/tests/disagg.rs # lib/sidecar/trtllm/src/tests/engine.rs
Signed-off-by: krishung5 <krish@nvidia.com>
…a-dynamo Signed-off-by: krishung5 <krish@nvidia.com>
…a-dynamo Signed-off-by: krishung5 <krish@nvidia.com> # Conflicts: # tests/serve/test_sidecar.py
…a-dynamo Signed-off-by: krishung5 <krish@nvidia.com> # Conflicts: # lib/sidecar/trtllm/src/convert/request_tests.rs
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
tanmayv25
reviewed
Nov 3, 2026
tanmayv25
left a comment
Contributor
There was a problem hiding this comment.
Two focused items for freeze: reserve the internal routing UUID namespace, and preserve the existing media behavior and coverage when integrating these PRs.
Signed-off-by: krishung5 <krish@nvidia.com>
nv-tusharma
reviewed
Nov 3, 2026
tanmayv25
approved these changes
Nov 3, 2026
nv-tusharma
approved these changes
Nov 3, 2026
jthomson04
approved these changes
Nov 3, 2026
jthomson04
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Non-blocking notes inline.
…ing-dynamo Signed-off-by: krishung5 <krish@nvidia.com> # Conflicts: # .github/workflows/shared-test-trtllm-nightly-image.yml # lib/sidecar/trtllm/ci/run_image_test.py # lib/sidecar/trtllm/src/client.rs # lib/sidecar/trtllm/src/convert.rs # lib/sidecar/trtllm/src/convert/request_tests.rs # lib/sidecar/trtllm/src/tests/e2e.rs
Contributor
Preserve the image, video, audio, and image-routing nightly scenarios and synchronize the Python bindings lockfile. Signed-off-by: krishung5 <krish@nvidia.com>
Signed-off-by: krishung5 <krish@nvidia.com>
krishung5
enabled auto-merge (squash)
November 4, 2026 16:21
1 task done
1 task done
This was referenced Nov 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Enable image-aware KV routing for the TensorRT-LLM OpenEngine sidecar. Forward frontend image identities through the negotiated engine contract and normalize native KV events to the routing hashes Dynamo expects.
Key Changes
239 added Rust production lines relative to main, excluding tests, CI, comments, and blank lines.
The shared UUID constant makes
dynamo-kv-routera normal dependency; it was already linked throughdynamo-llm, so this introduces no new package or release feature. The root and Python bindings lockfiles both record the dependency. Flat Serde wire decoding avoids the extra buffered-sequence clone from nested untagged derives.Automatic UUIDs include the full media URL, including its query string. TensorRT-LLM combines UUID and content identity, so identical images behind rotating signed URLs get separate native KV chains. Stable caller image UUIDs restore native reuse for unchanged content across those URLs, but disable automatic image-routing hints. The sidecar rejects decoded media, so
--frontend-decodingis unsupported here. Routing URLs must reference stable content.Validation
--mediaand--scenarioselect image, video, and audio correctly; zero executed tests or ignored tests fail validation.The native routing scenario uses real Qwen3-VL weights, checks answers and native cache reuse/isolation, captures ZMQ events, and compares the complete converted hash chain with the real Rust frontend preprocessor. CPU fixture checks of that comparison accepted the exact chain and rejected extra or missing image tokens. CPU unit runs intentionally ignore seven external-runtime tests. These nightly scenarios explicitly select the image, video, audio, and routing-event tests.
Issues and Ongoing Work
Closes DIS-2972.
Pull Request Checklist
Summary by CodeRabbit
New Features
Improvements