Repository navigation
Conversation
The previous pins (1.4.0rc3.dev202610290007 for examples and 1.4.0rc3.dev202610300406 for audio) predate NVIDIA/TensorRT-LLM#21125, so image KV events carry no image-routing metadata and image-aware KV routing silently falls back to text-prefix routing. The new nightly (TRT-LLM 58042459) includes #21125 and audio support, so one pin covers every example. Refs DYN-5534 Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
The examples pinned Dynamo nightlies from 20261013/20261023, built from main before image-aware KV routing (#17437, cherry-picked to 1.7.0 in #18089). Point the frontend, sidecar, and runtime images at the 1.7.0 release tags so the examples pair the image-routing TensorRT-LLM nightly with the Dynamo release that consumes its events. Refs DYN-5534 Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
Contributor
✅ Dynamo PR CI passed — run 43551128814 (attempt 1) on
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This is the 1.7.0 version of #18765. The 1.7.0 TRT-LLM sidecar examples could not do image-aware KV routing:
1.4.0rc3.dev202610290007, which predates NVIDIA/TensorRT-LLM#21125. Without it, the engine reports notrtllm_image_routing_versionand its KV events have no image-routing metadata, so the router drops the image block stores. QA found this on 1.7.0-rc.3 (DYN-5534).20261013-2210b07and20261023-75d0686nightlies frommain. Those predate feat(trtllm): add image-aware KV routing to OpenEngine sidecars #17437, which reached 1.7.0 through feat(trtllm): [cherry-pick 1.7.0] image-aware OpenEngine KV routing (#17437) #18089.#18765 is not merged yet. This PR differs from it on purpose:
mainuses the20261113-b43cd4anightlies, while this branch uses the 1.7.0 release images.Key Changes
1.4.0rc4.dev202611130009in all six manifests, the launch-script comments, and the README. The README also lists image-aware KV routing among the pin's capabilities and drops the separate audio pin.nvcr.io/nvidia/ai-dynamo/{dynamo-frontend,dynamo-sidecar,tensorrtllm-runtime}:1.7.0. This follows the branch'srelease: set version 1.7.0convention: the tags are not published yet and have no digest.runtimeVersionOverridebecomes"1.7.0". The override wins over the image tag, so leaving it at"1.6.0"would make the operator treat these images as 1.6.0.Only image pins, the version override, and docs change.
Validation
All runs were on a Kubernetes cluster with A100 GPUs. The
:1.7.0tags do not exist yet, so the test copies of the manifests used the matching 1.7.0-rc.3 staging images:dynamo-frontend,dynamo-sidecar, andtensorrtllm-runtime. The frontend and sidecar are the same amd64 digests QA reported in DYN-5534. The test copies also kept the earlierruntimeVersionOverride: "1.6.0". The 1.7.0 operator has no runtime feature gate above 1.6, so the version change does not alter rendered behavior.Image-aware routing on rc.3. I ran
agg_kv_router.yaml(2 workers) withQwen/Qwen3-VL-2B-Instruct. It received image X, then image Y, then image X five times with different text:kv_hit_ratecached_tokensWith the old TensorRT-LLM pin, the same check on
maingavekv_hit_rate0.0 for every repeat, and two of the five repeats landed on the other worker (#18765).Single-node examples on rc.3. All four became Ready and served Qwen/Qwen3-0.6B. Each run checked a chat request and a streaming request:
agg.yamlagg_kv_router.yamldisagg.yamldisagg_kv_router.yamlNot tested:
multinode_tp.yamlandmultinode_dp.yaml: the test cluster's DynamoGraphDeployment CRD has nospec.components[].roles.mainnightly operator (20261113). Thedisagg.yamlcomment says the images were "validated together", and that does not include the 1.7.0 operator.mainapart from pins, and they passed only onmain(chore(trtllm): bump sidecar examples to image-routing TRT-LLM and Dynamo nightlies #18765).Issues and Ongoing Work
Pull Request Checklist
/pr-checklistskill.