Skip to content

chore(trtllm): [cherry-pick 1.7.0] bump sidecar examples to image-routing TRT-LLM and Dynamo 1.7.0 images (#18765) - #18767

Open
tanmayv25 wants to merge 2 commits into
release/1.7.0from
tanmayv/dyn-5534-bump-trtllm-pin-1.7.0
Open

tanmayv25 wants to merge 2 commits into
release/1.7.0from
tanmayv/dyn-5534-bump-trtllm-pin-1.7.0

Conversation

@tanmayv25

Copy link
Copy Markdown
Contributor

Overview

This is the 1.7.0 version of #18765. The 1.7.0 TRT-LLM sidecar examples could not do image-aware KV routing:

#18765 is not merged yet. This PR differs from it on purpose: main uses the 20261113-b43cd4a nightlies, while this branch uses the 1.7.0 release images.

Key Changes

  • Commit 1 (cherry-pick of chore(trtllm): bump sidecar examples to image-routing TRT-LLM and Dynamo nightlies #18765's first commit): the TensorRT-LLM engine image becomes 1.4.0rc4.dev202611130009 in all six manifests, the launch-script comments, and the README. The README also lists image-aware KV routing among the pin's capabilities and drops the separate audio pin.
  • Commit 2 (1.7.0 only):
    • The Dynamo frontend, sidecar, and runtime images become nvcr.io/nvidia/ai-dynamo/{dynamo-frontend,dynamo-sidecar,tensorrtllm-runtime}:1.7.0. This follows the branch's release: set version 1.7.0 convention: the tags are not published yet and have no digest.
    • runtimeVersionOverride becomes "1.7.0". The override wins over the image tag, so leaving it at "1.6.0" would make the operator treat these images as 1.6.0.
    • Comments and README lines that said "nightly" now refer to the 1.7.0 images.

Only image pins, the version override, and docs change.

Validation

All runs were on a Kubernetes cluster with A100 GPUs. The :1.7.0 tags do not exist yet, so the test copies of the manifests used the matching 1.7.0-rc.3 staging images: dynamo-frontend, dynamo-sidecar, and tensorrtllm-runtime. The frontend and sidecar are the same amd64 digests QA reported in DYN-5534. The test copies also kept the earlier runtimeVersionOverride: "1.6.0". The 1.7.0 operator has no runtime feature gate above 1.6, so the version change does not alter rendered behavior.

Image-aware routing on rc.3. I ran agg_kv_router.yaml (2 workers) with Qwen/Qwen3-VL-2B-Instruct. It received image X, then image Y, then image X five times with different text:

Request Worker Router kv_hit_rate Engine cached_tokens
X (first) A 0.00 0 / 210
Y B 0.00 0 / 210
X, repeated 5 times A every time 0.86 every time 201 of ~212 every time

With the old TensorRT-LLM pin, the same check on main gave kv_hit_rate 0.0 for every repeat, and two of the five repeats landed on the other worker (#18765).

Single-node examples on rc.3. All four became Ready and served Qwen/Qwen3-0.6B. Each run checked a chat request and a streaming request:

Manifest Chat Streaming
agg.yaml HTTP 200 7 chunks
agg_kv_router.yaml HTTP 200 7 chunks
disagg.yaml HTTP 200 8 chunks
disagg_kv_router.yaml HTTP 200 8 chunks

Not tested:

  • multinode_tp.yaml and multinode_dp.yaml: the test cluster's DynamoGraphDeployment CRD has no spec.components[].roles.
  • The 1.7.0 operator: the runs used a main nightly operator (20261113). The disagg.yaml comment says the images were "validated together", and that does not include the 1.7.0 operator.
  • The launch scripts against 1.7.0 Dynamo: they are identical to main apart from pins, and they passed only on main (chore(trtllm): bump sidecar examples to image-routing TRT-LLM and Dynamo nightlies #18765).
  • Audio on the new TensorRT-LLM pin. That pin is newer than the audio pin the README used before.
  • The newest RCs: the staging registry now has up to 1.7.0-rc.7. These checks ran on rc.3, the RC QA reported against.

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

The previous pins (1.4.0rc3.dev202610290007 for examples and
1.4.0rc3.dev202610300406 for audio) predate NVIDIA/TensorRT-LLM#21125, so
image KV events carry no image-routing metadata and image-aware KV routing
silently falls back to text-prefix routing. The new nightly (TRT-LLM
58042459) includes #21125 and audio support, so one pin covers every example.

Refs DYN-5534

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
The examples pinned Dynamo nightlies from 20261013/20261023, built from
main before image-aware KV routing (#17437, cherry-picked to 1.7.0 in
#18089). Point the frontend, sidecar, and runtime images at the 1.7.0
release tags so the examples pair the image-routing TensorRT-LLM nightly
with the Dynamo release that consumes its events.

Refs DYN-5534

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@tanmayv25
tanmayv25 requested review from a team as code owners November 14, 2026 03:20
@github-actions github-actions Bot added chore documentation Improvements or additions to documentation labels Nov 14, 2026
@tanmayv25 tanmayv25 added the cherry-pick:requested Fix requested for the release branch in code freeze label Nov 14, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 43551128814 (attempt 1) on be83c0bfc8

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 43551128814. Updated on every full-CI run of this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cherry-pick:requested Fix requested for the release branch in code freeze chore documentation Improvements or additions to documentation size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant