Skip to content

chore(trtllm): bump sidecar examples to image-routing TRT-LLM and Dynamo nightlies - #18765

Merged
KrishnanPrash merged 3 commits into
mainfrom
tanmayv/dyn-5534-bump-trtllm-pin
Nov 16, 2026
Merged

KrishnanPrash merged 3 commits into
mainfrom
tanmayv/dyn-5534-bump-trtllm-pin

Conversation

@tanmayv25

@tanmayv25 tanmayv25 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Overview

The TRT-LLM sidecar examples could not do image-aware KV routing. Their pins predate the feature on both sides:

Images still worked, but the router never credited cached image blocks, so a repeated image landed on whichever worker had the lowest load.

Key Changes

  • TensorRT-LLM engine image: 1.4.0rc4.dev202611130009 in all six manifests, the launch-script comments, and the README.
  • Dynamo frontend, sidecar, and runtime images: 20261113-b43cd4a in all six manifests and the README.
  • README: lists image-aware KV routing among the capabilities of the upstream nightly, and drops the separate audio pin, because the example pin includes audio support.

Only image pins and docs change; there are no code changes.

Validation

These ran on a Kubernetes cluster with A100 GPUs, with the Dynamo operator from the 20261113 nightly.

Image-aware routing. I ran two copies of agg_kv_router.yaml (2 workers each), with the model changed to Qwen/Qwen3-VL-2B-Instruct. Both used the new Dynamo images; only the TensorRT-LLM image differed. Each copy received image X, then image Y, then image X five times with different text. Worker and hit rate come from nvext.extra_fields: ["worker_id", "timing"].

Repeats of image X New TRT-LLM (…dev202611130009) Old TRT-LLM (…dev202610290007)
Router kv_hit_rate 0.86 on all 5 0.0 on all 5
Worker chosen always the worker that already held X 3 on that worker, 2 on the other
Engine cached_tokens 201 of ~212 every time 201 only when it landed on the right worker, 4 otherwise

The old runtime logged no warning while it dropped the image events; #18501 adds that warning.

Examples on the new pins. All four single-node manifests became Ready and served Qwen/Qwen3-0.6B. Each run checked both a chat request and a streaming request:

Manifest Chat Streaming
agg.yaml HTTP 200 7 chunks
agg_kv_router.yaml HTTP 200 7 chunks, frontend connected to both workers' KV event streams
disagg.yaml HTTP 200 8 chunks
disagg_kv_router.yaml HTTP 200 8 chunks, KV event streams connected

The four launch scripts in launch/ ran unmodified in a 4-GPU pod built from the new TensorRT-LLM image, with DYN_DISCOVERY_BACKEND=file set as the README instructs. All four returned HTTP 200. The launch scripts pin no Dynamo images.

Not tested: multinode_tp.yaml and multinode_dp.yaml. The test cluster's DynamoGraphDeployment CRD has no spec.components[].roles, so the API server rejects these manifests.

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

Summary by CodeRabbit

  • Documentation
    • Updated TensorRT-LLM and Dynamo versions across the deployment examples.
    • Clarified that the standalone example supports image-aware KV routing.
    • Updated the audio example’s runtime version reference.

The previous pins (1.4.0rc3.dev202610290007 for examples and
1.4.0rc3.dev202610300406 for audio) predate NVIDIA/TensorRT-LLM#21125, so
image KV events carry no image-routing metadata and image-aware KV routing
silently falls back to text-prefix routing. The new nightly (TRT-LLM
58042459) includes #21125 and audio support, so one pin covers every example.

Refs DYN-5534

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
The examples pinned Dynamo nightlies from 20261013/20261023, which predate
image-aware KV routing for OpenEngine sidecars (#17437). Move the frontend,
sidecar, and runtime images to 20261113-b43cd4a so the examples pair the
image-routing TensorRT-LLM nightly with a Dynamo that consumes its events.

Refs DYN-5534

Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
@tanmayv25
tanmayv25 requested review from a team as code owners November 14, 2026 02:44
@github-actions github-actions Bot added chore documentation Improvements or additions to documentation labels Nov 14, 2026
@coderabbitai

coderabbitai Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The README, launch comments, and deployment manifests now reference newer TensorRT-LLM and Dynamo images. The README also updates the listed example capabilities and audio runtime reference.

Changes

TensorRT-LLM Image Pins

Layer / File(s) Summary
Deployment image references
lib/sidecar/trtllm/deploy/*.yaml
The deployment manifests update TensorRT-LLM and Dynamo image tags. The aggregate and KV-router manifests also update pinned image digests.
Example and launcher references
lib/sidecar/trtllm/README.md, lib/sidecar/trtllm/launch/*.sh
The README updates example image pins and capability descriptions. Launcher comments update the documented TensorRT-LLM image tag.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: Minimal · up to 0a974

This updates example image pins and documentation to versions that support image-aware KV routing. No concrete defects were identified. The multinode examples were not exercised on the test cluster, so deployers of those manifests should validate them in a compatible environment.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: bumping the TRT-LLM sidecar examples to the TensorRT-LLM and Dynamo nightlies that support image-aware routing. The 'chore(trtllm)' prefix is a conventional commit sty…
Description check Passed The description follows the repository template. It has Overview, Key Changes, Validation, and Issues and Ongoing Work sections, and it includes the checklist. The Overview explains the reason for the…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: bumping the TRT-LLM sidecar examples to the TensorRT-LLM and Dynamo nightlies that support image-aware routing. The 'chore(trtllm)' prefix is a conventional commit style and does not hurt clarity. The title is 82 characters, which is slightly long but still readable, and it matches the changeset, which only updates image pins and docs.

Full details: Description check

Explanation

The description follows the repository template. It has Overview, Key Changes, Validation, and Issues and Ongoing Work sections, and it includes the checklist. The Overview explains the reason for the pin bumps. Key Changes lists the image updates and README changes. Validation gives detailed Kubernetes and launch-script test results and states what was not tested (multinode manifests). The issue link is given as 'Refs DYN-5534' rather than 'Closes #XXXX', which is acceptable because the template's issue-link check is advisory. The checklist box is unchecked.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 2cac4d7 and 0a974c2.

Files selected for processing: 11

  • lib/sidecar/trtllm/README.md
  • lib/sidecar/trtllm/deploy/agg.yaml
  • lib/sidecar/trtllm/deploy/agg_kv_router.yaml
  • lib/sidecar/trtllm/deploy/disagg.yaml
  • lib/sidecar/trtllm/deploy/disagg_kv_router.yaml
  • lib/sidecar/trtllm/deploy/multinode_dp.yaml
  • lib/sidecar/trtllm/deploy/multinode_tp.yaml
  • lib/sidecar/trtllm/launch/agg.sh
  • lib/sidecar/trtllm/launch/agg_kv_router.sh
  • lib/sidecar/trtllm/launch/disagg.sh
  • lib/sidecar/trtllm/launch/disagg_kv_router.sh

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@devin-ai-integration

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 43547943984 (attempt 1) on 0a974c2694

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 43547943984. Updated on every full-CI run of this PR.

Comment thread lib/sidecar/trtllm/README.md Resolved
Signed-off-by: Krishnan Prashanth <140860868+KrishnanPrash@users.noreply.github.com>
@KrishnanPrash
KrishnanPrash enabled auto-merge (squash) November 16, 2026 14:29
@KrishnanPrash
KrishnanPrash merged commit 8de0d42 into main Nov 16, 2026
109 checks passed
@KrishnanPrash
KrishnanPrash deleted the tanmayv/dyn-5534-bump-trtllm-pin branch November 16, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore documentation Improvements or additions to documentation size/M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants