Repository navigation
chore(trtllm): bump sidecar examples to image-routing TRT-LLM and Dynamo nightlies - #18765
Conversation
The previous pins (1.4.0rc3.dev202610290007 for examples and 1.4.0rc3.dev202610300406 for audio) predate NVIDIA/TensorRT-LLM#21125, so image KV events carry no image-routing metadata and image-aware KV routing silently falls back to text-prefix routing. The new nightly (TRT-LLM 58042459) includes #21125 and audio support, so one pin covers every example. Refs DYN-5534 Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
The examples pinned Dynamo nightlies from 20261013/20261023, which predate image-aware KV routing for OpenEngine sidecars (#17437). Move the frontend, sidecar, and runtime images to 20261113-b43cd4a so the examples pair the image-routing TensorRT-LLM nightly with a Dynamo that consumes its events. Refs DYN-5534 Signed-off-by: tanmayv25 <tanmay2592@gmail.com>
|
No actionable comments were generated in the recent review. WalkthroughThe README, launch comments, and deployment manifests now reference newer TensorRT-LLM and Dynamo images. The README also updates the listed example capabilities and audio runtime reference. ChangesTensorRT-LLM Image Pins
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk:
|
✅ Dynamo PR CI passed — run 43547943984 (attempt 1) on
|
Signed-off-by: Krishnan Prashanth <140860868+KrishnanPrash@users.noreply.github.com>
Overview
The TRT-LLM sidecar examples could not do image-aware KV routing. Their pins predate the feature on both sides:
1.4.0rc3.dev202610290007, which predates NVIDIA/TensorRT-LLM#21125. Without it, the engine reports notrtllm_image_routing_versionand its KV events have no image-routing metadata, so the router drops the image block stores.20261013-2210b07and20261023-75d0686nightlies, which predate feat(trtllm): add image-aware KV routing to OpenEngine sidecars #17437 (image-aware KV routing for OpenEngine sidecars).Images still worked, but the router never credited cached image blocks, so a repeated image landed on whichever worker had the lowest load.
Key Changes
1.4.0rc4.dev202611130009in all six manifests, the launch-script comments, and the README.20261113-b43cd4ain all six manifests and the README.Only image pins and docs change; there are no code changes.
Validation
These ran on a Kubernetes cluster with A100 GPUs, with the Dynamo operator from the
20261113nightly.Image-aware routing. I ran two copies of
agg_kv_router.yaml(2 workers each), with the model changed toQwen/Qwen3-VL-2B-Instruct. Both used the new Dynamo images; only the TensorRT-LLM image differed. Each copy received image X, then image Y, then image X five times with different text. Worker and hit rate come fromnvext.extra_fields: ["worker_id", "timing"].…dev202611130009)…dev202610290007)kv_hit_ratecached_tokensThe old runtime logged no warning while it dropped the image events; #18501 adds that warning.
Examples on the new pins. All four single-node manifests became Ready and served Qwen/Qwen3-0.6B. Each run checked both a chat request and a streaming request:
agg.yamlagg_kv_router.yamldisagg.yamldisagg_kv_router.yamlThe four launch scripts in
launch/ran unmodified in a 4-GPU pod built from the new TensorRT-LLM image, withDYN_DISCOVERY_BACKEND=fileset as the README instructs. All four returned HTTP 200. The launch scripts pin no Dynamo images.Not tested:
multinode_tp.yamlandmultinode_dp.yaml. The test cluster's DynamoGraphDeployment CRD has nospec.components[].roles, so the API server rejects these manifests.Issues and Ongoing Work
Pull Request Checklist
/pr-checklistskill.Summary by CodeRabbit