Skip to content

fix(router): write SGLang wide image pads in MM routing tokens - #17048

Closed
KrishnanPrash wants to merge 1 commit into
kprashanth/dis-3202-sglang-wide-padsfrom
kprashanth/dis-3202-sglang-request-pads
Closed

KrishnanPrash wants to merge 1 commit into
kprashanth/dis-3202-sglang-wide-padsfrom
kprashanth/dis-3202-sglang-request-pads

Conversation

@KrishnanPrash

Copy link
Copy Markdown
Contributor

Summary

On SGLang main, the KV router never sees the cache of a repeated image, so it can send the repeat to a worker that must compute it again. SGLang now derives four wide pads from the hash that Dynamo forwards and writes them over each image placeholder run (sgl-project/sglang#43043). The frontend still writes one constant pad, so request block hashes never match KV-event block hashes. The SGLang worker now publishes the pad scheme wide_v1 when its engine derives pads from forwarded hashes, and the frontend writes the same four pads, folded as the KV events fold them. A worker that publishes no scheme, such as one on the pinned SGLang 0.5.22, keeps the single pad.

  • The worker decides: only the worker knows the pad scheme of its SGLang. An unknown value gives the single pad and a warning.
  • Same derivation: the frontend runs SHA-256 over the same tag and hash bytes as SGLang and restarts the cycle on each run. A unit test pins vectors from SGLang itself.
  • One prompt per WorkerSet: the scheme joins the cohort contract of the Qwen video contracts, so a set that disagrees uses the single pad.
  • Scope: video and mixed image and video requests keep the single pad and still miss on SGLang main, as they do today.

Validation

Two python -m dynamo.sglang workers (Qwen3-VL-2B) with the KV router on SGLang main 5e79333c95, same client script on the parent PR and on this PR:

before: image repeats kv_hit_rate 0.0, 3 of 5 repeats routed to the other worker
after:  both worker cards publish "sglang_mm_pad_scheme":"wide_v1"
after:  every image repeat kv_hit_rate 0.929 on the same worker, cached_tokens 208
both:   text repeat kv_hit_rate 0.968, 0 KV-event decode warnings

dynamo-llm with mm-routing: 3571 passed. The SGLang protocol test passes on 0.5.22 and on SGLang main.

Related Issues

Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
@KrishnanPrash
KrishnanPrash requested review from a team as code owners October 21, 2026 21:32
@github-actions github-actions Bot added fix backend::sglang Relates to the sglang backend frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Oct 21, 2026
)
.unwrap();

// SGLang `padding_sequence`: p0 p1 p2 p3 p0 ..., from p0 on each run.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment merely narrates the sequence shown by the immediately following expected array, while the test name already states that pads restart for each image run. It adds no lasting constraint beyond the executable assertion.

🤖 AI Fix

Remove this redundant comment.

@devin-ai-integration

Copy link
Copy Markdown
Contributor

❌ Dynamo PR CI failed — run 39679650390 (attempt 1) on f6235add03

Gate checks: ❌ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Framework 1-GPU arm64
vLLM ❌ 1

Failure details

1 job failed: the vLLM arm64 CPU-only pytest stage was cut off midway by a self-hosted runner container-exec error. No test failed, so this looks like infrastructure.

❌ vllm-runtime / Test cuda13.0, arm64: runner "custom container implementation failed" during pytest

Job: vllm-runtime / Test cuda13.0, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 119509197266 -R ai-dynamo/dynamo --log-failed

[gw3] [ 46%] PASSED components/src/dynamo/vllm/tests/test_vllm_unit.py::TestBenchmarkConfig::test_benchmark_config_kwargs_unpack
components/src/dynamo/vllm/tests/test_vllm_unit.py::TestBenchmarkConfig::test_benchmark_operational_controls_reach_scheduler_config
##[error]Executing the custom container implementation failed. Please contact your self hosted runner administrator.
...
❌ pre_merge_cpu: step outcome 'failure'
✅ pre_merge_cpu_e2e: pytest exited 0
⏭️  pre_merge_gpu_parallel: did not run
⏭️  pre_merge_gpu: did not run
##[error]A pytest stage did not record a passing result. See this step's log for the stage.

The pre_merge_cpu pytest run stopped at 46% with every test before that passing. The runner's Kubernetes container hook then reported Executing the custom container implementation failed, so the stage never recorded a result. The later pre_merge_cpu_e2e stage passed, and the GPU stages did not run. No pytest failure or assertion appears, which points to a runner/infrastructure failure. Rerunning the failed jobs will likely be enough.

For agents
{"pr": 17048, "run_id": 39679650390, "run_attempt": 1, "head_sha": "f6235add03388297e8d68843d4a6fbc6776b7d35",
 "failures": [{"job": "vllm-runtime / Test cuda13.0, arm64", "job_id": 119509197266, "failed_step": "Run CPU-only tests (parallelized)", "signature": "##[error]Executing the custom container implementation failed (runner container exec aborted pre_merge_cpu pytest at 46%)", "tests": [], "log_cmd": "gh run view --job 119509197266 -R ai-dynamo/dynamo --log-failed"}]}

Posted automatically by Devin for run 39679650390. Updated on every full-CI run of this PR.

@KrishnanPrash

Copy link
Copy Markdown
Contributor Author

Closing in favor of the DEP #17044 work in #17126, #17082 and sgl-project/sglang#45737.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::sglang Relates to the sglang backend fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant