Repository navigation
feat(mm-routing): integrate GLM-5.3-Flash image planner - #17121
Conversation
|
No actionable comments were generated in the recent review. WalkthroughThe change adds a GLM5-Next image-routing backend that counts image tokens and validates prompt layouts. Preprocessing uses validation results when building multimodal routing information. Tests cover configuration resolution, routing output, and invalid inputs. ChangesGLM5-Next Image Routing
Estimated code review effort: 3 (Moderate) | ~25 minutes Merge Risk:
|
e3d865a to
e15298f
Compare
|
/ok to test e15298f |
|
/ok to test e15298f |
✅ Dynamo PR CI passed — run 42915099478 (attempt 1) on
|
External PR triageBlocker: CI is green on the current head, but none of the requested CODEOWNER teams has reviewed since 2026-10-22 (6 days). Only the review bot has approved. Next action (@ai-dynamo/dynamo-frontend-codeowners @ai-dynamo/dynamo-multimodal-codeowners @ai-dynamo/dynamo-router-codeowners @ai-dynamo/dynamo-runtime-codeowners @ai-dynamo/dynamo-ops-codeowners): Review and approve, or request changes. Status as of 2026-10-28 for head |
|
/ok to test 3d54251 |
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
No P0 or P1 found. One P2 is inline. This review read head 2b49edb84e27b090f4036d279564e6725029e175 and its merge with main at 31d6751d28. The merge with the newer main tip d405ab1636 is also clean, and that commit changes none of the files below.
Findings:
- [P2]
tests/mm_router/test_router_rust_mm_glm5_next_e2e.py:386: the three SGLang tests fail by design on SGLang 0.5.23, which this branch pins. The inline comment has a tested gate.
Open items from other reviewers: none. All four review threads are resolved.
What this review ran and measured.
- Rust tests,
cargo test -p dynamo-llm --no-default-features --features mm-routing --locked --lib preprocessor::on macOS: the head passes 484, fails 0 and ignores 2. The merge withmain31d6751d28passes 488, fails 0 and ignores 2. The 7 new tests pass on both. cargo clippy -p dynamo-llm --no-default-features --features mm-routing --locked --all-targets --no-deps -- -D warningsis clean on the head and on the merge.- Bindings built with
maturin develop --uv --features mm-routing --locked:test_routing_image_token_id.pypasses 11 of 11. With the realconfig.jsonandprocessor_config.jsonofinference-optimization/GLM-5.3-Flash-2.6B-A2.3B-smallata2d7ca22e7,resolve_routing_image_token_idreturns154854. Withdo_center_crop: trueadded, it returnsNone. - For each new test file, a deliberate break of the code that it covers makes the file fail. Each break was restored afterwards:
| deliberate break | new tests that fail |
|---|---|
remove the glm5_next branch in ImageRoutingProcessor::try_new |
5 of 7 Rust tests, 2 of 11 binding tests |
remove the validate_image_prompt call in build_mm_exact_routing_info |
2 of the 3 tests in glm5_next_tests.rs |
make the adapter validate_image_prompt return Ok(()) |
3 of 7 Rust tests |
- CI lanes, by marker expression on
mainand on the merge: the 6 new E2E tests land only in nightlyvllm-h100-testandsglang-h100-test, in the sequential stage. The 11 binding tests land in thedynamo-pipelineCPU lane. No test ofmainchanges its lanes, and no test loses all of its lanes. tests/conftest.py, withQwen/Qwen3-0.6Bandcodelion/Qwen3-0.6B-accuracy-recovery-lora: a predownload ofrepo@<sha>resolves that revision offline, and a predownload of the default branch does not. A mark withoutrevision=callssnapshot_download(repo_id=..., revision=None), which is the default thatmainuses.- Dependencies: the 4 packages that both lock files add match their crates.io checksums. Every file in the published
dynamo-multimodal0.2.2 crate matches the same file at itsdynamo-multimodal-v0.2.2tag, except for theCargo.tomlthatcargo publishrewrites.cargo-deny0.19.0 with the Pre Merge arguments passes on the merge base, the head,mainand the merge, in both lock roots. RustSec and OSV list no advisory for the 4 packages. - CI on this head: 129 check runs, with 90 passed and 39 skipped. The two failures in attempt 1 of
PRrun 42071663260 passed on a rerun of the same SHA. The DGDR smoke job passed in attempt 2. The TRT-LLM amd64 job passed in attempt 4, withtest_deployment[aggregated_multimodal_router-2]passing. Pre Mergerust-tests (.)ran the 7 new Rust tests. Thedynamo-runtime / test / paralleljobs ran the 11 binding tests on amd64 and arm64. - Coverage facts: the nightly runs from 2026-10-13 to 2026-11-08 created 26 sets of vLLM and SGLang H100 jobs. Each of these jobs ended cancelled without a runner. The last H100 jobs with a runner ran on 2026-10-08, and they failed. No lane in
pr.yamlorpost-merge-ci.ymlnames a Hopper or Blackwell GPU. After this head mergedmain, #17862 movedmainto vLLM 0.33.0. The GB200 results posted on 2026-11-05 used vLLM 0.32.0. - This review did not run the E2E module. It needs Hopper or Blackwell GPUs.
|
thanks for reviewing folks! @ai-dynamo/dynamo-ops-codeowners @krishung5 can I have your help merging in this PR to prevent it from going stale? happy to address if you have more comments |
|
Hi @HanFa, could you address Dmitry's comments above? Would need a approval from the Ops team before this PR can be merged. Thanks! |
Head branch was pushed to by a user without write access
|
/ok to test 08255e9 |
Use dynamo-multimodal 0.2.2 for checkpoint-driven GLM image token counts and layout validation. Select the adapter before legacy flat-config loading and fall back to text-prefix routing when planning is unsupported or fails. Add Rust adapter and preprocessing tests plus real Python binding resolver tests with shared lightweight checkpoint fixtures. Three-backend routing E2E remains follow-up work. Validation: targeted preprocessor Rust tests passed (383 passed, 2 ignored); Python binding tests passed (11 passed); cargo fmt, Ruff, and git diff --check passed. Signed-off-by: Fang Han <fangh@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Rust planner tests and the Python resolver tests never run an engine, so they cannot catch the glm5_next planner and a worker disagreeing on image token expansion, or a hash mismatch between the frontend's routing blocks and the worker's KV events. Add an E2E module that drives the Rust frontend's KV router against a real vLLM or SGLang worker serving tiny-random/glm-5.3-flash, a 227 MB random checkpoint that keeps GLM-5.3-Flash's tokenizer, chat template, processor config and token ids. Per engine it asserts that the router's block total equals the engine's reported prompt length after its own expansion, that per-image token increments follow GLM geometry, that repeated images match nearly every published block and hit the engine's prefix cache, that a different image on the same prompt matches neither, and that two non-block-aligned images agree block for block with the second image's blocks invalidated exactly from the boundary on. GLM5-Next's DSA layers need sparse MLA attention, which vLLM 0.32 and SGLang 0.5.22 provide only on Hopper and Blackwell, so the module is nightly + h100 and runs in the vllm-h100-test and sglang-h100-test lanes. Exclude h100 tests from the nightly single-GPU vLLM lane the way the SGLang lane already does, so they do not land on its Ada runner. Signed-off-by: Fang Han <fangh@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first GPU run of the GLM E2E module, on GB200 in review, failed in two ways. tiny-random/glm-5.3-flash does not start: its kv_lora_rank of 256 is not a sparse-MLA head size vLLM accepts, and SGLang cannot load its 32x32 block-FP8 weights. With a BF16 checkpoint both engines serve, but they size KV blocks for GLM5-Next's hybrid KDA layers themselves (vLLM advertised 4352 tokens, SGLang 256) while the assertions assumed 64, and a 4096-token context cannot fill even one vLLM block. Switch to inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small pinned at a2d7ca22e78d. The workers load the pinned snapshot from a local path, so both engines and the frontend's planner read the same files. Read the block size the worker advertised from the frontend's model_kv_cache_block_size metric, and derive the image sizes and every block assertion from it. The large image covers about one and a half blocks and the small one just over a block, so every image prompt spans several blocks and a second image always reaches a block the first did not. Raise the context to 32768 tokens. Bound cold overlaps and engine reuse by the blocks the text alone fills, measured with a text-only request. Drop the kernel pins, which only the block-FP8 checkpoint needed, and the frontend's --kv-cache-block-size, which does not apply to discovered workers. Drop the "resolved image-placeholder token id" log assertion: GLM's placeholder comes from config.json's image_token_id, and that path does not log it. Signed-off-by: Fang Han <fangh@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review follow-up on 97296c2. The shared predownload fetched the checkpoint's default branch before it turned on HF_HUB_OFFLINE. Once that branch moves past the pinned revision, a clean runner would not find the pinned snapshot. The model mark now takes revision=: pytest_collection_modifyitems records it as repo_id@revision and download_models fetches that revision. The GLM module passes its pin and only looks the snapshot up in the cache. In the multi-image test, AC_cold must now also match more blocks than the text-only prompt fills. The old [A repeat - 1, A repeat] bound accepted zero when the A repeat matched a single block, as it does at GB200's 4352-token vLLM block, so losing the first image's cached blocks would have passed. _send now returns a RequestObservation and checks the worker's expansion against the router's block count before returning. That removes the tuple unpacking and the per-test expansion calls. Signed-off-by: Fang Han <fangh@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review follow-up on 2b49edb. SGLang 0.5.23, which pyproject.toml pins, carries sgl-project/sglang#43043: it fills image tokens with pad values of its own, and the SGLang publisher marks its workers as owning their multimodal identity. Since ai-dynamo#17126 the router cuts each KV store at the first block holding such a value and keeps that block and every later one out of its index. Every image prompt in this module holds image tokens in its first block, so image requests match no router blocks, and the repeat-overlap assertions fail by design in all three SGLang tests. The GB200 run on 2026-11-05 showed this. Skip the SGLang parameter when sglang_has_wide_mm_pad_values() reports the wide format, the probe tests/serve/multimodal_profiles/sglang.py already uses, until exact SGLang image routing lands in ai-dynamo#17082. The import stays in the fixture so collection never imports SGLang. SGLang 0.5.22 builds still run the three tests, and the vLLM tests are unchanged. Signed-off-by: Fang Han <fangh@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Head branch was pushed to by a user without write access
08255e9 to
82fe162
Compare
|
/ok to test 82fe162 |
harryskim
left a comment
There was a problem hiding this comment.
Docs review. No .md/.mdx files change, so these are comment and description notes only.
PR description: the Context/Validation sections read as a rebase log (commit-hash mappings, per-revision checks), and the test counts conflict (503 vs 405 passed on the same rebased base). A single current validation summary would be easier to review. The CodeRabbit summary also says "GLM-5 Next models" while the rest says GLM-5.3-Flash.
Also: if the repo has a multimodal-routing page or model support matrix, GLM-5.3-Flash (glm5_next) should be added there, or the PR should note that it is deferred.
Docs review follow-up on 82fe162. Name the files ImageRoutingProcessor::try_new needs: a glm5_next checkpoint reads config.json plus preprocessor_config.json, or processor_config.json when that file is absent, and every other model needs preprocessor_config.json. Say that on Err the preprocessor logs a warning and routes by text prefix, that only model_type "glm5_next" takes the GLM path, and that validate_image_prompt applies no constraints by default. In the GLM E2E module, describe the SGLang skip condition without version or PR numbers, which stay in the skip itself, and date the Hardware and Block size figures. List GLM-5.3-Flash in the vLLM row of the multimodal KV routing support matrix. vLLM 0.32.0 passed all three E2E tests on GB200. Signed-off-by: Fang Han <fangh@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed the docs review in d55e949 and in the description:
The new head |
harryskim
left a comment
There was a problem hiding this comment.
approved with the changes
|
/ok to test d55e949 |
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
No P0, P1 or P2 found. This review read head d55e949203a4b86db4c5f09df3ea23ede0b06b05 and its merge with main at 3e84909a1971a9fa7dcbd3ee5c739e6f5cf6da7c.
- [P3]
lib/llm/src/preprocessor/glm5_next_tests.rs:7: after a merge withmain, the import needscompat.rs. The fix is inline. - The earlier P2 about SGLang 0.5.23 is fixed in
82fe162c99. The results are in that thread. - Open: CI has not run the E2E module, because no nightly H100 job got a runner.
What this review measured.
- Nightly H100 jobs. None got a runner since 2026-10-13. The jobs of 2026-11-08 to 2026-11-10 were cancelled after 24 hours, and the jobs of 2026-11-11 still wait.
- Rebase.
git range-diffshows the three test commits unchanged. The first commit changed only context lines in the two lock files. Between the own diff of2b49edb84eand the own diff of this head, 47 lines in 3 files differ. All of them come from82fe162c99andd55e949203. On this head and on the merge, both lock files add the same 4 packages at the same versions and checksums. They change no other package, apart from the new dependency entry ofdynamo-llm. - Commit
d55e949203. With comments and docstrings removed,image.rsand the E2E module are the same before and after it. A planted one-character code change in each file shows up in the same comparison. The new doc comments inimage.rsmatch the code.spec_from_model_dirindynamo-multimodal0.2.2 readsconfig.jsonandpreprocessor_config.json. Ifpreprocessor_config.jsonis missing, it readsprocessor_config.jsoninstead. Onlymodel_typeglm5_nexttakes the GLM path. OnErr, the preprocessor logs a warning on targetmm_routingand routes without image awareness. - Support matrix. The GB200 results of 2026-11-05 used vLLM 0.32.0.
mainpins vLLM 0.33.0 since #17862, and the base of this head has it. The Rust code of this PR reads checkpoint metadata and calls nothing in vLLM. Exact routing also needs the worker to expand each image the same way. In vLLM,vllm/models/glm5next/common/multimodal.pyand its processing test are the same files in v0.32.0 and v0.33.0. No E2E run used vLLM 0.33.0. - Rust on macOS, with
--no-default-features --features mm-routing. Thepreprocessor::tests pass 503 on this head and 506 on the merge, with 0 failed and 2 ignored. The 7 GLM tests pass on both. Clippy with--all-targets --no-deps -- -D warningspasses on both. Withblock-manageron, macOS does not builddynamo-llm, and all errors are inlib/llm/src/block_manager/. CI'srust-clippy (.)linted that feature set on the merge withc2a6b68bc7. - Bindings on the merge, built with
maturin develop --uv --features mm-routing --locked.test_routing_image_token_id.pypasses 11 of 11. Withresolve_routing_image_token_idhidden, as in a wheel withoutmm-routing, the 11 tests skip and do not fail. After #18199, both branches ofcontainer/templates/wheel_builder.Dockerfilestill turn onmm-routing, and that file is the same on this head and on the merge. - Both lock roots of the merge pass
cargo tree --locked.cargo-deny0.19.0 with the CI arguments passes in both. - Log filter of the E2E module. Every test reads the
[ROUTING] Best:line. Onmain,dynamo_llm::kv_router::routing_host::kv_selectionemits it at debug level. With the module'sDYN_LOG, a frontend and onedynamo.mockerworker on the merge logged 2 such lines for 2 requests, andtests/utils/router_logs.pyparsed both. Withoutdynamo_llm::kv_router=debug, they logged none. In this run,dynamo_kv_router::scheduling=debugturned on no line after #18055. The othertests/mm_router/modules now usedynamo_scheduler=debug,dynamo_admission=debug,dynamo_worker_selection=debug. No test result reads those lines. - CI on this head: 133 check runs, 98 passed, 35 skipped, 0 failed.
rust-tests (.)ran the 7 new Rust tests, and bothdynamo-runtime / test / paralleljobs ran the 11 binding tests. All passed. No CI job built this head with #18055. Pre Merge tested the merge withmainatc2a6b68bc7, which has #18199 but not #18055.
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
Approving head d55e949203a4b86db4c5f09df3ea23ede0b06b05.
The earlier P2 is fixed in 82fe162c99. On SGLang builds with wide pad values, the 3 SGLang E2E tests skip. On older builds, they still start the worker. The rebase changed only context lines in the lock files, and d55e949203 changes only comments, docstrings and one docs line. On macOS, with --features mm-routing, the Rust tests and clippy pass on this head and on its merge with main at 3e84909a19. The 11 binding tests pass on that merge, and CI ran them on this head on amd64 and arm64. Both lock roots of the merge pass cargo tree --locked and cargo-deny.
Open: the P3 at lib/llm/src/preprocessor/glm5_next_tests.rs:7. After a merge with main, the import resolves only through compat.rs, which 1.10 removes. Its thread blocks the merge until someone resolves it.
Not run: the E2E module, which CI selects only in the nightly H100 lanes. From 2026-10-13 to 2026-11-11, no nightly H100 job got a runner, so CI has not run the module. No CI job built this head with #18055, so the merge with the router crate split was tested only on macOS.
Summary
Connect GLM-5.3-Flash image routing to the default Rust frontend through the shared
glm5_nextplanner. Use checkpoint metadata for image token counts and validate the rendered prompt layout before constructing exact routing tokens. Unsupported configuration, accounting failures, and layout failures retain text-prefix routing.Details
dynamo-multimodaldependency to0.2.2with default features disabled; update both Cargo lockfiles.processor_config.jsoncheckpoints are supported.tests/mm_router/test_router_rust_mm_glm5_next_e2e.py: nightly H100 E2E that drives the Rust frontend's KV router against a real vLLM or SGLang worker servinginference-optimization/GLM-5.3-Flash-2.6B-A2.3B-smallpinned ata2d7ca22e78d(a 5 GB BF16 cut of GLM-5.3-Flash that keeps its tokenizer, chat template, processor config and token ids; the shared predownload fetches that revision and the workers load the snapshot from a local path). The engines size KV blocks for the model's hybrid KDA layers themselves, so the module reads the block size the worker registered from the frontend'sdynamo_frontend_model_kv_cache_block_sizemetric and derives the image sizes and every block assertion from it. Per engine it checks worker expansion (router block total vs the engine's reported prompt length, per-image increments following GLM geometry), repeated-image cache overlap plus the engine's prefix-cache hit, different-image isolation at the router and in the engine cache, and two images whose runs straddle block boundaries, where swapping the second image must keep the first image's blocks. On SGLang builds with wide multimodal pad values (0.5.23, whichmainpins), the router keeps image-bearing blocks out of its index since fix(mm-routing): handle SGLang engine-owned multimodal pad values #17126, so the SGLang tests skip there until feat(mm-routing): restore SGLang multimodal KV routing with routing-label spans #17082. Excludeh100tests from the nightly single-GPU vLLM lane the way the SGLang lane already does.tests/conftest.py:@pytest.mark.model(repo_id, revision=...)pins the shared predownload to that revision, so a test that loads a fixed revision finds it cached onceHF_HUB_OFFLINEis set. The collected entry readsrepo_id@revision, anddownload_modelspasses the revision tosnapshot_download. Marks withoutrevision=download the default branch as before.docs/fern/pages/use-cases/image-understanding/vlm-performance-optimization/multimodal-kv-routing.md.Context
Follow-up to #16947, now merged into
main. The branch is six commits onmainat72ca6717d: the GLM integration commit, the E2E module requested in review, its rework after the first GPU run (the original checkpoint did not start, and the engines do not use 64-token KV blocks for this model), the review follow-up for that rework, the SGLang skip on wide-pad builds, and the docs review follow-up.Supersedes the temporary stacked review at HanFa#3.
Where should the reviewer start?
Start with
lib/llm/src/preprocessor/mm_routing/glm5_next.rs, then the adapter selection and layout-validation call sites.lib/llm/src/preprocessor/glm5_next_tests.rsexercises the preprocessing boundary;lib/bindings/python/tests/test_routing_image_token_id.pyexercises the existing Rust resolver exposed to Python.tests/mm_router/test_router_rust_mm_glm5_next_e2e.pyis the engine-backed check of the same contract.Validation
Current as of
d55e94920, onmainat72ca6717d:cargo test -p dynamo-llm --no-default-features --features mm-routing --locked --lib preprocessor::passes 503 tests with 2 ignored, andcargo clippywith the same features and-D warningsis clean. Both lockfiles pindynamo-multimodal0.2.2 and passcargo metadata --locked.82fe162c9;d55e94920changes only comments, docstrings and one docs line. Thedynamo-runtime / test / paralleljobs ran the 11 binding resolver tests inlib/bindings/python/tests/test_routing_image_token_id.py, and all passed.mainpins, kept its engine cache hits, but the router matched no image-bearing blocks (fix(mm-routing): handle SGLang engine-owned multimodal pad values #17126), so the module skips SGLang on such builds until feat(mm-routing): restore SGLang multimodal KV routing with routing-label spans #17082. The module needs Hopper or Blackwell for sparse MLA attention, so it runs only in the nightlyvllm-h100-testandsglang-h100-testlanes; per review, those lanes had no runner from 2026-10-13 to 2026-11-08.pytest --collect-onlycollects 6 tests. The two H100 lane expressions select 3 each, and no other lane selects any. With the realsglang_has_wide_mm_pad_values()against stub SGLang schemas, the 0.5.23 schema skips the 3 SGLang tests and the 0.5.22 schema runs them.The support matrix lists GLM-5.3-Flash for vLLM, where the E2E passed. TensorRT-LLM coverage and runs with the full model remain follow-up work.
Related Issues
Part of #16948.
Internal tracking: DIS-3123 (multimodal-aware KV routing model support).
Consumes the shared planner from ai-dynamo/frontend-crates#412, released in
dynamo-multimodal0.2.2.Summary by CodeRabbit
glm5_next) models, using checkpoint settings to determine image token counts and routing information.