Skip to content

feat(mm-routing): integrate GLM-5.3-Flash image planner - #17121

Merged
krishung5 merged 6 commits into
ai-dynamo:mainfrom
HanFa:codex/dis-3123-glm-image-routing-upstream
Nov 12, 2026
Merged

krishung5 merged 6 commits into
ai-dynamo:mainfrom
HanFa:codex/dis-3123-glm-image-routing-upstream

Conversation

@HanFa

@HanFa HanFa commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Connect GLM-5.3-Flash image routing to the default Rust frontend through the shared glm5_next planner. Use checkpoint metadata for image token counts and validate the rendered prompt layout before constructing exact routing tokens. Unsupported configuration, accounting failures, and layout failures retain text-prefix routing.

Details

  • Pin the optional dynamo-multimodal dependency to 0.2.2 with default features disabled; update both Cargo lockfiles.
  • Select the GLM adapter before the legacy flat-config loader so nested processor_config.json checkpoints are supported.
  • Delegate image counts and layout validation to the shared planner. Expand image placeholders using the existing repeated-pad path, retaining delimiters and the worker-bound prompt.
  • Add Rust coverage for checkpoint-driven geometry, single/multiple images, layout parity, content-sensitive routing tokens, and fallback without partial routing/hash installation.
  • Add real Python binding resolver tests using shared lightweight checkpoint fixtures, including nested/flat configs and unsupported or malformed metadata.
  • Add tests/mm_router/test_router_rust_mm_glm5_next_e2e.py: nightly H100 E2E that drives the Rust frontend's KV router against a real vLLM or SGLang worker serving inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small pinned at a2d7ca22e78d (a 5 GB BF16 cut of GLM-5.3-Flash that keeps its tokenizer, chat template, processor config and token ids; the shared predownload fetches that revision and the workers load the snapshot from a local path). The engines size KV blocks for the model's hybrid KDA layers themselves, so the module reads the block size the worker registered from the frontend's dynamo_frontend_model_kv_cache_block_size metric and derives the image sizes and every block assertion from it. Per engine it checks worker expansion (router block total vs the engine's reported prompt length, per-image increments following GLM geometry), repeated-image cache overlap plus the engine's prefix-cache hit, different-image isolation at the router and in the engine cache, and two images whose runs straddle block boundaries, where swapping the second image must keep the first image's blocks. On SGLang builds with wide multimodal pad values (0.5.23, which main pins), the router keeps image-bearing blocks out of its index since fix(mm-routing): handle SGLang engine-owned multimodal pad values #17126, so the SGLang tests skip there until feat(mm-routing): restore SGLang multimodal KV routing with routing-label spans #17082. Exclude h100 tests from the nightly single-GPU vLLM lane the way the SGLang lane already does.
  • tests/conftest.py: @pytest.mark.model(repo_id, revision=...) pins the shared predownload to that revision, so a test that loads a fixed revision finds it cached once HF_HUB_OFFLINE is set. The collected entry reads repo_id@revision, and download_models passes the revision to snapshot_download. Marks without revision= download the default branch as before.
  • List GLM-5.3-Flash in the vLLM row of the support matrix in docs/fern/pages/use-cases/image-understanding/vlm-performance-optimization/multimodal-kv-routing.md.

Context

Follow-up to #16947, now merged into main. The branch is six commits on main at 72ca6717d: the GLM integration commit, the E2E module requested in review, its rework after the first GPU run (the original checkpoint did not start, and the engines do not use 64-token KV blocks for this model), the review follow-up for that rework, the SGLang skip on wide-pad builds, and the docs review follow-up.

Supersedes the temporary stacked review at HanFa#3.

Where should the reviewer start?

Start with lib/llm/src/preprocessor/mm_routing/glm5_next.rs, then the adapter selection and layout-validation call sites. lib/llm/src/preprocessor/glm5_next_tests.rs exercises the preprocessing boundary; lib/bindings/python/tests/test_routing_image_token_id.py exercises the existing Rust resolver exposed to Python. tests/mm_router/test_router_rust_mm_glm5_next_e2e.py is the engine-backed check of the same contract.

Validation

Current as of d55e94920, on main at 72ca6717d:

  • Rust: cargo test -p dynamo-llm --no-default-features --features mm-routing --locked --lib preprocessor:: passes 503 tests with 2 ignored, and cargo clippy with the same features and -D warnings is clean. Both lockfiles pin dynamo-multimodal 0.2.2 and pass cargo metadata --locked.
  • CI: every required check passed on 82fe162c9; d55e94920 changes only comments, docstrings and one docs line. The dynamo-runtime / test / parallel jobs ran the 11 binding resolver tests in lib/bindings/python/tests/test_routing_image_token_id.py, and all passed.
  • Planner geometry: review matched the pinned planner against the Transformers reference across 5,040 inputs.
  • E2E on GPU: review ran the module on GB200 on 2026-11-05 with one worker per backend. vLLM 0.32.0 and SGLang 0.5.22 each passed all 3 tests. SGLang 0.5.23, which main pins, kept its engine cache hits, but the router matched no image-bearing blocks (fix(mm-routing): handle SGLang engine-owned multimodal pad values #17126), so the module skips SGLang on such builds until feat(mm-routing): restore SGLang multimodal KV routing with routing-label spans #17082. The module needs Hopper or Blackwell for sparse MLA attention, so it runs only in the nightly vllm-h100-test and sglang-h100-test lanes; per review, those lanes had no runner from 2026-10-13 to 2026-11-08.
  • E2E selection and skip: pytest --collect-only collects 6 tests. The two H100 lane expressions select 3 each, and no other lane selects any. With the real sglang_has_wide_mm_pad_values() against stub SGLang schemas, the 0.5.23 schema skips the 3 SGLang tests and the 0.5.22 schema runs them.
  • E2E assertions: a model of the router and engine caches passes every assertion for worker block sizes from 64 to 7920 tokens and text lengths of 1 to 79 tokens. It fails the isolation checks when image content is left out of either cache key.
  • Pinned predownload: against the Hub, predownloading the pinned revision resolves it offline, and predownloading the default branch does not.

The support matrix lists GLM-5.3-Flash for vLLM, where the E2E passed. TensorRT-LLM coverage and runs with the full model remain follow-up work.

Related Issues

Part of #16948.

Internal tracking: DIS-3123 (multimodal-aware KV routing model support).

Consumes the shared planner from ai-dynamo/frontend-crates#412, released in dynamo-multimodal 0.2.2.

Summary by CodeRabbit

  • New Features
    • Added image-routing support for GLM-5.3-Flash (glm5_next) models, using checkpoint settings to determine image token counts and routing information.
  • Bug Fixes
    • Image prompts with unsupported layouts or invalid configuration no longer receive exact routing information, helping prevent incorrect routing.

@HanFa
HanFa requested review from a team as code owners October 22, 2026 19:02
@copy-pr-bot

copy-pr-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@HanFa
HanFa temporarily deployed to external_collaborator October 22, 2026 19:02 — with GitHub Actions Inactive
@HanFa
HanFa temporarily deployed to external_collaborator October 22, 2026 19:02 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot added feat external-contribution Pull request is from an external contributor labels Oct 22, 2026
@github-actions github-actions Bot added the frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` label Oct 22, 2026
@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

The change adds a GLM5-Next image-routing backend that counts image tokens and validates prompt layouts. Preprocessing uses validation results when building multimodal routing information. Tests cover configuration resolution, routing output, and invalid inputs.

Changes

GLM5-Next Image Routing

Layer / File(s) Summary
Routing configuration and backend
lib/llm/Cargo.toml, lib/llm/tests/data/glm5-next-routing/*, lib/llm/src/preprocessor/mm_routing/{mod.rs,glm5_next.rs}, lib/bindings/python/tests/test_routing_image_token_id.py
Adds GLM5-Next model and processor settings, enables the dynamo-multimodal dependency, and implements checkpoint-based image-token counting and prompt validation. Tests cover token-ID resolution, token counts, and invalid configurations.
Preprocessor validation and routing behavior
lib/llm/src/preprocessor.rs, lib/llm/src/preprocessor/mm_routing/image.rs, lib/llm/src/preprocessor/glm5_next_tests.rs
Adds a prompt-validation hook and calls it before constructing multimodal routing information. Tests cover routing output, validation failures, and image gathering behavior.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: Minimal · up to e3d86

No identified issue blocks merging after normal checks.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title clearly identifies the main change: integrating the GLM-5.3-Flash image planner into multimodal routing.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description is detailed and covers the change, key implementation details, validation, and follow-up work. Its headings differ from the template, but the required information is mostly present. It…
Full details: Description check

Explanation

The description is detailed and covers the change, key implementation details, validation, and follow-up work. Its headings differ from the template, but the required information is mostly present. It lists related issues but does not explicitly state that it closes an issue, and it omits the pull request checklist.

Finishing Touches

Fix failing CI checks

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
ℹ️ Recent review info

⚙️ Run configuration

📥 Commits

Reviewing files that changed from the base of the PR and between cf620cd and e3d865a.

⛔ Files ignored due to path filters: 2

  • Cargo.lock is excluded by !**/*.lock
  • lib/bindings/python/Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing: 9

  • lib/bindings/python/tests/test_routing_image_token_id.py
  • lib/llm/Cargo.toml
  • lib/llm/src/preprocessor.rs
  • lib/llm/src/preprocessor/glm5_next_tests.rs
  • lib/llm/src/preprocessor/mm_routing/glm5_next.rs
  • lib/llm/src/preprocessor/mm_routing/image.rs
  • lib/llm/src/preprocessor/mm_routing/mod.rs
  • lib/llm/tests/data/glm5-next-routing/config.json
  • lib/llm/tests/data/glm5-next-routing/processor_config.json

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@HanFa
HanFa force-pushed the codex/dis-3123-glm-image-routing-upstream branch from e3d865a to e15298f Compare October 26, 2026 18:54
@HanFa
HanFa temporarily deployed to external_collaborator October 26, 2026 18:54 — with GitHub Actions Inactive
@HanFa

HanFa commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e15298f

furionw commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

/ok to test e15298f

@devin-ai-integration

devin-ai-integration Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 42915099478 (attempt 1) on d55e949203

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 42915099478. Updated on every full-CI run of this PR.

@devin-ai-integration

Copy link
Copy Markdown
Contributor

External PR triage

Blocker: CI is green on the current head, but none of the requested CODEOWNER teams has reviewed since 2026-10-22 (6 days). Only the review bot has approved.

Next action (@ai-dynamo/dynamo-frontend-codeowners @ai-dynamo/dynamo-multimodal-codeowners @ai-dynamo/dynamo-router-codeowners @ai-dynamo/dynamo-runtime-codeowners @ai-dynamo/dynamo-ops-codeowners): Review and approve, or request changes.

Status as of 2026-10-28 for head e15298f7ea. This comment is edited in place on later runs.

Written by Devin

@rmccorm4
rmccorm4 requested a review from krishung5 October 28, 2026 19:31

@krishung5 krishung5 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall lgtm!

@HanFa
HanFa temporarily deployed to external_collaborator October 29, 2026 00:56 — with GitHub Actions Inactive
@krishung5

Copy link
Copy Markdown
Contributor

/ok to test 3d54251

@HanFa
HanFa requested a review from krishung5 October 29, 2026 17:17

@dmitry-tokarev-nv dmitry-tokarev-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No P0 or P1 found. One P2 is inline. This review read head 2b49edb84e27b090f4036d279564e6725029e175 and its merge with main at 31d6751d28. The merge with the newer main tip d405ab1636 is also clean, and that commit changes none of the files below.

Findings:

  • [P2] tests/mm_router/test_router_rust_mm_glm5_next_e2e.py:386: the three SGLang tests fail by design on SGLang 0.5.23, which this branch pins. The inline comment has a tested gate.

Open items from other reviewers: none. All four review threads are resolved.

What this review ran and measured.
  • Rust tests, cargo test -p dynamo-llm --no-default-features --features mm-routing --locked --lib preprocessor:: on macOS: the head passes 484, fails 0 and ignores 2. The merge with main 31d6751d28 passes 488, fails 0 and ignores 2. The 7 new tests pass on both.
  • cargo clippy -p dynamo-llm --no-default-features --features mm-routing --locked --all-targets --no-deps -- -D warnings is clean on the head and on the merge.
  • Bindings built with maturin develop --uv --features mm-routing --locked: test_routing_image_token_id.py passes 11 of 11. With the real config.json and processor_config.json of inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small at a2d7ca22e7, resolve_routing_image_token_id returns 154854. With do_center_crop: true added, it returns None.
  • For each new test file, a deliberate break of the code that it covers makes the file fail. Each break was restored afterwards:
deliberate break new tests that fail
remove the glm5_next branch in ImageRoutingProcessor::try_new 5 of 7 Rust tests, 2 of 11 binding tests
remove the validate_image_prompt call in build_mm_exact_routing_info 2 of the 3 tests in glm5_next_tests.rs
make the adapter validate_image_prompt return Ok(()) 3 of 7 Rust tests
  • CI lanes, by marker expression on main and on the merge: the 6 new E2E tests land only in nightly vllm-h100-test and sglang-h100-test, in the sequential stage. The 11 binding tests land in the dynamo-pipeline CPU lane. No test of main changes its lanes, and no test loses all of its lanes.
  • tests/conftest.py, with Qwen/Qwen3-0.6B and codelion/Qwen3-0.6B-accuracy-recovery-lora: a predownload of repo@<sha> resolves that revision offline, and a predownload of the default branch does not. A mark without revision= calls snapshot_download(repo_id=..., revision=None), which is the default that main uses.
  • Dependencies: the 4 packages that both lock files add match their crates.io checksums. Every file in the published dynamo-multimodal 0.2.2 crate matches the same file at its dynamo-multimodal-v0.2.2 tag, except for the Cargo.toml that cargo publish rewrites. cargo-deny 0.19.0 with the Pre Merge arguments passes on the merge base, the head, main and the merge, in both lock roots. RustSec and OSV list no advisory for the 4 packages.
  • CI on this head: 129 check runs, with 90 passed and 39 skipped. The two failures in attempt 1 of PR run 42071663260 passed on a rerun of the same SHA. The DGDR smoke job passed in attempt 2. The TRT-LLM amd64 job passed in attempt 4, with test_deployment[aggregated_multimodal_router-2] passing. Pre Merge rust-tests (.) ran the 7 new Rust tests. The dynamo-runtime / test / parallel jobs ran the 11 binding tests on amd64 and arm64.
  • Coverage facts: the nightly runs from 2026-10-13 to 2026-11-08 created 26 sets of vLLM and SGLang H100 jobs. Each of these jobs ended cancelled without a runner. The last H100 jobs with a runner ran on 2026-10-08, and they failed. No lane in pr.yaml or post-merge-ci.yml names a Hopper or Blackwell GPU. After this head merged main, #17862 moved main to vLLM 0.33.0. The GB200 results posted on 2026-11-05 used vLLM 0.32.0.
  • This review did not run the E2E module. It needs Hopper or Blackwell GPUs.

@HanFa

HanFa commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

thanks for reviewing folks! @ai-dynamo/dynamo-ops-codeowners @krishung5 can I have your help merging in this PR to prevent it from going stale? happy to address if you have more comments

@krishung5

Copy link
Copy Markdown
Contributor

Hi @HanFa, could you address Dmitry's comments above? Would need a approval from the Ops team before this PR can be merged. Thanks!

auto-merge was automatically disabled November 9, 2026 23:38

Head branch was pushed to by a user without write access

@HanFa
HanFa temporarily deployed to external_collaborator November 9, 2026 23:38 — with GitHub Actions Inactive
@krishung5

Copy link
Copy Markdown
Contributor

/ok to test 08255e9

@rmccorm4
rmccorm4 enabled auto-merge (squash) November 10, 2026 06:34
HanFa and others added 5 commits November 9, 2026 22:45
Use dynamo-multimodal 0.2.2 for checkpoint-driven GLM image token counts and layout validation. Select the adapter before legacy flat-config loading and fall back to text-prefix routing when planning is unsupported or fails.

Add Rust adapter and preprocessing tests plus real Python binding resolver tests with shared lightweight checkpoint fixtures. Three-backend routing E2E remains follow-up work.

Validation: targeted preprocessor Rust tests passed (383 passed, 2 ignored); Python binding tests passed (11 passed); cargo fmt, Ruff, and git diff --check passed.
Signed-off-by: Fang Han <fangh@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Rust planner tests and the Python resolver tests never run an
engine, so they cannot catch the glm5_next planner and a worker
disagreeing on image token expansion, or a hash mismatch between the
frontend's routing blocks and the worker's KV events. Add an E2E module
that drives the Rust frontend's KV router against a real vLLM or SGLang
worker serving tiny-random/glm-5.3-flash, a 227 MB random checkpoint
that keeps GLM-5.3-Flash's tokenizer, chat template, processor config
and token ids.

Per engine it asserts that the router's block total equals the engine's
reported prompt length after its own expansion, that per-image token
increments follow GLM geometry, that repeated images match nearly every
published block and hit the engine's prefix cache, that a different
image on the same prompt matches neither, and that two non-block-aligned
images agree block for block with the second image's blocks invalidated
exactly from the boundary on.

GLM5-Next's DSA layers need sparse MLA attention, which vLLM 0.32 and
SGLang 0.5.22 provide only on Hopper and Blackwell, so the module is
nightly + h100 and runs in the vllm-h100-test and sglang-h100-test
lanes. Exclude h100 tests from the nightly single-GPU vLLM lane the way
the SGLang lane already does, so they do not land on its Ada runner.

Signed-off-by: Fang Han <fangh@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first GPU run of the GLM E2E module, on GB200 in review, failed in
two ways. tiny-random/glm-5.3-flash does not start: its kv_lora_rank of
256 is not a sparse-MLA head size vLLM accepts, and SGLang cannot load
its 32x32 block-FP8 weights. With a BF16 checkpoint both engines serve,
but they size KV blocks for GLM5-Next's hybrid KDA layers themselves
(vLLM advertised 4352 tokens, SGLang 256) while the assertions assumed
64, and a 4096-token context cannot fill even one vLLM block.

Switch to inference-optimization/GLM-5.3-Flash-2.6B-A2.3B-small pinned
at a2d7ca22e78d. The workers load the pinned snapshot from a local path,
so both engines and the frontend's planner read the same files. Read
the block size the worker advertised from the frontend's
model_kv_cache_block_size metric, and derive the image sizes and every
block assertion from it. The large image covers about one and a half
blocks and the small one just over a block, so every image prompt spans
several blocks and a second image always reaches a block the first did
not. Raise the context to 32768 tokens. Bound cold overlaps and engine
reuse by the blocks the text alone fills, measured with a text-only
request.

Drop the kernel pins, which only the block-FP8 checkpoint needed, and
the frontend's --kv-cache-block-size, which does not apply to
discovered workers. Drop the "resolved image-placeholder token id" log
assertion: GLM's placeholder comes from config.json's image_token_id,
and that path does not log it.

Signed-off-by: Fang Han <fangh@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review follow-up on 97296c2.

The shared predownload fetched the checkpoint's default branch before it
turned on HF_HUB_OFFLINE. Once that branch moves past the pinned
revision, a clean runner would not find the pinned snapshot. The model
mark now takes revision=: pytest_collection_modifyitems records it as
repo_id@revision and download_models fetches that revision. The GLM
module passes its pin and only looks the snapshot up in the cache.

In the multi-image test, AC_cold must now also match more blocks than
the text-only prompt fills. The old [A repeat - 1, A repeat] bound
accepted zero when the A repeat matched a single block, as it does at
GB200's 4352-token vLLM block, so losing the first image's cached blocks
would have passed.

_send now returns a RequestObservation and checks the worker's expansion
against the router's block count before returning. That removes the
tuple unpacking and the per-test expansion calls.

Signed-off-by: Fang Han <fangh@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review follow-up on 2b49edb.

SGLang 0.5.23, which pyproject.toml pins, carries sgl-project/sglang#43043:
it fills image tokens with pad values of its own, and the SGLang publisher
marks its workers as owning their multimodal identity. Since ai-dynamo#17126 the
router cuts each KV store at the first block holding such a value and keeps
that block and every later one out of its index. Every image prompt in this
module holds image tokens in its first block, so image requests match no
router blocks, and the repeat-overlap assertions fail by design in all three
SGLang tests. The GB200 run on 2026-11-05 showed this.

Skip the SGLang parameter when sglang_has_wide_mm_pad_values() reports the
wide format, the probe tests/serve/multimodal_profiles/sglang.py already
uses, until exact SGLang image routing lands in ai-dynamo#17082. The import stays in
the fixture so collection never imports SGLang. SGLang 0.5.22 builds still
run the three tests, and the vLLM tests are unchanged.

Signed-off-by: Fang Han <fangh@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
auto-merge was automatically disabled November 10, 2026 07:00

Head branch was pushed to by a user without write access

@HanFa
HanFa force-pushed the codex/dis-3123-glm-image-routing-upstream branch from 08255e9 to 82fe162 Compare November 10, 2026 07:00
@HanFa
HanFa temporarily deployed to external_collaborator November 10, 2026 07:00 — with GitHub Actions Inactive
@furionw

furionw commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

/ok to test 82fe162

@harryskim harryskim left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs review. No .md/.mdx files change, so these are comment and description notes only.

PR description: the Context/Validation sections read as a rebase log (commit-hash mappings, per-revision checks), and the test counts conflict (503 vs 405 passed on the same rebased base). A single current validation summary would be easier to review. The CodeRabbit summary also says "GLM-5 Next models" while the rest says GLM-5.3-Flash.

Also: if the repo has a multimodal-routing page or model support matrix, GLM-5.3-Flash (glm5_next) should be added there, or the PR should note that it is deferred.

Docs review follow-up on 82fe162.

Name the files ImageRoutingProcessor::try_new needs: a glm5_next
checkpoint reads config.json plus preprocessor_config.json, or
processor_config.json when that file is absent, and every other model
needs preprocessor_config.json. Say that on Err the preprocessor logs a
warning and routes by text prefix, that only model_type "glm5_next"
takes the GLM path, and that validate_image_prompt applies no
constraints by default.

In the GLM E2E module, describe the SGLang skip condition without
version or PR numbers, which stay in the skip itself, and date the
Hardware and Block size figures.

List GLM-5.3-Flash in the vLLM row of the multimodal KV routing support
matrix. vLLM 0.32.0 passed all three E2E tests on GB200.

Signed-off-by: Fang Han <fangh@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@HanFa
HanFa requested a review from a team as a code owner November 10, 2026 20:17
@HanFa
HanFa temporarily deployed to external_collaborator November 10, 2026 20:17 — with GitHub Actions Inactive
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 10, 2026
@HanFa

HanFa commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the docs review in d55e949 and in the description:

  • Context and Validation now give one current summary. The 405 count came from the earlier base; 503 is current on main at 72ca6717d.
  • The CodeRabbit summary now says GLM-5.3-Flash (glm5_next).
  • GLM-5.3-Flash is listed in the vLLM row of the support matrix in multimodal-kv-routing.md. vLLM 0.32.0 passed all three E2E tests on GB200.

The new head d55e94920 changes only comments, docstrings and that docs line, and needs another /ok to test.

@harryskim harryskim left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved with the changes

@krishung5

Copy link
Copy Markdown
Contributor

/ok to test d55e949

@krishung5
krishung5 enabled auto-merge (squash) November 10, 2026 22:29

@dmitry-tokarev-nv dmitry-tokarev-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No P0, P1 or P2 found. This review read head d55e949203a4b86db4c5f09df3ea23ede0b06b05 and its merge with main at 3e84909a1971a9fa7dcbd3ee5c739e6f5cf6da7c.

  • [P3] lib/llm/src/preprocessor/glm5_next_tests.rs:7: after a merge with main, the import needs compat.rs. The fix is inline.
  • The earlier P2 about SGLang 0.5.23 is fixed in 82fe162c99. The results are in that thread.
  • Open: CI has not run the E2E module, because no nightly H100 job got a runner.
What this review measured.
  • Nightly H100 jobs. None got a runner since 2026-10-13. The jobs of 2026-11-08 to 2026-11-10 were cancelled after 24 hours, and the jobs of 2026-11-11 still wait.
  • Rebase. git range-diff shows the three test commits unchanged. The first commit changed only context lines in the two lock files. Between the own diff of 2b49edb84e and the own diff of this head, 47 lines in 3 files differ. All of them come from 82fe162c99 and d55e949203. On this head and on the merge, both lock files add the same 4 packages at the same versions and checksums. They change no other package, apart from the new dependency entry of dynamo-llm.
  • Commit d55e949203. With comments and docstrings removed, image.rs and the E2E module are the same before and after it. A planted one-character code change in each file shows up in the same comparison. The new doc comments in image.rs match the code. spec_from_model_dir in dynamo-multimodal 0.2.2 reads config.json and preprocessor_config.json. If preprocessor_config.json is missing, it reads processor_config.json instead. Only model_type glm5_next takes the GLM path. On Err, the preprocessor logs a warning on target mm_routing and routes without image awareness.
  • Support matrix. The GB200 results of 2026-11-05 used vLLM 0.32.0. main pins vLLM 0.33.0 since #17862, and the base of this head has it. The Rust code of this PR reads checkpoint metadata and calls nothing in vLLM. Exact routing also needs the worker to expand each image the same way. In vLLM, vllm/models/glm5next/common/multimodal.py and its processing test are the same files in v0.32.0 and v0.33.0. No E2E run used vLLM 0.33.0.
  • Rust on macOS, with --no-default-features --features mm-routing. The preprocessor:: tests pass 503 on this head and 506 on the merge, with 0 failed and 2 ignored. The 7 GLM tests pass on both. Clippy with --all-targets --no-deps -- -D warnings passes on both. With block-manager on, macOS does not build dynamo-llm, and all errors are in lib/llm/src/block_manager/. CI's rust-clippy (.) linted that feature set on the merge with c2a6b68bc7.
  • Bindings on the merge, built with maturin develop --uv --features mm-routing --locked. test_routing_image_token_id.py passes 11 of 11. With resolve_routing_image_token_id hidden, as in a wheel without mm-routing, the 11 tests skip and do not fail. After #18199, both branches of container/templates/wheel_builder.Dockerfile still turn on mm-routing, and that file is the same on this head and on the merge.
  • Both lock roots of the merge pass cargo tree --locked. cargo-deny 0.19.0 with the CI arguments passes in both.
  • Log filter of the E2E module. Every test reads the [ROUTING] Best: line. On main, dynamo_llm::kv_router::routing_host::kv_selection emits it at debug level. With the module's DYN_LOG, a frontend and one dynamo.mocker worker on the merge logged 2 such lines for 2 requests, and tests/utils/router_logs.py parsed both. Without dynamo_llm::kv_router=debug, they logged none. In this run, dynamo_kv_router::scheduling=debug turned on no line after #18055. The other tests/mm_router/ modules now use dynamo_scheduler=debug,dynamo_admission=debug,dynamo_worker_selection=debug. No test result reads those lines.
  • CI on this head: 133 check runs, 98 passed, 35 skipped, 0 failed. rust-tests (.) ran the 7 new Rust tests, and both dynamo-runtime / test / parallel jobs ran the 11 binding tests. All passed. No CI job built this head with #18055. Pre Merge tested the merge with main at c2a6b68bc7, which has #18199 but not #18055.

@dmitry-tokarev-nv dmitry-tokarev-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving head d55e949203a4b86db4c5f09df3ea23ede0b06b05.

The earlier P2 is fixed in 82fe162c99. On SGLang builds with wide pad values, the 3 SGLang E2E tests skip. On older builds, they still start the worker. The rebase changed only context lines in the lock files, and d55e949203 changes only comments, docstrings and one docs line. On macOS, with --features mm-routing, the Rust tests and clippy pass on this head and on its merge with main at 3e84909a19. The 11 binding tests pass on that merge, and CI ran them on this head on amd64 and arm64. Both lock roots of the merge pass cargo tree --locked and cargo-deny.

Open: the P3 at lib/llm/src/preprocessor/glm5_next_tests.rs:7. After a merge with main, the import resolves only through compat.rs, which 1.10 removes. Its thread blocks the merge until someone resolves it.

Not run: the E2E module, which CI selects only in the nightly H100 lanes. From 2026-10-13 to 2026-11-11, no nightly H100 job got a runner, so CI has not run the module. No CI job built this head with #18055, so the merge with the router crate split was tested only on macOS.

@krishung5
krishung5 merged commit 02be33b into ai-dynamo:main Nov 12, 2026
133 checks passed

This branch was previously deployed

1 inactive deployment
external_collaborator — d55e9492 Deployed Nov 10, 2026 by HanFa via ok-to-test #49444
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants