Skip to content

[Perf] [NPU] multimodal: Add bounded cross-request Qwen3-VL image batching - #44233

Open
Big2Wheel wants to merge 1 commit into
sgl-project:mainfrom
Big2Wheel:wyl/npu-vit-batching
Open

Big2Wheel wants to merge 1 commit into
sgl-project:mainfrom
Big2Wheel:wyl/npu-vit-batching

Conversation

@Big2Wheel

@Big2Wheel Big2Wheel commented Oct 14, 2026 •

Copy link
Copy Markdown

Motivation

The NPU chunked multimodal path encodes image-cache misses per request even
when the scheduler has already selected multiple compatible image requests.
This PR adds an opt-in, bounded path for Qwen3-VL image encoding, with real
Ascend A2 correctness and performance validation.

Related to the cross-request ViT batching item in #43519, not intended to
close the whole roadmap issue.

Related work: #38368 proposes a generic encoder input-token budget. The small
partitioning helper here overlaps that infrastructure concern; the distinct
scope here is NPU enablement, conservative model/mode gating and A2 validation.
The NPU opt-in should reuse the generic budget mechanism when that lands rather
than maintaining two independent batching-budget implementations long term.

Modifications

  • Add SGLANG_NPU_VIT_MAX_BATCH_PATCHES, default 0 (disabled).
  • Enable the existing cross-request per-image cache/deduplication/scatter path
    for a single-NPU, unquantized Qwen3VLForConditionalGeneration image encoder.
  • Partition unique cache misses by input patches before spatial merging.
    Never split an image: an individually oversized image runs alone.
  • Do not wait for new requests or change the scheduler's request selection.
  • Keep the per-request path for video, other model classes, TP > 1, DP
    encoders, quantized models, ViT graph execution and unsupported input forms.
  • Add CPU regression coverage, a self-contained NPU image/cache smoke test,
    and Ascend environment-variable documentation.

There are no shipped vision-model forward changes and no new CANN operators.
Decode graphs can still be enabled; the visual encoder remains eager.
The patch budget is not a strict total-memory limit.

Accuracy Tests

Environment: one Ascend A2 910B3 64GB, Qwen3-VL-8B-Instruct BF16, TP1,
CANN 9.0.0, torch/torch-npu 2.10.0, Transformers 5.19.0, tokenizers 0.23.1.
Used main-cann9.0.0-910b with the checkout-required Transformers version in
an isolated dependency directory. A3/A5 have not been tested.

  • test_mm_chunked_embedding_unit.py: 35 passed. Includes budget bounds,
    oversized/empty input, cache on/off, duplicate misses, cache hits, mode
    gates and partial chunk slices in original request order.
  • New NPU test: mixed sizes, multi-image order, >512-token image chunks and
    changed text prefixes with image-cache reuse. Passed standalone server
    launch/teardown, eager execution and decode-graph execution on A2.
  • Decode graph logs confirm real replay, not just capture. A separate
    128-request concurrency-8 smoke run passed 128/128.
  • Temporary probes confirmed actual cross-request visual calls. Batched
    versus individual BF16 features are not bitwise identical (cosine about
    0.99945–0.99955); no probes are included in the patch.

Real-image paired comparison: 192/192 answer strings agree exactly, with no
newly incorrect answers and no truncated completions in either arm.

Fixed subset Metric Disabled Enabled
ChartQA test, 64 distinct images relaxed numeric accuracy (5%) / text match 85.94% 85.94%
DocVQA validation shard 0/6, 64 distinct images ANLS 92.10% 92.10%
VQAv2 validation shard 0/68, 64 distinct images normalized majority-answer exact match 67.19% 67.19%

Protocol: fixed seed 42 per source file, deduplicate images before selecting
64 samples, combined shuffle seed 17; longest image edge limited to 1024
without upscaling. Both arms use identical PNGs/prompts, temperature=0,
seed=42, max_tokens=64, concurrency 8, chunked prefill 4096, eager mode,
and separately restarted servers. Enabled budget: 8192 input patches.
This is a small paired regression check, not an official full-dataset
benchmark or proof of universal equivalence. VQAv2 uses majority-answer
exact match, not the official consensus score.

Temporary shape-only logs confirm 55 visual calls with 2–5 distinct images;
the maximum observed call had 7936 input patches. Excluding server warmup,
192 images required 192 encoder calls when disabled and 119 when enabled.
The logs did not copy tensors to CPU or force synchronization and are removed.

Source files (SHA256 verified before selecting samples):

  • HuggingFaceM4/ChartQA: data/test-00000-of-00001-e2cd0b7a0f9eb20d.parquet
    (efcf31333368c485eb569ec7bd4cc3694bd0e8e1a06c503124b51d27c924491d).
  • lmms-lab-encoder/DocVQA: DocVQA/validation-00000-of-00006.parquet
    (a31507f0b700ac64f3ead52057c5dc3ccfb0baadcc62bc0d19d17159f080a4d8).
  • lmms-lab-encoder/VQAv2: data/validation-00000-of-00068.parquet
    (f21976c1b29cc26655613395d51c3e301ffd8c29d43ebfc499bde92657dfd933).

The NPU smoke test is registered in the existing full-1-npu-a3 nightly
suite; its A3 execution still needs community CI. Local results above are A2.

Speed Tests and Profiling

Initial A2 measurements on main 68c18a5701 plus this patch, before rebasing
onto 6daae9c98e for the real-image accuracy checks:

TP1 BF16, eager, patch budget 2048 vs disabled, chunk 2048, context 8192,
max-running-requests 8, mem-fraction-static 0.7. A local HTTP streaming client
sent unique mixed-height 448-wide synthetic images at concurrency 4:
8 excluded warmup requests, then 3 × 24 measured requests per arm.
No numerical probes or NPU synchronizations were injected in measured runs.

Metric Disabled Enabled
TTFT p50 470.09 ms 384.36 ms
TTFT p95 494.79 ms 404.80 ms
Aggregate requests/s 7.226 8.572

All 72 answers per arm matched expectations. A reverse-order fresh-server
repeat produced 384.10 ms p50. The observed ~18% TTFT reduction is specific
to this workload, not a general VLM speedup guarantee. Performance on the
real-image accuracy subset is not used as a benchmark here.

Checklist

  • Changed-file pre-commit checks pass.
  • CPU tests and real-NPU smoke tests added and run.
  • Documentation updated; mint validate and mint broken-links passed.
  • Complete the paired real-image accuracy comparison.
  • Preserve existing behavior by default and document unsupported modes.

CI States

Latest PR Test (Base): ❌ Run #38529068077
Latest PR Test (Extra): ❌ Run #38529068201
Latest PR Test (AMD ROCm 10): ❌ Run #38529068208

Signed-off-by: Big2Wheel <1404207590@qq.com>
@cx22757

cx22757 commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

hi, thank you for your contribution. Due to the involvement of multiple models such as qwen3-vl, mimo-vl, GLM-4.1v, etc., our other colleague is also working on this feature. Could you collaborate with him to complete this task? He is conducting testing on other models.

@Big2Wheel

Copy link
Copy Markdown
Author

Thanks for the feedback! We would be happy to collaborate. Could you tag your colleague and share the relevant PR or implementation branch? We can align on the design and divide the work to avoid duplication.

This PR currently focuses on opt-in cross-request image encoding for single-NPU, unquantized Qwen3-VL, with an input-patch budget to bound each encoder batch. We have validated Qwen3-VL-8B-Instruct BF16 on A2 (910B3): all 192 answer strings matched exactly between batching disabled and enabled across fixed subsets of ChartQA, DocVQA, and VQAv2 (64 examples each). We also tested chunked prefill, image-cache reuse, and compatibility with decode NPU graphs, while keeping the ViT encoder in eager mode.

We currently only have access to A2 hardware and can continue contributing A2 correctness, performance, and regression testing while coordinating with your colleague on broader model coverage. We are flexible about the implementation and PR organization—the goal is to converge on a maintainable solution together.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation npu

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants