Repository navigation
Conversation
Signed-off-by: Big2Wheel <1404207590@qq.com>
|
hi, thank you for your contribution. Due to the involvement of multiple models such as qwen3-vl, mimo-vl, GLM-4.1v, etc., our other colleague is also working on this feature. Could you collaborate with him to complete this task? He is conducting testing on other models. |
|
Thanks for the feedback! We would be happy to collaborate. Could you tag your colleague and share the relevant PR or implementation branch? We can align on the design and divide the work to avoid duplication. This PR currently focuses on opt-in cross-request image encoding for single-NPU, unquantized Qwen3-VL, with an input-patch budget to bound each encoder batch. We have validated Qwen3-VL-8B-Instruct BF16 on A2 (910B3): all 192 answer strings matched exactly between batching disabled and enabled across fixed subsets of ChartQA, DocVQA, and VQAv2 (64 examples each). We also tested chunked prefill, image-cache reuse, and compatibility with decode NPU graphs, while keeping the ViT encoder in eager mode. We currently only have access to A2 hardware and can continue contributing A2 correctness, performance, and regression testing while coordinating with your colleague on broader model coverage. We are flexible about the implementation and PR organization—the goal is to converge on a maintainable solution together. |
Motivation
The NPU chunked multimodal path encodes image-cache misses per request even
when the scheduler has already selected multiple compatible image requests.
This PR adds an opt-in, bounded path for Qwen3-VL image encoding, with real
Ascend A2 correctness and performance validation.
Related to the cross-request ViT batching item in #43519, not intended to
close the whole roadmap issue.
Related work: #38368 proposes a generic encoder input-token budget. The small
partitioning helper here overlaps that infrastructure concern; the distinct
scope here is NPU enablement, conservative model/mode gating and A2 validation.
The NPU opt-in should reuse the generic budget mechanism when that lands rather
than maintaining two independent batching-budget implementations long term.
Modifications
SGLANG_NPU_VIT_MAX_BATCH_PATCHES, default0(disabled).for a single-NPU, unquantized
Qwen3VLForConditionalGenerationimage encoder.Never split an image: an individually oversized image runs alone.
encoders, quantized models, ViT graph execution and unsupported input forms.
and Ascend environment-variable documentation.
There are no shipped vision-model forward changes and no new CANN operators.
Decode graphs can still be enabled; the visual encoder remains eager.
The patch budget is not a strict total-memory limit.
Accuracy Tests
Environment: one Ascend A2 910B3 64GB, Qwen3-VL-8B-Instruct BF16, TP1,
CANN 9.0.0, torch/torch-npu 2.10.0, Transformers 5.19.0, tokenizers 0.23.1.
Used
main-cann9.0.0-910bwith the checkout-required Transformers version inan isolated dependency directory. A3/A5 have not been tested.
test_mm_chunked_embedding_unit.py: 35 passed. Includes budget bounds,oversized/empty input, cache on/off, duplicate misses, cache hits, mode
gates and partial chunk slices in original request order.
changed text prefixes with image-cache reuse. Passed standalone server
launch/teardown, eager execution and decode-graph execution on A2.
128-request concurrency-8 smoke run passed 128/128.
versus individual BF16 features are not bitwise identical (cosine about
0.99945–0.99955); no probes are included in the patch.
Real-image paired comparison: 192/192 answer strings agree exactly, with no
newly incorrect answers and no truncated completions in either arm.
Protocol: fixed seed 42 per source file, deduplicate images before selecting
64 samples, combined shuffle seed 17; longest image edge limited to 1024
without upscaling. Both arms use identical PNGs/prompts,
temperature=0,seed=42,max_tokens=64, concurrency 8, chunked prefill 4096, eager mode,and separately restarted servers. Enabled budget: 8192 input patches.
This is a small paired regression check, not an official full-dataset
benchmark or proof of universal equivalence. VQAv2 uses majority-answer
exact match, not the official consensus score.
Temporary shape-only logs confirm 55 visual calls with 2–5 distinct images;
the maximum observed call had 7936 input patches. Excluding server warmup,
192 images required 192 encoder calls when disabled and 119 when enabled.
The logs did not copy tensors to CPU or force synchronization and are removed.
Source files (SHA256 verified before selecting samples):
HuggingFaceM4/ChartQA:data/test-00000-of-00001-e2cd0b7a0f9eb20d.parquet(
efcf31333368c485eb569ec7bd4cc3694bd0e8e1a06c503124b51d27c924491d).lmms-lab-encoder/DocVQA:DocVQA/validation-00000-of-00006.parquet(
a31507f0b700ac64f3ead52057c5dc3ccfb0baadcc62bc0d19d17159f080a4d8).lmms-lab-encoder/VQAv2:data/validation-00000-of-00068.parquet(
f21976c1b29cc26655613395d51c3e301ffd8c29d43ebfc499bde92657dfd933).The NPU smoke test is registered in the existing
full-1-npu-a3nightlysuite; its A3 execution still needs community CI. Local results above are A2.
Speed Tests and Profiling
Initial A2 measurements on main
68c18a5701plus this patch, before rebasingonto
6daae9c98efor the real-image accuracy checks:TP1 BF16, eager, patch budget 2048 vs disabled, chunk 2048, context 8192,
max-running-requests 8, mem-fraction-static 0.7. A local HTTP streaming client
sent unique mixed-height 448-wide synthetic images at concurrency 4:
8 excluded warmup requests, then 3 × 24 measured requests per arm.
No numerical probes or NPU synchronizations were injected in measured runs.
All 72 answers per arm matched expectations. A reverse-order fresh-server
repeat produced 384.10 ms p50. The observed ~18% TTFT reduction is specific
to this workload, not a general VLM speedup guarantee. Performance on the
real-image accuracy subset is not used as a benchmark here.
Checklist
mint validateandmint broken-linkspassed.CI States
Latest PR Test (Base): ❌ Run #38529068077
Latest PR Test (Extra): ❌ Run #38529068201
Latest PR Test (AMD ROCm 10): ❌ Run #38529068208