Repository navigation
[Fix] [XPU] ernie45_vl/dllm: Build the visual-token mask only for image or video prefill, and fix dLLM serving - #41730
desmond-intel wants to merge 6 commits into
Conversation
d15d722 to
8c78773
Compare
|
Closing this as it is getting merged as part of #42627. |
Ernie4_5_VLMoeForConditionalGeneration.forward built the visual-token routing mask on every forward, including every decode step. On the intel_xpu (level_zero) backend the small per-step device allocations in _set_visual_token_mask (.to, cat, isin) are never reclaimed, so a docvqa run exhausts USM and dies with 'level_zero backend failed with error: 40 (UR_RESULT_ERROR_OUT_OF_RESOURCES)' after a few dozen documents. The mask only routes prefill/extend image tokens to the vision experts; decode tokens are always text, where an all-False mask is equivalent to None (the text-experts path, same as CUDA-graph capture). Compute it only for non-decode batches that contain image inputs. Validated on Intel XPU (tp4, docvqa_val, serialized): 80/80 docs, 368k tokens, zero error-40 (previously crashed at ~44 docs); answers unchanged.
|
Reopening so this fix can be reviewed on its own; the same change is also part of #42627. If that merges first, I'll close this one. |
8c78773 to
69662a3
Compare
|
CUDA check on 1x NVIDIA H200 NVL (tp 1). Both runs use the same image (
No regression on CUDA. docvqa covers image prefill, which still builds the mask. gsm8k covers text-only batches, which now skip it. sglang serve --model-path baidu/ERNIE-4.5-VL-28B-A3B-PT --tp-size 1 --mem-fraction-static 0.85 \
--trust-remote-code --context-length 8192 --enable-multimodal # gsm8k: same, without --enable-multimodal
python3 -m lmms_eval --model openai --model_args model_version=baidu/ERNIE-4.5-VL-28B-A3B-PT,num_concurrent=8 \
--tasks docvqa_val --batch_size 8 --limit 1000 --seed 42 --gen_kwargs enable_thinking=false,max_new_tokens=64
lm_eval --model local-completions \
--model_args pretrained=baidu/ERNIE-4.5-VL-28B-A3B-PT,base_url=http://127.0.0.1:30000/v1/completions,num_concurrent=16 \
--tasks gsm8k --num_fewshot 5 --batch_size 16 --seed 42 --gen_kwargs max_gen_toks=1024 |
Source: sgl-project#41730 Source-head: 69662a3 Mechanically replayed source net patch onto origin/main b74fd39.
|
@desmond-intel — forwarding @mingfeima’s comment from #46623: Video-only prefill does not satisfy Keep the mask construction for both image and video input! |
|
@mingfeima @airMeng Fixed in dfb591e. The mask is now built whenever a prefill batch has image or video inputs, so video-only prefill routes its video tokens through the vision experts again, as on main. Decode and text-only batches still skip it, which is what the XPU fix needs. Added |
The mask was only built when the batch had image inputs, so a video-only prefill got no mask and its video tokens went through the text experts on every device. Build it whenever the batch has image or video inputs; decode and text-only batches still skip it. Add test/registered/unit/models/test_ernie45_vl_visual_mask.py (CPU): a video-only prefill must mask its video tokens, and decode and text-only batches must not build the mask. The video case fails without this change.
Moved here from sgl-project#41773 (its two commits, squashed). Build extend metadata for DLLM_EXTEND (fix dLLM decode crash): DLLM_EXTEND is excluded from is_extend_or_draft_extend_or_mixed(). On XPU, where dLLM runs eager rather than via CUDA graph, init_forward_metadata fell through every branch and left metadata.page_table None, crashing at the page_size>1 strided-page step (AttributeError: 'NoneType' object has no attribute 'shape'). Route DLLM_EXTEND through the extend branch so page_table is built. Verified on intel_xpu tp2: LLaDA2.1-mini gsm8k (--dllm-algorithm LowConfidence) serves and scores flexible-extract 0.8 on 5 samples, no crash. Index KV per token for unsupported page sizes (fix dLLM blank output): dLLM serving sets page_size to the diffusion block size (_dllm_page_size), which overrides the intel_xpu 64/128 page constraint. SDAR uses block 4, so the backend passes a page-4 KV cache and page table to flash_attn_with_kvcache. The XPU paged kernels are exact at page 1, 64 and 128; at page 2/4/8/16 a query block (q_len > 1) whose KV spans more than one page returns wrong output, and decode raises "Unsupported page size". SDAR-8B on intel_xpu produces blank output as a result. For non-MLA attention without DCP, fall back to per-token indexing (page_size = 1) when the page size is outside {1, 64, 128}. The KV pool layout is unchanged; the existing page-1 path views the cache as [-1, 1, H, D] and skips the page-table striding, which is valid for any physical page size.
Run test_ernie45_vl_visual_mask.py in stage-a-test-1-gpu-xpu instead of the CPU base suite.
Keep it out of CI and nightly: stage-a-test-1-gpu-xpu also runs in the nightly XPU wheel job. Run it locally with python3 -m pytest test/manual/models/test_ernie45_vl_visual_mask.py.
This PR has two Intel XPU fixes: the ERNIE-4.5-VL visual-token mask, and dLLM serving on
intel_xpu, which moved here from #41773.Motivation
1. ERNIE-4.5-VL: visual-token mask on every decode step
On Intel XPU,
baidu/ERNIE-4.5-VL-28B-A3B-PTcrashes during docvqa after a few dozen documents:Ernie4_5_VLMoeForConditionalGeneration.forwardcalls_set_visual_token_maskon every forward, including every decode step. Each call allocates small device tensors (_visual_token_ids_tensor_cache.to(device),torch.as_tensor(pad_values, device=...),torch.cat,torch.isin). The level_zero backend does not reclaim them, so one prefill plus N decode steps per document adds up until device memory runs out.The mask only decides which tokens go to the vision experts (
ernie45_moe_vl.py). ANonemask and an all-False mask take the same text-experts path, which is also the path CUDA-graph capture uses. Decode tokens are generated text, so the per-step mask is always all-False.2. dLLM serving on intel_xpu
Diffusion LMs (
--dllm-algorithm) on--attention-backend intel_xpuhit two XPU bugs:inclusionAI/LLaDA2.1-mini,JetLM/SDAR-8B-Chat):AttributeError: 'NoneType' object has no attribute 'shape'inXPUAttentionBackend.init_forward_metadata.ForwardMode.DLLM_EXTENDis inis_extend()but not inis_extend_or_draft_extend_or_mixed(), which guards the XPU extend-metadata branch. On CUDA,DLLM_EXTENDruns via CUDA graph and never reaches this eager branch; on XPU it matches no branch and leavesmetadata.page_table = None.JetLM/SDAR-8B-Chat):_dllm_page_sizesetspage_sizeto the dLLM block size (4 for SDAR, 32 for LLaDA2), overriding the intel_xpu 64/128 page constraint. The sgl-kernel-xpu paged kernels are exact at page 1/64/128. At page 2/4/8/16, a query block whose KV spans more than one page returns wrong output (max abs err 1.4 to 3.9 at page 4), and SDAR generates\n\n\n....Modifications
1. ERNIE-4.5-VL
python/sglang/srt/models/ernie45_vl.py, build the mask only for non-decode batches that contain image or video inputs; otherwise setvisual_token_mask = None.2. dLLM
python/sglang/srt/layers/attention/xpu_backend.py:DLLM_EXTENDinto the extend-metadata branch (a multi-query block forward, so extend-shaped metadata is correct).page_size = 1) when the page size is not 1/64/128. This reuses the existing page-1 path; the KV pool layout is unchanged.Both dLLM changes are in the XPU attention backend only.
#42627 carries earlier versions of both fixes, without the video-only mask fix and the page-size fallback.
Accuracy Tests
1. ERNIE-4.5-VL
Intel Arc Pro B60, tp-size 4,
--attention-backend intel_xpu, lmms_evaldocvqa_val:num_concurrent=1,--max-running-requests 1)num_concurrent=8,--mem-fraction-static 0.70, no serialization flagsThe anls run sets
enable_thinking=falsein the chat template. ERNIE-4.5-VL opens a<think>block by default, and docvqa's short generation budget cuts it off before the answer.Reproduce (serialized run):
2. dLLM
Intel Arc Pro B60,
--dllm-algorithm LowConfidence --attention-backend intel_xpu, gsm8k 5-shot chat.JetLM/SDAR-8B-Chat, tp1:\boxed{}answer correctCounting answers written as
The answer is: $N$without\boxed{}(last number in the response): 0.9189. Paper: 88.4. lm_eval flexible-extract reads 0.0804 on this model because its regex takes the trailing$.from$\boxed{N}$.(1176 of 1319 docs).inclusionAI/LLaDA2.1-mini, tp2 (page 32), limit 10: flexible / strict 0.8 / 0.5, same before and after the page-size change.Speed Tests and Profiling
isinand a few small allocations; image and video prefill are unchanged.inclusionAI/LLaDA2.1-mini(page 32) ran about 20% slower in the 10-question check.Checklist
test/manual/models/test_ernie45_vl_visual_mask.py(manual, kept out of CI and nightly) covers the ERNIE-4.5-VL routing gate: a video-only prefill must mask its video tokens, and decode and text-only batches must not build the mask. The ERNIE-4.5-VL crash and the dLLM fixes need full models on XPU, which the runs above cover.CI States
Latest PR Test (Base): ❌ Run #40658998077
Latest PR Test (Extra): ❌ Run #40658998089
Latest PR Test (AMD ROCm 10): ❌ Run #40658998103