Skip to content

[Fix] [XPU] ernie45_vl/dllm: Build the visual-token mask only for image or video prefill, and fix dLLM serving - #41730

Closed
desmond-intel wants to merge 6 commits into
sgl-project:mainfrom
desmond-intel:model-serve/ernie-vl-usm-fix
Closed

desmond-intel wants to merge 6 commits into
sgl-project:mainfrom
desmond-intel:model-serve/ernie-vl-usm-fix

Conversation

@desmond-intel

@desmond-intel desmond-intel commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

This PR has two Intel XPU fixes: the ERNIE-4.5-VL visual-token mask, and dLLM serving on intel_xpu, which moved here from #41773.

Motivation

1. ERNIE-4.5-VL: visual-token mask on every decode step

On Intel XPU, baidu/ERNIE-4.5-VL-28B-A3B-PT crashes during docvqa after a few dozen documents:

RuntimeError: level_zero backend failed with error: 40 (UR_RESULT_ERROR_OUT_OF_RESOURCES)

Ernie4_5_VLMoeForConditionalGeneration.forward calls _set_visual_token_mask on every forward, including every decode step. Each call allocates small device tensors (_visual_token_ids_tensor_cache.to(device), torch.as_tensor(pad_values, device=...), torch.cat, torch.isin). The level_zero backend does not reclaim them, so one prefill plus N decode steps per document adds up until device memory runs out.

The mask only decides which tokens go to the vision experts (ernie45_moe_vl.py). A None mask and an all-False mask take the same text-experts path, which is also the path CUDA-graph capture uses. Decode tokens are generated text, so the per-step mask is always all-False.

2. dLLM serving on intel_xpu

Diffusion LMs (--dllm-algorithm) on --attention-backend intel_xpu hit two XPU bugs:

  1. Crash at warmup (inclusionAI/LLaDA2.1-mini, JetLM/SDAR-8B-Chat): AttributeError: 'NoneType' object has no attribute 'shape' in XPUAttentionBackend.init_forward_metadata. ForwardMode.DLLM_EXTEND is in is_extend() but not in is_extend_or_draft_extend_or_mixed(), which guards the XPU extend-metadata branch. On CUDA, DLLM_EXTEND runs via CUDA graph and never reaches this eager branch; on XPU it matches no branch and leaves metadata.page_table = None.
  2. Blank output (JetLM/SDAR-8B-Chat): _dllm_page_size sets page_size to the dLLM block size (4 for SDAR, 32 for LLaDA2), overriding the intel_xpu 64/128 page constraint. The sgl-kernel-xpu paged kernels are exact at page 1/64/128. At page 2/4/8/16, a query block whose KV spans more than one page returns wrong output (max abs err 1.4 to 3.9 at page 4), and SDAR generates \n\n\n....

Modifications

1. ERNIE-4.5-VL

  • In python/sglang/srt/models/ernie45_vl.py, build the mask only for non-decode batches that contain image or video inputs; otherwise set visual_token_mask = None.
  • Image and video prefill/extend batches still build the mask, so vision routing is unchanged. Decode and text-only batches take the text-experts path, as before.

2. dLLM

python/sglang/srt/layers/attention/xpu_backend.py:

  • Route DLLM_EXTEND into the extend-metadata branch (a multi-query block forward, so extend-shaped metadata is correct).
  • For non-MLA attention without DCP, index KV per token (page_size = 1) when the page size is not 1/64/128. This reuses the existing page-1 path; the KV pool layout is unchanged.

Both dLLM changes are in the XPU attention backend only.

#42627 carries earlier versions of both fixes, without the video-only mask fix and the page-size fallback.

Accuracy Tests

1. ERNIE-4.5-VL

Intel Arc Pro B60, tp-size 4, --attention-backend intel_xpu, lmms_eval docvqa_val:

Run Docs Result
Before, serialized (num_concurrent=1, --max-running-requests 1) ~44 crash, error 40
After, same settings 80/80 368,682 tokens, no error 40
After, num_concurrent=8, --mem-fraction-static 0.70, no serialization flags 150/150 anls 0.9137, no error 40

The anls run sets enable_thinking=false in the chat template. ERNIE-4.5-VL opens a <think> block by default, and docvqa's short generation budget cuts it off before the answer.

Reproduce (serialized run):

SGLANG_USE_SGL_XPU=1 python3 -m sglang.launch_server --model-path baidu/ERNIE-4.5-VL-28B-A3B-PT \
  --device xpu --tp-size 4 --mem-fraction-static 0.85 --trust-remote-code --port 30000 \
  --context-length 8192 --disable-radix-cache --disable-overlap-schedule --max-running-requests 1 \
  --enable-multimodal --attention-backend intel_xpu

python3 -m lmms_eval --model openai_compatible \
  --model_args model_version=baidu/ERNIE-4.5-VL-28B-A3B-PT,num_concurrent=1 \
  --tasks docvqa_val --batch_size 16 --limit 80

2. dLLM

Intel Arc Pro B60, --dllm-algorithm LowConfidence --attention-backend intel_xpu, gsm8k 5-shot chat.

JetLM/SDAR-8B-Chat, tp1:

docs \boxed{} answer correct
crash fix only 10 0/10 (blank output)
this PR 10 7/10 (triton backend: 7/10)
this PR 1319 0.8635 (1139/1319)

Counting answers written as The answer is: $N$ without \boxed{} (last number in the response): 0.9189. Paper: 88.4. lm_eval flexible-extract reads 0.0804 on this model because its regex takes the trailing $. from $\boxed{N}$. (1176 of 1319 docs).

inclusionAI/LLaDA2.1-mini, tp2 (page 32), limit 10: flexible / strict 0.8 / 0.5, same before and after the page-size change.

python -m sglang.launch_server --model-path JetLM/SDAR-8B-Chat --trust-remote-code \
  --attention-backend intel_xpu --dllm-algorithm LowConfidence --tp-size 1

Speed Tests and Profiling

  • ERNIE-4.5-VL: no speed measurement. Each decode step now skips one isin and a few small allocations; image and video prefill are unchanged.
  • dLLM: the per-token fallback applies only when the page size is outside 1/64/128. inclusionAI/LLaDA2.1-mini (page 32) ran about 20% slower in the 10-question check.

Checklist

  • Format your code according to the Format code with pre-commit.
  • Add unit tests according to the Run and add unit tests. test/manual/models/test_ernie45_vl_visual_mask.py (manual, kept out of CI and nightly) covers the ERNIE-4.5-VL routing gate: a video-only prefill must mask its video tokens, and decode and text-only batches must not build the mask. The ERNIE-4.5-VL crash and the dLLM fixes need full models on XPU, which the runs above cover.
  • Update documentation according to Write documentations. Not needed.
  • Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
  • Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): ❌ Run #40658998077
Latest PR Test (Extra): ❌ Run #40658998089
Latest PR Test (AMD ROCm 10): ❌ Run #40658998103

@desmond-intel
desmond-intel force-pushed the model-serve/ernie-vl-usm-fix branch 2 times, most recently from d15d722 to 8c78773 Compare October 8, 2026 07:44
@desmond-intel

Copy link
Copy Markdown
Contributor Author

Closing this as it is getting merged as part of #42627.

Ernie4_5_VLMoeForConditionalGeneration.forward built the visual-token routing
mask on every forward, including every decode step. On the intel_xpu (level_zero)
backend the small per-step device allocations in _set_visual_token_mask (.to, cat,
isin) are never reclaimed, so a docvqa run exhausts USM and dies with
'level_zero backend failed with error: 40 (UR_RESULT_ERROR_OUT_OF_RESOURCES)'
after a few dozen documents.

The mask only routes prefill/extend image tokens to the vision experts; decode
tokens are always text, where an all-False mask is equivalent to None (the
text-experts path, same as CUDA-graph capture). Compute it only for
non-decode batches that contain image inputs.

Validated on Intel XPU (tp4, docvqa_val, serialized): 80/80 docs, 368k tokens,
zero error-40 (previously crashed at ~44 docs); answers unchanged.
@desmond-intel

Copy link
Copy Markdown
Contributor Author

Reopening so this fix can be reviewed on its own; the same change is also part of #42627. If that merges first, I'll close this one.

@desmond-intel desmond-intel reopened this Oct 14, 2026
@desmond-intel
desmond-intel force-pushed the model-serve/ernie-vl-usm-fix branch from 8c78773 to 69662a3 Compare October 14, 2026 07:41
@desmond-intel desmond-intel changed the title [Bugfix] ernie45_vl: only build the visual-token mask for image prefill batches [Fix] ernie45_vl: Only build the visual-token mask for image prefill batches Oct 15, 2026
@desmond-intel

Copy link
Copy Markdown
Contributor Author

CUDA check on 1x NVIDIA H200 NVL (tp 1). Both runs use the same image (lmsysorg/sglang:nightly-dev-20261015-a6a8c560) with the sglang Python source taken from each commit:

  • before: merge-base 6daae9c98e
  • after: this PR 69662a3a88
Task Before After
docvqa_val, 1000 docs, anls 0.9269 0.9270
docvqa_val, all 5349 docs, anls 0.9221 0.9224
gsm8k, 1319 questions, strict / flexible 0.7005 / 0.8514 0.7036 / 0.8560

No regression on CUDA. docvqa covers image prefill, which still builds the mask. gsm8k covers text-only batches, which now skip it.

sglang serve --model-path baidu/ERNIE-4.5-VL-28B-A3B-PT --tp-size 1 --mem-fraction-static 0.85 \
  --trust-remote-code --context-length 8192 --enable-multimodal   # gsm8k: same, without --enable-multimodal

python3 -m lmms_eval --model openai --model_args model_version=baidu/ERNIE-4.5-VL-28B-A3B-PT,num_concurrent=8 \
  --tasks docvqa_val --batch_size 8 --limit 1000 --seed 42 --gen_kwargs enable_thinking=false,max_new_tokens=64

lm_eval --model local-completions \
  --model_args pretrained=baidu/ERNIE-4.5-VL-28B-A3B-PT,base_url=http://127.0.0.1:30000/v1/completions,num_concurrent=16 \
  --tasks gsm8k --num_fewshot 5 --batch_size 16 --seed 42 --gen_kwargs max_gen_toks=1024

@pramodkumar-habanalabs pramodkumar-habanalabs added the run-ci CI: run the baseline test suite on this PR label Oct 27, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Oct 27, 2026
Source: sgl-project#41730
Source-head: 69662a3

Mechanically replayed source net patch onto origin/main b74fd39.
@airMeng

airMeng commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

@desmond-intel — forwarding @mingfeima’s comment from #46623:
#46623 (comment)

Video-only prefill does not satisfy contains_image_inputs(), so this clears the visual mask and routes video tokens through text experts. This also changes CUDA behavior.

Keep the mask construction for both image and video input!
better to add a video-only routing regression test.

@desmond-intel

Copy link
Copy Markdown
Contributor Author

@mingfeima @airMeng Fixed in dfb591e. The mask is now built whenever a prefill batch has image or video inputs, so video-only prefill routes its video tokens through the vision experts again, as on main. Decode and text-only batches still skip it, which is what the XPU fix needs.

Added test/registered/unit/models/test_ernie45_vl_visual_mask.py (CPU): a video-only prefill must mask its video tokens, and decode and text-only batches must not build the mask. The video case fails without the fix; both pass with it, on this branch and on current main.

The mask was only built when the batch had image inputs, so a video-only
prefill got no mask and its video tokens went through the text experts on
every device. Build it whenever the batch has image or video inputs; decode
and text-only batches still skip it.

Add test/registered/unit/models/test_ernie45_vl_visual_mask.py (CPU): a
video-only prefill must mask its video tokens, and decode and text-only
batches must not build the mask. The video case fails without this change.
@desmond-intel desmond-intel changed the title [Fix] ernie45_vl: Only build the visual-token mask for image prefill batches [Fix] [XPU] ernie45_vl/dllm: Build the visual-token mask only for image or video prefill, and fix dLLM serving Oct 28, 2026
Moved here from sgl-project#41773 (its two commits, squashed).

Build extend metadata for DLLM_EXTEND (fix dLLM decode crash):
DLLM_EXTEND is excluded from is_extend_or_draft_extend_or_mixed(). On XPU,
where dLLM runs eager rather than via CUDA graph, init_forward_metadata fell
through every branch and left metadata.page_table None, crashing at the
page_size>1 strided-page step (AttributeError: 'NoneType' object has no
attribute 'shape'). Route DLLM_EXTEND through the extend branch so page_table
is built. Verified on intel_xpu tp2: LLaDA2.1-mini gsm8k (--dllm-algorithm
LowConfidence) serves and scores flexible-extract 0.8 on 5 samples, no crash.

Index KV per token for unsupported page sizes (fix dLLM blank output):
dLLM serving sets page_size to the diffusion block size (_dllm_page_size),
which overrides the intel_xpu 64/128 page constraint. SDAR uses block 4, so
the backend passes a page-4 KV cache and page table to flash_attn_with_kvcache.
The XPU paged kernels are exact at page 1, 64 and 128; at page 2/4/8/16 a
query block (q_len > 1) whose KV spans more than one page returns wrong
output, and decode raises "Unsupported page size". SDAR-8B on intel_xpu
produces blank output as a result.

For non-MLA attention without DCP, fall back to per-token indexing
(page_size = 1) when the page size is outside {1, 64, 128}. The KV pool
layout is unchanged; the existing page-1 path views the cache as
[-1, 1, H, D] and skips the page-table striding, which is valid for any
physical page size.
Run test_ernie45_vl_visual_mask.py in stage-a-test-1-gpu-xpu instead of the
CPU base suite.
Keep it out of CI and nightly: stage-a-test-1-gpu-xpu also runs in the
nightly XPU wheel job. Run it locally with
python3 -m pytest test/manual/models/test_ernie45_vl_visual_mask.py.
airMeng pushed a commit to airMeng/sglang that referenced this pull request Oct 29, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Oct 29, 2026
@desmond-intel

Copy link
Copy Markdown
Contributor Author

This landed in main through #46623 (80afdc1), which carries this PR's head 5328b98: the ERNIE-4.5-VL image-or-video mask gate and the dLLM fix from #41773. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants