Skip to content

Guard get_masked_input_and_mask torch.compile against XPU, not just NPU - #43843

Closed
vshekhawat-hlab wants to merge 2 commits into
sgl-project:mainfrom
vshekhawat-hlab:fix/vocab-parallel-embedding-torch-compile-xpu
Closed

vshekhawat-hlab wants to merge 2 commits into
sgl-project:mainfrom
vshekhawat-hlab:fix/vocab-parallel-embedding-torch-compile-xpu

Conversation

@vshekhawat-hlab

@vshekhawat-hlab vshekhawat-hlab commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

VocabParallelEmbedding.get_masked_input_and_mask is decorated with @torch.compile(dynamic=True, backend=get_compiler_backend(), disable=_is_npu) — this guards against NPU but not XPU.

torch.compile/Inductor codegen on Intel XPU under tensor parallelism is unreliable for this function, causing either:

  • a scheduler watchdog timeout (stuck in Inductor's compile-worker subprocess, see torch/_inductor/compile_worker/subproc_pool.py in the thread dump), or
  • a hard crash directly inside the compiled function: RuntimeError: level_zero backend failed with error: 40 (UR_RESULT_ERROR_OUT_OF_RESOURCES)

This affects any TP>1 vision model that routes position-embedding lookups through VocabParallelEmbedding on intel_xpu, observed on meta-models/Muse-Glimmer-30B (TP4) and internlm/Intern-S2-Preview (TP8) — both crash/hang inside get_masked_input_and_mask via VocabParallelEmbedding.forward → _embed_local_shard.

sglang/srt/speculative/spec_utils.py already guards the identical pattern with disable=_is_npu or _is_xpu; this applies the same guard here.

Fix

Add _is_xpu = is_xpu() and change the decorator to disable=_is_npu or _is_xpu.

Test plan

Validated on real XPU hardware (4x Intel Arc Pro B60, TP4) against the exact lmsysorg/sglang:v0.5.21-xpu release image running meta-models/Muse-Glimmer-30B:

  • Before fix: crashes inside get_masked_input_and_mask (via muse_glimmer.py vision tower → position_embedding_table) on the very first vision-embedding lookup, every run.
  • After fix: the crash no longer occurs at that call site, confirmed across multiple runs — the model proceeds past vision embedding into decode.

Note: on this specific 24GB-per-card hardware at TP4, a separate, pre-existing mem_fraction_static headroom issue (unrelated to this fix) can still surface a UR_RESULT_ERROR_OUT_OF_RESOURCES later during decode at the default 0.85 static KV-cache fraction — confirmed independent of this change and resolved by lowering --mem-fraction-static. That is a deployment-config tuning issue, not an sglang code bug, and is being tracked separately.


CI States

Latest PR Test (Base): ❌ Run #40054640444
Latest PR Test (Extra): ❌ Run #40054640448
Latest PR Test (AMD ROCm 10): ❌ Run #40054640482

get_masked_input_and_mask in VocabParallelEmbedding is decorated with
@torch.compile(..., disable=_is_npu), guarding against NPU but not XPU.
torch.compile/Inductor codegen on Intel XPU under tensor parallelism is
unreliable, causing scheduler watchdog timeouts (stuck compiling) or
level_zero UR_RESULT_ERROR_OUT_OF_RESOURCES crashes directly inside the
compiled function, for any TP>1 vision model that routes position-embedding
lookups through VocabParallelEmbedding (e.g. meta-models/Muse-Glimmer-30B,
internlm/Intern-S2-Preview).

sglang/srt/speculative/spec_utils.py already guards the identical pattern
with disable=_is_npu or _is_xpu; this applies the same guard here.

@airMeng airMeng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you enable the related tests so I can include in #45201?

@github-actions

Copy link
Copy Markdown
Contributor

Required checks failed on a8f1a67; this PR cannot be merged until they pass.

PR title (run)

PR title must use "[Type] scope: title" or "[Type] [Hardware] scope: title". Type must be one of: Feat, Fix, Refactor, Perf, Doc, Test, CI, Chore. Scope is required and may be any module or model name, with unrestricted letter case (letters, numbers, dots, underscores, slashes, or hyphens). Hardware is optional; use a single tag without spaces. Use uppercase hardware tags (AMD, NVIDIA, NPU, MI300X), except for exactly these four spellings: Intel, Apple, Huawei, Ascend. Reverts must use GitHub's default format: Revert "original title". Use one space between fields and after the colon, with a non-empty title and no leading or trailing whitespace. Examples: "[Fix] attention: Fix KV cache indexing", "[Feat] qwen3: Add model support", "[Perf] [AMD] flash-attn: Optimize decode performance".
Edit the title; the check reruns on its own.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants