Repository navigation
Guard get_masked_input_and_mask torch.compile against XPU, not just NPU - #43843
vshekhawat-hlab wants to merge 2 commits into
Conversation
get_masked_input_and_mask in VocabParallelEmbedding is decorated with @torch.compile(..., disable=_is_npu), guarding against NPU but not XPU. torch.compile/Inductor codegen on Intel XPU under tensor parallelism is unreliable, causing scheduler watchdog timeouts (stuck compiling) or level_zero UR_RESULT_ERROR_OUT_OF_RESOURCES crashes directly inside the compiled function, for any TP>1 vision model that routes position-embedding lookups through VocabParallelEmbedding (e.g. meta-models/Muse-Glimmer-30B, internlm/Intern-S2-Preview). sglang/srt/speculative/spec_utils.py already guards the identical pattern with disable=_is_npu or _is_xpu; this applies the same guard here.
|
Required checks failed on a8f1a67; this PR cannot be merged until they pass. PR title (run)
|
Summary
VocabParallelEmbedding.get_masked_input_and_maskis decorated with@torch.compile(dynamic=True, backend=get_compiler_backend(), disable=_is_npu)— this guards against NPU but not XPU.torch.compile/Inductor codegen on Intel XPU under tensor parallelism is unreliable for this function, causing either:
torch/_inductor/compile_worker/subproc_pool.pyin the thread dump), orRuntimeError: level_zero backend failed with error: 40 (UR_RESULT_ERROR_OUT_OF_RESOURCES)This affects any TP>1 vision model that routes position-embedding lookups through
VocabParallelEmbeddingonintel_xpu, observed onmeta-models/Muse-Glimmer-30B(TP4) andinternlm/Intern-S2-Preview(TP8) — both crash/hang insideget_masked_input_and_maskviaVocabParallelEmbedding.forward→_embed_local_shard.sglang/srt/speculative/spec_utils.pyalready guards the identical pattern withdisable=_is_npu or _is_xpu; this applies the same guard here.Fix
Add
_is_xpu = is_xpu()and change the decorator todisable=_is_npu or _is_xpu.Test plan
Validated on real XPU hardware (4x Intel Arc Pro B60, TP4) against the exact
lmsysorg/sglang:v0.5.21-xpurelease image runningmeta-models/Muse-Glimmer-30B:get_masked_input_and_mask(viamuse_glimmer.pyvision tower →position_embedding_table) on the very first vision-embedding lookup, every run.Note: on this specific 24GB-per-card hardware at TP4, a separate, pre-existing
mem_fraction_staticheadroom issue (unrelated to this fix) can still surface aUR_RESULT_ERROR_OUT_OF_RESOURCESlater during decode at the default 0.85 static KV-cache fraction — confirmed independent of this change and resolved by lowering--mem-fraction-static. That is a deployment-config tuning issue, not an sglang code bug, and is being tracked separately.CI States
Latest PR Test (Base): ❌ Run #40054640444
Latest PR Test (Extra): ❌ Run #40054640448
Latest PR Test (AMD ROCm 10): ❌ Run #40054640482