Skip to content

[Perf] [XPU] diffusion: Use the AOT SYCL fused residual-norm-scale-shift kernel - #41401

Merged
mingfeima merged 3 commits into
sgl-project:mainfrom
ckvermaAI:wan-sycl
Oct 23, 2026
Merged

mingfeima merged 3 commits into
sgl-project:mainfrom
ckvermaAI:wan-sycl

Conversation

@ckvermaAI

@ckvermaAI ckvermaAI commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

On XPU, ScaleResidualLayerNormScaleShift.forward_xpu runs a triton kernel that blocks regional torch.compile of a Wan2.2 DiT block. Inductor re-lowers the kernel and Intel triton fails to compile the result:

triton/backends/intel/compiler.py: make_llir -> RuntimeError: PassManager::run failed

This happens in the first denoise step, so compiled Wan2.2 cannot run on XPU today. sgl-kernel-xpu ships an AOT SYCL kernel for the same op (sgl-project/sgl-kernel-xpu#497), in the sglang-kernel-xpu 0.4.0 wheel already pinned in pyproject_xpu.toml. It is a registered custom op, so inductor keeps it opaque and compiled regions call it unchanged.

Modifications

  • kernels/ops/diffusion/norm/scale_residual_norm_scale_shift_sycl.py (new): the SYCL backend, exported through the ops.diffusion facade.
    • can_use_fused_scale_residual_norm_scale_shift_sycl reuses the triton eligibility check for shape, device and contiguity, and adds the kernel's dtype rule. Gate, weight and bias must share one dtype. Scale and shift are cast to it only when the cast is exact (same dtype, or an upcast to fp32); any other mix stays on triton.
    • fused_scale_residual_norm_scale_shift_sycl calls torch.ops.sgl_kernel directly, so operands are validated once by the op's C++ checks.
    • A register_fake companion lets Dynamo trace the op, since sgl-kernel-xpu registers only the XPU implementation.
  • layernorm.py: forward_xpu prefers the SYCL kernel, then triton, then forward_native. The import sits in the existing _is_xpu block next to fused_inplace_qknorm_rope, so other platforms never load it.

No model code changes. Wan's self-attention site passes bf16 1-element null shift/scale next to an fp32 gate and fp32 affine parameters. The exact upcast keeps that site on the SYCL path, so other devices see no change.

Accuracy Tests

Speed Tests and Profiling

Wan2.2-I2V-A14B, TP=4 on 4x Arc Pro B60, 832x480 resolution, 21 frames, 20 steps, warm caches, first run of the session discarded:

CCL_ZE_IPC_EXCHANGE=pidfd sglang generate \
  --model-path Wan-AI/Wan2.2-I2V-A14B-Diffusers --num-gpus 4 --tp-size 4 \
  --text-encoder-cpu-offload --pin-cpu-memory \
  --image-path <i2v_input.JPG> --prompt "<prompt>" \
  --height 480 --width 832 --num-frames 21 --num-inference-steps 20 --seed 42 \
  [--enable-torch-compile --regional-compile]   # SGLANG_TORCH_COMPILE_MODE=default
Mode Denoise before (s) Denoise after (s)
eager 100.43 / 100.10 100.10 / 100.05
regional compile fails at step 0 82.74 / 82.74
  • Eager is unchanged. Both kernels cost about 1.8 ms per call on the GPU at the Wan shape, and the host queues all steps well ahead of the device.
  • Regional compile now runs, and its denoise time is 17% below eager (1.21x).
  • Compiled end-to-end time is still higher than eager because --enable-torch-compile also compiles vae.decode. That is unrelated to this PR.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ✅ Run #39526484868
Latest PR Test (Extra): ❌ Run #39526484892
Latest PR Test (AMD ROCm 10): ❌ Run #39526484808

@github-actions github-actions Bot added diffusion SGLang Diffusion jit-kernel labels Sep 27, 2026
@ckvermaAI

Copy link
Copy Markdown
Contributor Author

Sycl kernel PR in sgl-kernel-xpu: sgl-project/sgl-kernel-xpu#497

@siju-samuel siju-samuel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/tag-run-ci-label

@siju-samuel

Copy link
Copy Markdown
Contributor

/tag-run-ci-label

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 6, 2026
@ckvermaAI ckvermaAI changed the title [XPU][Diffusion] Use the AOT SYCL fused residual-norm-scale-shift kernel [Perf] [XPU] diffusion: Use the AOT SYCL fused residual-norm-scale-shift kernel Oct 16, 2026

@mingfeima mingfeima left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try to simplify the code.

AI usually put too many self-guarded logic, time and time again.

@mingfeima mingfeima added intel xpu intel gpu with device `torch.xpu` labels Oct 16, 2026
@mingfeima

Copy link
Copy Markdown
Collaborator

@ckvermaAI perf data with and without this PR?

Why: on XPU, ScaleResidualLayerNormScaleShift.forward_xpu runs a triton
kernel that blocks regional torch.compile of a Wan2.2 DiT block. Inductor
re-lowers it and Intel triton fails with "PassManager::run failed" in the
first denoise step.

What: add sgl-kernel-xpu's AOT SYCL kernel for the same op as a backend under
kernels/ops/diffusion/norm/ and prefer it in forward_xpu, keeping triton as
the fallback. The op is a registered custom op, which inductor keeps opaque,
and a register_fake companion lets Dynamo trace it. The eligibility check
reuses the triton one and adds the kernel's dtype rule: gate, weight and bias
share one dtype, and scale and shift are cast to it only when the cast is
exact. That keeps Wan's bf16 null shift/scale on the SYCL path without
touching the model code.

Result on Wan2.2-I2V-A14B, TP=4 on 4x Arc Pro B60, 832x480, 21 frames,
20 steps:

  eager denoise:            100.26 s before, 100.07 s after
  regional compile denoise: fails before, 82.74 s after (1.21x vs eager)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@ckvermaAI

Copy link
Copy Markdown
Contributor Author

@ckvermaAI perf data with and without this PR?

@mingfeima Updated the description with performance data and address your review comments

@mingfeima
mingfeima merged commit 76c382e into sgl-project:main Oct 23, 2026
171 of 184 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

diffusion SGLang Diffusion intel jit-kernel run-ci CI: run the baseline test suite on this PR xpu intel gpu with device `torch.xpu`

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants