Repository navigation
[Perf] [XPU] diffusion: Use the AOT SYCL fused residual-norm-scale-shift kernel - #41401
Merged
Merged
Conversation
Contributor
Author
|
Sycl kernel PR in sgl-kernel-xpu: sgl-project/sgl-kernel-xpu#497 |
siju-samuel
reviewed
Sep 29, 2026
This was referenced Sep 30, 2026
ckvermaAI
marked this pull request as ready for review
October 5, 2026 17:09
ckvermaAI
requested review from
AgainstEntropy,
BBuf,
HaiShaw,
kevin-mii,
mickqian,
niehen6174,
ping1jing2,
yichiche and
yingluosanqian
as code owners
October 5, 2026 17:09
Contributor
|
/tag-run-ci-label |
mingfeima
requested changes
Oct 16, 2026
mingfeima
left a comment
Collaborator
There was a problem hiding this comment.
try to simplify the code.
AI usually put too many self-guarded logic, time and time again.
Collaborator
|
@ckvermaAI perf data with and without this PR? |
Why: on XPU, ScaleResidualLayerNormScaleShift.forward_xpu runs a triton kernel that blocks regional torch.compile of a Wan2.2 DiT block. Inductor re-lowers it and Intel triton fails with "PassManager::run failed" in the first denoise step. What: add sgl-kernel-xpu's AOT SYCL kernel for the same op as a backend under kernels/ops/diffusion/norm/ and prefer it in forward_xpu, keeping triton as the fallback. The op is a registered custom op, which inductor keeps opaque, and a register_fake companion lets Dynamo trace it. The eligibility check reuses the triton one and adds the kernel's dtype rule: gate, weight and bias share one dtype, and scale and shift are cast to it only when the cast is exact. That keeps Wan's bf16 null shift/scale on the SYCL path without touching the model code. Result on Wan2.2-I2V-A14B, TP=4 on 4x Arc Pro B60, 832x480, 21 frames, 20 steps: eager denoise: 100.26 s before, 100.07 s after regional compile denoise: fails before, 82.74 s after (1.21x vs eager) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
Author
@mingfeima Updated the description with performance data and address your review comments |
mingfeima
approved these changes
Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On XPU,
ScaleResidualLayerNormScaleShift.forward_xpuruns a triton kernel that blocks regionaltorch.compileof a Wan2.2 DiT block. Inductor re-lowers the kernel and Intel triton fails to compile the result:This happens in the first denoise step, so compiled Wan2.2 cannot run on XPU today. sgl-kernel-xpu ships an AOT SYCL kernel for the same op (sgl-project/sgl-kernel-xpu#497), in the
sglang-kernel-xpu0.4.0 wheel already pinned inpyproject_xpu.toml. It is a registered custom op, so inductor keeps it opaque and compiled regions call it unchanged.Modifications
kernels/ops/diffusion/norm/scale_residual_norm_scale_shift_sycl.py(new): the SYCL backend, exported through theops.diffusionfacade.can_use_fused_scale_residual_norm_scale_shift_syclreuses the triton eligibility check for shape, device and contiguity, and adds the kernel's dtype rule. Gate, weight and bias must share one dtype. Scale and shift are cast to it only when the cast is exact (same dtype, or an upcast to fp32); any other mix stays on triton.fused_scale_residual_norm_scale_shift_syclcallstorch.ops.sgl_kerneldirectly, so operands are validated once by the op's C++ checks.register_fakecompanion lets Dynamo trace the op, since sgl-kernel-xpu registers only the XPU implementation.layernorm.py:forward_xpuprefers the SYCL kernel, then triton, thenforward_native. The import sits in the existing_is_xpublock next tofused_inplace_qknorm_rope, so other platforms never load it.No model code changes. Wan's self-attention site passes bf16 1-element null shift/scale next to an fp32 gate and fp32 affine parameters. The exact upcast keeps that site on the SYCL path, so other devices see no change.
Accuracy Tests
Speed Tests and Profiling
Wan2.2-I2V-A14B, TP=4 on 4x Arc Pro B60, 832x480 resolution, 21 frames, 20 steps, warm caches, first run of the session discarded:
--enable-torch-compilealso compilesvae.decode. That is unrelated to this PR.Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ✅ Run #39526484868
Latest PR Test (Extra): ❌ Run #39526484892
Latest PR Test (AMD ROCm 10): ❌ Run #39526484808