Repository navigation
[Sglang-Diffusion-XPU] Add support for sage_attn - #45485
ANSHUMAN87 wants to merge 3 commits into
Conversation
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
Performance:
|
| Shape (tokens x heads x head_dim) | fa |
sage_attn |
Speedup |
|---|---|---|---|
| Wan2.1 480p self-attention, 32760 x 12 x 128 | 94.26 ms | 81.29 ms | 1.16x |
| Long sequence, 16384 x 24 x 64 | 27.14 ms | 22.82 ms | 1.19x |
| Z-Image 1024², 4224 x 30 x 128 | 4.30 ms | 4.12 ms | 1.04x |
| 4096 x 24 x 128 | 3.21 ms | 3.26 ms | 0.98x |
| 4429 x 24 x 64 | 1.78 ms | 1.95 ms | 0.91x |
| Text cross-attention, 32760 q x 512 kv x 12 x 128 | 1.88 ms | 2.23 ms | 0.85x |
- Where it pays off: gains grow with sequence length.
- Where it loses: on short sequences and text cross-attention, the INT8 quantization prepass (about 0.5 ms per call at image sizes) costs more than the INT8 Q·Kᵀ saves.
- Accuracy vs fp32 attention (all test shapes, fp16 and bf16): cosine >= 0.9999, relative L2 error ~1.3e-2, LSE max abs error < 0.025.
End to end: sglang generate, fa / sage / sage / fa
Run with --warmup-mode request and SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1. "Steady step" is the median of denoise steps 1..N; step 0 includes the DiT upload from the CPU offload that is enabled automatically on 24 GB.
| Model | Backend | Steady denoise step | Request total | Peak memory | Quality vs fa |
|---|---|---|---|---|---|
| Wan2.1-T2V-1.3B, 832x480, 81 frames, 10 steps | fa |
8772 / 8781 ms | 107.86 / 107.96 s | 22.26 GB | reference (two runs bit-identical) |
sage_attn |
8035 / 8033 ms (-8.5%) | 100.50 / 100.50 s (-6.9%) | 22.26 GB | PSNR 32.40 dB mean / 30.24 dB worst frame; SSIM 0.931 / 0.923 | |
torch_sdpa |
10460 ms | 124.76 s | 22.26 GB | PSNR 35.73 / 34.58 dB; SSIM 0.951 / 0.947 | |
| Z-Image-Turbo, 1024x1024, 9 steps | fa |
878.6 / 878.4 ms | 17.31 / 17.30 s | 12.38 GB | reference (bit-identical) |
sage_attn |
875.4 / 874.3 ms (-0.4%) | 17.27 / 17.27 s (-0.2%) | 12.38 GB | PSNR 20.74 dB, SSIM 0.823 | |
torch_sdpa |
903.9 ms | 17.53 s | 12.38 GB | PSNR 23.58 dB, SSIM 0.880 |
- Why the gains differ: with
fa, attention is 66% of a Wan denoise step (5.77 s of 8.79 s) but only 15% of a Z-Image step (133 ms of 875 ms); GEMMs (58%) and element-wise ops dominate Z-Image. The profiler window covers 7 steps, normalized per step. - Wan quality passes the repo's video thresholds (SSIM 0.92 / PSNR 24 dB).
- Z-Image quality: the 8-step distilled model amplifies any attention rounding change; even exact
torch_sdpadiffers fromfaby a similar amount. The global flag also puts the Qwen3 text encoder on Sage. For image models,--component-attention-backends transformer=sage_attnkeeps the text encoder onfa. - Recipes: Z-Image follows the cookbook's Intel BMG recipe (1 GPU, default flags). Wan2.1 is not a documented BMG recipe; it was run to measure a long, attention-heavy sequence, with 10 steps instead of the default 50 because per-step cost is unchanged.
Commands
export ZE_AFFINITY_MASK=2 SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1
# BACKEND = fa | sage_attn | torch_sdpa
sglang generate --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
--prompt "A curious raccoon wearing a tiny scarf walks through a snowy pine forest at dusk, cinematic lighting" \
--width 832 --height 480 --num-frames 81 --num-inference-steps 10 --seed 42 \
--warmup-mode request --attention-backend $BACKEND --save-output --perf-dump-path wan_$BACKEND.json
sglang generate --model-path Tongyi-MAI/Z-Image-Turbo \
--prompt "A photorealistic portrait of an old fisherman mending nets on a wooden pier at sunrise, detailed skin texture" \
--width 1024 --height 1024 --seed 42 \
--warmup-mode request --attention-backend $BACKEND --save-output --perf-dump-path zimage_$BACKEND.json
JH...!
Performance post Driver and Triton Version upgrade(As it caused regression, had to change tile configs to chose between perf and accuracy):
|
| Shape (tokens x heads x head_dim) | fa |
sage_attn |
Speedup |
|---|---|---|---|
| Long sequence, 16384 x 24 x 64 | 26.92 ms | 21.24 ms | 1.27x |
| Wan2.1 480p self-attention, 32760 x 12 x 128 | 95.67 ms | 87.12 ms | 1.10x |
| Z-Image 1024², 4224 x 30 x 128 | 4.47 ms | 4.18 ms | 1.07x |
| 4096 x 24 x 128 | 3.42 ms | 3.37 ms | 1.01x |
| 4429 x 24 x 64 | 1.77 ms | 1.82 ms | 0.97x |
| Text cross-attention, 32760 q x 512 kv x 12 x 128 | 1.99 ms | 2.08 ms | 0.96x |
| 1024 x 24 x 64 | 0.11 ms | 0.14 ms | 0.79x |
- Where it pays off: gains grow with sequence length.
- Where it loses: on short sequences and text cross-attention, the INT8 quantization prepass and host launch cost exceed what INT8 Q·Kᵀ saves.
- Quantization: Q is quantized per 128 rows (64 for head_dim 64) and K per 64 rows, and each attention tile uses exactly one Q scale and one K scale. A per-row scale vector inside the loop made triton-xpu 3.8 route it through shared memory on every K/V iteration (3x slower). Same speed on triton-xpu 3.7.2.
- Accuracy vs fp32 attention (all test shapes, fp16 and bf16): cosine >= 0.9999, relative L2 error ~1.3e-2, LSE max abs error < 0.025.
End to end: sglang generate, fa / sage / sage / fa
Run with --warmup-mode request and SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1. "Steady step" is the median of denoise steps 1..N.
| Model | Backend | Steady denoise step | Request total | Peak memory | Quality vs fa |
|---|---|---|---|---|---|
| Wan2.1-T2V-1.3B, 832x480, 81 frames, 10 steps | fa |
8845 / 8848 ms | 100.84 / 100.89 s | 7.44 GB | reference (two runs bit-identical) |
sage_attn |
8248 / 8256 ms (-6.7%) | 94.87 / 94.88 s (-5.9%) | 7.44 GB | PSNR 33.26 dB mean / 30.70 dB worst frame; SSIM 0.940 / 0.936 | |
torch_sdpa |
10420 ms | 116.51 s | 7.44 GB | PSNR 33.69 / 31.02 dB; SSIM 0.943 / 0.939 | |
| Z-Image-Turbo, 1024x1024, 9 steps | fa |
875.9 / 876.8 ms | 9.37 / 9.38 s | 12.38 GB | reference (bit-identical) |
sage_attn |
872.8 / 873.3 ms (-0.4%) | 9.34 / 9.36 s (-0.3%) | 12.38 GB | PSNR 20.66 dB, SSIM 0.824 | |
torch_sdpa |
897.8 ms | 9.57 s | 12.38 GB | PSNR 23.15 dB, SSIM 0.873 |
- Why the gains differ: attention is about two thirds of a Wan denoise step but only about 15% of a Z-Image step. The Z-Image change is below the 1.5% end-to-end bar, so this PR claims no speedup for 1024² images.
- Wan quality passes the repo's video thresholds (SSIM 0.92 / PSNR 24 dB) and sits close to the exact
torch_sdpanoise floor. - Z-Image quality: the 8-step distilled model amplifies any attention rounding change; even exact
torch_sdpadiffers fromfaby a similar amount. For image models,--component-attention-backends transformer=sage_attnkeeps the text encoder onfa. - Wan on 24 GB XPU needs
--component-residency text_encoder=component-offloadwith any backend: the default warmup preload of its 21 GiB fp32 text encoder leaves too little memory for the Level Zero driver and the first denoise step fails withUR_RESULT_ERROR_OUT_OF_RESOURCES. This is unrelated to this PR;faandtorch_sdpafail the same way. - Recipes: Z-Image follows the cookbook's Intel BMG recipe (1 GPU, default flags). Wan2.1 is not a documented BMG recipe; it was run to measure a long, attention-heavy sequence, with 10 steps instead of 50 because per-step cost is unchanged.
Commands
export ZE_AFFINITY_MASK=2 SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1
# BACKEND = fa | sage_attn | torch_sdpa
sglang generate --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
--prompt "A curious raccoon wearing a tiny scarf walks through a snowy pine forest at dusk, cinematic lighting" \
--width 832 --height 480 --num-frames 81 --num-inference-steps 10 --seed 42 \
--warmup-mode request --component-residency text_encoder=component-offload \
--attention-backend $BACKEND --save-output --perf-dump-path wan_$BACKEND.json
sglang generate --model-path Tongyi-MAI/Z-Image-Turbo \
--prompt "A photorealistic portrait of an old fisherman mending nets on a wooden pier at sunrise, detailed skin texture" \
--width 1024 --height 1024 --seed 42 \
--warmup-mode request --attention-backend $BACKEND --save-output --perf-dump-path zimage_$BACKEND.jsonJH...!
35ad110 to
63fb179
Compare
|
Update: On XPU,
Setup:
Wan2.1-T2V-1.3B, 832x480, 81 frames, 10 steps (
Z-Image-Turbo, 1024x1024
For reference, torch_sdpa vs fa with the same settings gives 33.69 / 0.943 on Wan and 23.15 / 0.873 on Z-Image. Peak memory is the same for all backends: 7.44 GB for Wan and 12.38 GB for Z-Image. Attention kernel alone (bf16; median of interleaved runs):
ARK and Triton were timed in separate processes, each against its own fa run, because importing ARK sets an IGC environment variable for the whole process. The fa timings in the two processes agreed within 0.1%, except cross-attention at about 2.5%. Speedups are against fa. ARK is slower than fa on short sequences, but neither model above hits that range. Accuracy against an fp32 reference is the same for both kernels (relative error 1.30e-2 for ARK vs 1.32e-2 for Triton at the Z-Image shape). Tests
JH...! |
|
Required checks failed on 63fb179; this PR cannot be merged until they pass. PR title (run)
|
|
@mingfeima please help review, Thanks! |
kpjeeja
left a comment
There was a problem hiding this comment.
The title and description need rework. The title only describes part of the PR, and the description leaves out ARK, a new dependency, a setup requirement, its limitations, and test evidence. It also doesn't back up the performance claim in the docs. correct the PR title and msg according to sglang required format.
| ) | ||
| return None | ||
| # ARK only prints when its XPU library fails to load, then every call raises | ||
| if ark.xpu_lib is None: |
There was a problem hiding this comment.
except ImportError:
...
ARK only prints when its XPU library fails to load, then every call raises
if ark.xpu_lib is None:
Only ImportError is caught, and ark.xpu_lib is read without a guard. An OSError from ARK's native library during import,an installed auto-round-lib version without xpu_lib or sagev1, raises inside SageAttentionImpl.init. Model load then crashes instead of falling back to Triton. Use getattr(ark, "xpu_lib", None) and getattr(ark, "sagev1", None), catch (ImportError, OSError, RuntimeError), log the error, and return None.
| # auto-round-lib 0.16.0 returns one LSE per 16 query rows | ||
| logger.warning_once( | ||
| "Intel ARK SageAttention does not return a per-row LSE; calls that " | ||
| "need it (ring attention) run SGLang's Triton kernel instead" |
There was a problem hiding this comment.
if return_lse:
# auto-round-lib 0.16.0 returns one LSE per 16 query rows
logger.warning_once(
"Intel ARK SageAttention does not return a per-row LSE; calls that "
"need it (ring attention) run SGLang's Triton kernel instead"
)
Ring attention is a supported, expected setup (supports_ring_rotation returns True), so logging this at warning level looks like a fault in every ring deployment. Use info_once. Also reword the comment to say ARK's LSE layout can't be used for the ring merge, rather than the cryptic "one LSE per 16 query rows".
| "Intel ARK SageAttention does not return a per-row LSE; calls that " | ||
| "need it (ring attention) run SGLang's Triton kernel instead" | ||
| ) | ||
| elif head_dim in _ARK_HEAD_DIMS: |
There was a problem hiding this comment.
elif head_dim in _ARK_HEAD_DIMS:
return ark_sagev1(
_as_dense_nhd(query),
The ARK branch skips the input checks the Triton path runs in _validate (sage_attn_triton.py:353): fp16/bf16 only, matching Q/K/V shapes and dtypes, and causal requiring q_len == kv_len. A causal call where Q and KV lengths differ raises a clear ValueError on Triton, but could silently get a differently aligned mask on ARK. Run the same validation before dispatching to ARK, or send causal and q_len != kv_len to Triton. GQA does not need a guard, since the parity test already covers 8 query heads over 2 KV heads.
| "need it (ring attention) run SGLang's Triton kernel instead" | ||
| ) | ||
| elif head_dim in _ARK_HEAD_DIMS: | ||
| return ark_sagev1( |
There was a problem hiding this comment.
return ark_sagev1(
If ARK loads but fails on a particular call (sequence length, tile or device limits), the whole denoise run fails even though Triton would work. Rather than wrapping every call in try/except, which hides real errors and adds hot-path cost, consider running a tiny probe call inside _load_ark_sagev1 so a broken ARK is caught once at load.
| self.packed_trailing_padding = extra_impl_args.get( | ||
| "packed_trailing_padding", False | ||
| ) | ||
| self._ark_sagev1 = _load_ark_sagev1() if current_platform.is_xpu() else None |
There was a problem hiding this comment.
self._ark_sagev1 = _load_ark_sagev1() if current_platform.is_xpu() else None
Installing the standard XPU diffusion extra now switches every non-ring sage_attn call to ARK, which produces slightly different numerics, and nothing lets users opt out. If ARK regresses on a model or driver, the only workarounds are uninstalling a pinned dependency or breaking LD_LIBRARY_PATH. Consider extending the existing SGLANG_USE_SAGE_ATTENTION_IMPL env var with triton (force the in-tree kernel) and ark (require ARK, and error if it can't load), checked before _load_ark_sagev1(). Keep auto-selection as the default.
| ) | ||
| if dtype not in (torch.float16, torch.bfloat16): | ||
| raise ValueError(f"SageAttention requires fp16/bf16 but got dtype={dtype}") | ||
| logger.info("Using Sage Attention (Triton) backend on Intel XPU.") |
There was a problem hiding this comment.
logger.info("Using Sage Attention (Triton) backend on Intel XPU.")
This message is now wrong when ARK runs. Log "(ARK or Triton)", or rely on the kernel-specific message _load_ark_sagev1 already emits.
| "auto_round_kernel": fake_ark, | ||
| "auto_round_kernel.version": fake_version, | ||
| } | ||
| with patch.dict(sys.modules, fake_modules): |
There was a problem hiding this comment.
def test_unloadable_ark_library_falls_back_to_triton(self):
...
with patch.dict(sys.modules, fake_modules):
self.assertIsNone(sage_attn._load_ark_sagev1.__wrapped__())
The test name promises a fallback to Triton, but the test only checks that the loader returns None. Several paths are untested:
- ARK success: ark_sagev1 is called with is_causal, scale, tensor_layout="NHD" and dense strides, and Triton is not called.
- Head dim 96 falls back to Triton.
- The ImportError branch of the loader.
- The current_platform.is_xpu() gate in init.
Add MagicMock tests that assert on call_args.kwargs and on whether Triton was called.
| checkpoint-engine = ["checkpoint-engine @ git+https://github.com/MoonshotAI/checkpoint-engine.git@f772d1858204fa82bdb0436fdfec137f5de2972c"] | ||
| diffusion = [ | ||
| "addict==2.4.0", | ||
| "auto-round-lib==0.16.0", |
There was a problem hiding this comment.
"auto-round-lib==0.16.0",
This puts a native-library wheel in the default XPU diffusion extra, but the code and docs treat ARK as optional. If the pinned wheel isn't published for a platform or Python version on the index the XPU build uses, every pip install .[diffusion] there fails. Check that the wheel resolves (pip install .[diffusion] --dry-run against the XPU index), and consider a platform marker or a separate optional extra.
| pytestmark = pytest.mark.skipif( | ||
| ark_sagev1 is None, reason="Intel XPU with a loadable auto-round-lib required" | ||
| ) | ||
|
|
There was a problem hiding this comment.
ark_sagev1 = sage_attn._load_ark_sagev1() if is_xpu() else None
pytestmark = pytest.mark.skipif(
ark_sagev1 is None, reason="Intel XPU with a loadable auto-round-lib required"
)
This test will likely skip on XPU CI and still show green:
- pr-test-xpu.yml:175 installs . without the diffusion extra.
- docker/xpu.Dockerfile never sets LD_LIBRARY_PATH to /opt/venv/lib, which the docs in this PR say ARK needs.
Together with the comment above, the new default kernel ships with no effective CI coverage. Install auto-round-lib and
set LD_LIBRARY_PATH for the XPU CI job. When running on XPU CI, fail instead of skipping if ARK can't load.
Current change add triton path for sage_attn. The kernels are borrowed from original repo .
JH...!
CI States
Latest PR Test (Base): ❌ Run #40933660935
Latest PR Test (Extra): ❌ Run #40933661090
Latest PR Test (AMD ROCm 10): ❌ Run #40933660978