Repository navigation
Conversation
Selected with --attention-backend sage_attn_3 (or per request with --attention-backend-override sage_attn_3): on Intel XPU it maps to the new XPUSageAttention3Backend instead of falling back to SDPA. The kernels come from sgl-kernel-xpu (sgl_kernel.sageattn_v3 / sageattn_v3_hybrid); without them, or for a non-BF16 model, sage_attn_3 resolves to XPU flash attention (the XPU default, ring-capable) with a warning instead of failing in the model build. Configured with --attention-backend-config: - sage_v3_kind: hybrid (MXFP4 QK^T, MXFP8 PV; default) or pure (MXFP4 QK^T and PV). - sage_v3_bf16_blocks: block indices whose self-attention stays on BF16 flash attention, e.g. [0, 33, 34, 38] for Wan2.2 (all-MXFP4 output is grainy). Self-attention layers that fit SageAttention V3 (batch 1, MHA, head dim 128, non-causal, default softmax scale, BF16) run the kernel; cross-attention, other shapes, the listed blocks and LSE (ring) calls keep XPU flash attention, so the backend is ring-capable on XPU (sage_attn_3 joins RING_CAPABLE_ATTENTION_BACKENDS there); under --ring-degree > 1 every step is a ring (LSE) call, so it runs XPU flash attention, as the docs say. The kernel is probed once at layer construction, so a device it cannot run on fails before model load, and a BF16 block list on a model without `blocks.<N>.` prefixes is rejected. The kernel is registered as torch.ops.sglang.xpu_sage_v3_attn with a fake impl, so torch.compile traces it without a graph break. It uses torch.library.custom_op rather than direct_register_custom_op because XpuPlatform reports the CUDA dispatch key. The row-strided V slice of a fused QKV projection is passed in place. The UT checks cosine, relative L2 and the output/flash norm ratio, so a mis-scaled output fails. Kernel cases need Xe3P with the kernel; selection cases run on any XPU and the config parser cases anywhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Route the USPAttention ring branch on XPU to deep_symm.ring_attn when SGLANG_DIFFUSION_XPU_DEEPSYMM_RING_PRECISION is bf16, fp8, fp8_mixed or mxfp4 (default off keeps the torch ring). The variable is read once at import, so the default "off" path makes no torch.compiler.disable call and a bad value fails at startup. - fp8_mixed / mxfp4 run only for batch 1 in deep_symm, so B > 1 takes the bf16 ring (a Wan request with n=2 would otherwise raise on the first layer). - deep_symm's shm control block is node-local, so a ring group that spans hosts falls back to the torch ring (checked once per group, gathering hostnames over the SP group's gloo group, not a oneCCL object gather). - deep_symm keys its IPC arenas on the K/V shape and never evicts them. The first ring call of a request that needs a shape deep_symm does not hold frees the arenas that request has not used, before deep_symm allocates, so a new resolution, frame count or batch never sits next to the previous request's arenas (all ring ranks see the same shapes in the same order, so they clear together; the shapes are the resolved ones, e.g. from an I2V condition image). Arenas are also freed after the full-shape warmup probe (next to IPC_A2A.drop_staging) and in destroy_model_parallel, which also drops the id(ring_group)-keyed host check (a recreated group could otherwise reuse a stale id on some ranks only and hang the others) and does not let a failed peer handshake block group teardown. - A failed ring pass (e.g. a peer timeout) leaves a poisoned relay in deep_symm's cache, so the next ring call clears it first instead of every later request with that shape failing. - Layers with dropout keep the torch ring (deep_symm has none); the package is imported at the first eligible layer, so a missing deep-symm fails before the first ring collective; batch > 1 on the MX precisions logs once. The variable is documented in environment_variables.mdx. Tests drive USPAttention on two ranks (bit-equal to the direct DeepSymm call, torch ring when off) and cover B=2, cross-host, arena release on a shape change (never two shape sets held, no rebuild on a repeat), recovery after a failed pass, the worker marking each request before its forward, and teardown through the real destroy_model_parallel(), also when the deep_symm clear fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pSymm ring With ring parallelism on the XPU DeepSymm ring every self-attention layer takes the ring, so the Sage V3 BF16 block list had no effect and every block ran MXFP4 (grainy Wan2.2 video). Layers in the blocks listed by --attention-backend-config sage_v3_bf16_blocks now take the bf16 ring with --attention-backend fa (the XPU default) as well as sage_attn_3, as in the setup behind the 2-card Wan2.2 T2V numbers (blocks 0,33,34,38). sage_v3_kind has no effect under the ring; the docs say so. The list is read only when the DeepSymm ring is on, so constructing the default XPU backend still needs no server args. Under fa a layer without a `blocks.<N>.` index logs a warning and stays on the ring precision (sage_attn_3 rejects it at construction, as before). Tests: bf16 ring for listed blocks, and not for a block whose index only shares a prefix, on both XPU impls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
|
|
||
|
|
||
| def _bf16_blocks_from_config() -> frozenset[int]: | ||
| """--attention-backend-config sage_v3_bf16_blocks, on any XPU backend.""" |
There was a problem hiding this comment.
Under fa this key now sets DeepSymm ring precision with no Sage kernel in play, and the SGLANG_DIFFUSION_XPU_DEEPSYMM_RING_PRECISION docs row never mentions it, so fa + mxfp4 users won't find it. A backend-neutral name is free while #48059 is unmerged but needs an alias after. Worth renaming?
| with sage_attn_3.""" | ||
| if _xpu_count() < 1: | ||
| pytest.skip("needs an XPU") | ||
| pytest.importorskip("sgl_kernel", reason="needs sgl-kernel-xpu") |
There was a problem hiding this comment.
deepsymm_ring_precision calls _import_ring_attn_ipc_forward(), so on an XPU without deep-symm this test and the other ring_precision cases fail with ImportError (11 on a B60) instead of testing routing. Stubbing that import in the ring_precision fixture, as fake_arenas does, keeps them running on any XPU.
| <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`sage_attn_3`</td> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)", whiteSpace: "nowrap"}}>`SAGE_ATTN_3`</td> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Requires SageAttention3 installed per upstream instructions.</td> | ||
| <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>CUDA: requires SageAttention3 installed per upstream instructions. Intel XPU (Xe3P): SageAttention V3 from sgl-kernel-xpu (MXFP4) for batch-1, non-causal BF16 self-attention with head dim 128, no GQA and the default softmax scale; other layers, and every step under ring parallelism (<code>--ring-degree</code> > 1), keep XPU flash attention. Without the kernel, or for non-BF16 models, it falls back to XPU flash attention. Configure via <code>--attention-backend-config</code>.</td> |
There was a problem hiding this comment.
With the DeepSymm ring on, ring steps don't keep XPU flash attention; they run DeepSymm at the env precision, as the sage_v3_bf16_blocks row now says. This line, the "Ring steps run XPU flash attention" note in the support table and the Sage info_once in _resolve_sage_v3_kind still say otherwise.
Depends on #48059 (SageAttention V3) and #48322 (DeepSymm ring); this PR's own change is the top commit.
Motivation
With ring parallelism on the XPU DeepSymm ring every self-attention layer takes the ring, so
sage_v3_bf16_blockshas no effect there and every block runs MXFP4: the Wan2.2 video is grainy.Modifications
--attention-backend-config sage_v3_bf16_blockstake the bf16 DeepSymm ring, with--attention-backend fa(the XPU default) as well assage_attn_3.sage_v3_kindhas no effect under the ring; the docs say so.faa layer without ablocks.<N>.index logs a warning and stays on the ring precision (sage_attn_3rejects it at construction).Accuracy Tests
Wan2.2-T2V-A14B 1280x720x81, 40 steps, 2 cards, DeepSymm
mxfp4ring: all 40 blocks on MXFP4 is grainy; with blocks 0, 33, 34, 38 on the bf16 ring the video is clean (VBench, 8 prompts: subject consistency 0.971, imaging quality 0.706, aesthetic quality 0.613).Tested on 4x Intel XPU (Xe3P), torch 2.16, sgl-kernel-xpu main: the targeted unit tests pass, and the full
multimodal_gen/test/unitdirectory shows no new failures againstmain(same 42 failures / 3 errors, from tests that need CUDA ortrimesh).Speed Tests and Profiling
Wan2.2-T2V-A14B 1280x720x81, 40 steps, 2 Intel XPU (Xe3P) cards at 2500 MHz, cache off, MXFP8 GEMMs (#48060), DeepSymm
mxfp4ring + bf16 blocks 0,33,34,38: 469.8 s end to end (11.72 s/step, peak 80.1 GB).Checklist
🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ❌ Run #43435962507
Latest PR Test (Extra): ❌ Run #43435962195
Latest PR Test (AMD ROCm 10): ❌ Run #43435962345
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: