Skip to content

[Feat] [Intel] diffusion: Keep sage_v3_bf16_blocks at BF16 in the DeepSymm ring - #50331

Open
ckvermaAI wants to merge 3 commits into
sgl-project:mainfrom
ckvermaAI:dev/ckvermaAI/diffusion-xpu-deepsymm-ring-bf16-blocks
Open

ckvermaAI wants to merge 3 commits into
sgl-project:mainfrom
ckvermaAI:dev/ckvermaAI/diffusion-xpu-deepsymm-ring-bf16-blocks

Conversation

@ckvermaAI

@ckvermaAI ckvermaAI commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Depends on #48059 (SageAttention V3) and #48322 (DeepSymm ring); this PR's own change is the top commit.

Motivation

With ring parallelism on the XPU DeepSymm ring every self-attention layer takes the ring, so sage_v3_bf16_blocks has no effect there and every block runs MXFP4: the Wan2.2 video is grainy.

Modifications

  • Layers in the blocks listed by --attention-backend-config sage_v3_bf16_blocks take the bf16 DeepSymm ring, with --attention-backend fa (the XPU default) as well as sage_attn_3. sage_v3_kind has no effect under the ring; the docs say so.
  • The list is read only when the DeepSymm ring is on, so constructing the default XPU backend needs no server args. Under fa a layer without a blocks.<N>. index logs a warning and stays on the ring precision (sage_attn_3 rejects it at construction).
  • Test: listed blocks take the bf16 ring and a block whose index only shares a prefix does not, on both XPU impls.

Accuracy Tests

Wan2.2-T2V-A14B 1280x720x81, 40 steps, 2 cards, DeepSymm mxfp4 ring: all 40 blocks on MXFP4 is grainy; with blocks 0, 33, 34, 38 on the bf16 ring the video is clean (VBench, 8 prompts: subject consistency 0.971, imaging quality 0.706, aesthetic quality 0.613).

Tested on 4x Intel XPU (Xe3P), torch 2.16, sgl-kernel-xpu main: the targeted unit tests pass, and the full multimodal_gen/test/unit directory shows no new failures against main (same 42 failures / 3 errors, from tests that need CUDA or trimesh).

Speed Tests and Profiling

Wan2.2-T2V-A14B 1280x720x81, 40 steps, 2 Intel XPU (Xe3P) cards at 2500 MHz, cache off, MXFP8 GEMMs (#48060), DeepSymm mxfp4 ring + bf16 blocks 0,33,34,38: 469.8 s end to end (11.72 s/step, peak 80.1 GB).

Checklist

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #43435962507
Latest PR Test (Extra): ❌ Run #43435962195
Latest PR Test (AMD ROCm 10): ❌ Run #43435962345

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

Chetan Verma and others added 3 commits November 13, 2026 04:51
Selected with --attention-backend sage_attn_3 (or per request with
--attention-backend-override sage_attn_3): on Intel XPU it maps to the new
XPUSageAttention3Backend instead of falling back to SDPA. The kernels come
from sgl-kernel-xpu (sgl_kernel.sageattn_v3 / sageattn_v3_hybrid); without
them, or for a non-BF16 model, sage_attn_3 resolves to XPU flash attention
(the XPU default, ring-capable) with a warning instead of failing in the
model build.
Configured with --attention-backend-config:
- sage_v3_kind: hybrid (MXFP4 QK^T, MXFP8 PV; default) or pure (MXFP4 QK^T and PV).
- sage_v3_bf16_blocks: block indices whose self-attention stays on BF16 flash
  attention, e.g. [0, 33, 34, 38] for Wan2.2 (all-MXFP4 output is grainy).

Self-attention layers that fit SageAttention V3 (batch 1, MHA, head dim 128,
non-causal, default softmax scale, BF16) run the kernel; cross-attention,
other shapes, the listed blocks and LSE (ring) calls keep XPU flash
attention, so the backend is ring-capable on XPU (sage_attn_3 joins
RING_CAPABLE_ATTENTION_BACKENDS there); under --ring-degree > 1 every step
is a ring (LSE) call, so it runs XPU flash attention, as the docs say. The kernel is probed once at layer
construction, so a device it cannot run on fails before model load, and a
BF16 block list on a model without `blocks.<N>.` prefixes is rejected.

The kernel is registered as torch.ops.sglang.xpu_sage_v3_attn with a fake
impl, so torch.compile traces it without a graph break. It uses
torch.library.custom_op rather than direct_register_custom_op because
XpuPlatform reports the CUDA dispatch key. The row-strided V slice of a
fused QKV projection is passed in place.

The UT checks cosine, relative L2 and the output/flash norm ratio, so a
mis-scaled output fails. Kernel cases need Xe3P with the kernel; selection
cases run on any XPU and the config parser cases anywhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Route the USPAttention ring branch on XPU to deep_symm.ring_attn when
SGLANG_DIFFUSION_XPU_DEEPSYMM_RING_PRECISION is bf16, fp8, fp8_mixed or
mxfp4 (default off keeps the torch ring). The variable is read once at
import, so the default "off" path makes no torch.compiler.disable call and
a bad value fails at startup.

- fp8_mixed / mxfp4 run only for batch 1 in deep_symm, so B > 1 takes the
  bf16 ring (a Wan request with n=2 would otherwise raise on the first layer).
- deep_symm's shm control block is node-local, so a ring group that spans
  hosts falls back to the torch ring (checked once per group, gathering
  hostnames over the SP group's gloo group, not a oneCCL object gather).
- deep_symm keys its IPC arenas on the K/V shape and never evicts them. The
  first ring call of a request that needs a shape deep_symm does not hold
  frees the arenas that request has not used, before deep_symm allocates, so
  a new resolution, frame count or batch never sits next to the previous
  request's arenas (all ring ranks see the same shapes in the same order, so
  they clear together; the shapes are the resolved ones, e.g. from an I2V
  condition image). Arenas are also freed after the full-shape warmup probe
  (next to IPC_A2A.drop_staging) and in destroy_model_parallel, which also
  drops the id(ring_group)-keyed host check (a recreated group could
  otherwise reuse a stale id on some ranks only and hang the others) and
  does not let a failed peer handshake block group teardown.
- A failed ring pass (e.g. a peer timeout) leaves a poisoned relay in
  deep_symm's cache, so the next ring call clears it first instead of every
  later request with that shape failing.
- Layers with dropout keep the torch ring (deep_symm has none); the package
  is imported at the first eligible layer, so a missing deep-symm fails before
  the first ring collective; batch > 1 on the MX precisions logs once.

The variable is documented in environment_variables.mdx. Tests drive
USPAttention on two ranks (bit-equal to the direct DeepSymm call, torch ring
when off) and cover B=2, cross-host, arena release on a shape change (never
two shape sets held, no rebuild on a repeat), recovery after a failed pass, the worker marking each request before its forward,
and teardown through the real destroy_model_parallel(), also when the
deep_symm clear fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pSymm ring

With ring parallelism on the XPU DeepSymm ring every self-attention layer
takes the ring, so the Sage V3 BF16 block list had no effect and every block
ran MXFP4 (grainy Wan2.2 video). Layers in the blocks listed by
--attention-backend-config sage_v3_bf16_blocks now take the bf16 ring with
--attention-backend fa (the XPU default) as well as sage_attn_3, as in the
setup behind the 2-card Wan2.2 T2V numbers (blocks 0,33,34,38).
sage_v3_kind has no effect under the ring; the docs say so.

The list is read only when the DeepSymm ring is on, so constructing the
default XPU backend still needs no server args. Under fa a layer without a
`blocks.<N>.` index logs a warning and stays on the ring precision
(sage_attn_3 rejects it at construction, as before).

Tests: bf16 ring for listed blocks, and not for a block whose index only
shares a prefix, on both XPU impls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mintlify

mintlify Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 3830391.

Approve preview

@github-actions github-actions Bot added documentation Improvements or additions to documentation diffusion SGLang Diffusion labels Nov 13, 2026


def _bf16_blocks_from_config() -> frozenset[int]:
"""--attention-backend-config sage_v3_bf16_blocks, on any XPU backend."""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Under fa this key now sets DeepSymm ring precision with no Sage kernel in play, and the SGLANG_DIFFUSION_XPU_DEEPSYMM_RING_PRECISION docs row never mentions it, so fa + mxfp4 users won't find it. A backend-neutral name is free while #48059 is unmerged but needs an alias after. Worth renaming?

with sage_attn_3."""
if _xpu_count() < 1:
pytest.skip("needs an XPU")
pytest.importorskip("sgl_kernel", reason="needs sgl-kernel-xpu")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deepsymm_ring_precision calls _import_ring_attn_ipc_forward(), so on an XPU without deep-symm this test and the other ring_precision cases fail with ImportError (11 on a B60) instead of testing routing. Stubbing that import in the ring_precision fixture, as fake_arenas does, keeps them running on any XPU.

<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`sage_attn_3`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)", whiteSpace: "nowrap"}}>`SAGE_ATTN_3`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Requires SageAttention3 installed per upstream instructions.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>CUDA: requires SageAttention3 installed per upstream instructions. Intel XPU (Xe3P): SageAttention V3 from sgl-kernel-xpu (MXFP4) for batch-1, non-causal BF16 self-attention with head dim 128, no GQA and the default softmax scale; other layers, and every step under ring parallelism (<code>--ring-degree</code> &gt; 1), keep XPU flash attention. Without the kernel, or for non-BF16 models, it falls back to XPU flash attention. Configure via <code>--attention-backend-config</code>.</td>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the DeepSymm ring on, ring steps don't keep XPU flash attention; they run DeepSymm at the env precision, as the sage_v3_bf16_blocks row now says. This line, the "Ring steps run XPU flash attention" note in the support table and the Sage info_once in _resolve_sage_v3_kind still say otherwise.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

diffusion SGLang Diffusion documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants