Repository navigation
[Perf] [XPU] diffusion: Avoid QKV copies in flash attention - #41944
Merged
Merged
Conversation
AgainstEntropy
requested review from
BBuf,
HaiShaw,
kevin-mii,
mickqian,
niehen6174,
ping1jing2 and
yichiche
as code owners
September 30, 2026 22:31
Collaborator
Author
|
/tag-and-rerun-ci |
AgainstEntropy
force-pushed
the
xpu/flux2-attn-strided-qkv
branch
from
September 30, 2026 22:56
546344f to
66df650
Compare
…t copies `XPUAttentionImpl.forward` called `.contiguous()` on q, k and v before flash attention. In FLUX.2's single-stream blocks they are head-packed, row-strided slices of the fused projection output, so every block paid three [tokens, heads, 128] copies. The sgl-kernel-xpu flash attention kernel reads rows through their stride: with q/k/v as slices of the fused projection (token stride 13824) the output is bit-identical to the contiguous inputs. Use the input as-is when it views as [rows, heads, head_dim] with a unit last-dim stride, heads packed at head_dim, and 16-byte aligned rows; otherwise copy as before. FLUX.2-dev, 1024x1024, TP4 on 4x Arc Pro B70, eager: the 144 copies per DiT forward (8.8 ms profiled) are gone; DiT forward -6 ms (-0.8%). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AgainstEntropy
force-pushed
the
xpu/flux2-attn-strided-qkv
branch
from
September 30, 2026 23:06
66df650 to
ca46349
Compare
mingfeima
approved these changes
Oct 16, 2026
mickqian
approved these changes
Oct 24, 2026
4 of 5 tasks
rahulvijayaraghavan
approved these changes
Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
XPUAttentionImpl.forwardcalled.contiguous()on q, k and v before everyflash-attention call:
In FLUX.2's single-stream blocks q/k/v are head-packed, row-strided slices of
one fused projection output (
[tokens, 13824]=q | k | v | mlp), so each.contiguous()materialised a fresh[tokens, heads, 128]tensor — 3 copiesper block, 144 per DiT forward, 8.8 ms of profiled device time.
The sgl-kernel-xpu flash-attention kernel reads rows through their stride, so
the copies are not needed.
Modifications
multimodal_gen/runtime/layers/attention/backends/xpu_backend.py: new_as_fa_inputhelper used for q, k and v. It passes the tensor throughuntouched when the
[rows, heads, head_dim]view is one the kernel can readdirectly — unit last-dim stride, heads packed at
head_dim, row stride amultiple of 8, 16-byte-aligned base — and otherwise copies exactly as before.
This is XPU-only; no other backend is touched.
Accuracy Tests
Kernel-level probe at the FLUX.2 shape (
[4608, 12, 128], token stride 13824):output with strided q/k/v is
torch.equalto output with contiguous inputs.The same probe also confirms the kernel accepts a strided output buffer and
leaves neighbouring columns untouched (not used by this PR — see follow-up).
test_flux2_xpu_norm_fusions.py::test_attention_reads_fused_projection_slices_in_place(added here): asserts
_as_fa_inputreturns the input's owndata_ptrfor afused-projection slice, and that
XPUAttentionImpl.forwardgives atorch.equalresult for strided and contiguous inputs. The file is now 6 cases,all passing on Arc Pro B70.
End to end: no bit-exact fallback warnings; images differ from the parent commit
only at oneCCL run-to-run noise level (see PR 2).
Speed Tests and Profiling
End to end (FLUX.2-dev, 1024x1024, 8 steps, bf16, eager, warm, TP4 on 4x
Arc Pro B70), 2 runs per side:
torch.profiler: the 144
aten::copy_kernels per forward (8.8 ms) are gone.Flash attention itself costs ~1.2 ms more per forward with strided reads, so the
net saving is smaller than the copies removed.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ✅ Run #39920323973
Latest PR Test (Extra): ❌ Run #39920324009
Latest PR Test (AMD ROCm 10): ❌ Run #39920323980