Skip to content

[Perf] [XPU] diffusion: Avoid QKV copies in flash attention - #41944

Merged
mickqian merged 3 commits into
xpu/flux2-catfree-out-projfrom
xpu/flux2-attn-strided-qkv
Oct 24, 2026
Merged

mickqian merged 3 commits into
xpu/flux2-catfree-out-projfrom
xpu/flux2-attn-strided-qkv

Conversation

@AgainstEntropy

@AgainstEntropy AgainstEntropy commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

XPUAttentionImpl.forward called .contiguous() on q, k and v before every
flash-attention call:

q_ = query.contiguous().reshape(bsz * seqlen_q, nheads_q, d)
k_ = key.contiguous().reshape(bsz * seqlen_k, nheads_k, d)
v_ = value.contiguous().reshape(bsz * seqlen_k, nheads_k, d)

In FLUX.2's single-stream blocks q/k/v are head-packed, row-strided slices of
one fused projection output ([tokens, 13824] = q | k | v | mlp), so each
.contiguous() materialised a fresh [tokens, heads, 128] tensor — 3 copies
per block, 144 per DiT forward, 8.8 ms of profiled device time.

The sgl-kernel-xpu flash-attention kernel reads rows through their stride, so
the copies are not needed.

Modifications

multimodal_gen/runtime/layers/attention/backends/xpu_backend.py: new
_as_fa_input helper used for q, k and v. It passes the tensor through
untouched when the [rows, heads, head_dim] view is one the kernel can read
directly — unit last-dim stride, heads packed at head_dim, row stride a
multiple of 8, 16-byte-aligned base — and otherwise copies exactly as before.

This is XPU-only; no other backend is touched.

Accuracy Tests

Kernel-level probe at the FLUX.2 shape ([4608, 12, 128], token stride 13824):
output with strided q/k/v is torch.equal to output with contiguous inputs.
The same probe also confirms the kernel accepts a strided output buffer and
leaves neighbouring columns untouched (not used by this PR — see follow-up).

test_flux2_xpu_norm_fusions.py::test_attention_reads_fused_projection_slices_in_place
(added here): asserts _as_fa_input returns the input's own data_ptr for a
fused-projection slice, and that XPUAttentionImpl.forward gives a
torch.equal result for strided and contiguous inputs. The file is now 6 cases,
all passing on Arc Pro B70.

End to end: no bit-exact fallback warnings; images differ from the parent commit
only at oneCCL run-to-run noise level (see PR 2).

Speed Tests and Profiling

End to end (FLUX.2-dev, 1024x1024, 8 steps, bf16, eager, warm, TP4 on 4x
Arc Pro B70), 2 runs per side:

denoise (8 steps) per DiT forward
parent commit 6.140 / 6.129 s —
this commit 6.077 / 6.095 s −6 ms (−0.8%)

torch.profiler: the 144 aten::copy_ kernels per forward (8.8 ms) are gone.
Flash attention itself costs ~1.2 ms more per forward with strided reads, so the
net saving is smaller than the copies removed.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ✅ Run #39920323973
Latest PR Test (Extra): ❌ Run #39920324009
Latest PR Test (AMD ROCm 10): ❌ Run #39920323980

@AgainstEntropy

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Sep 30, 2026
@AgainstEntropy
AgainstEntropy force-pushed the xpu/flux2-attn-strided-qkv branch from 546344f to 66df650 Compare September 30, 2026 22:56
…t copies

`XPUAttentionImpl.forward` called `.contiguous()` on q, k and v before
flash attention. In FLUX.2's single-stream blocks they are head-packed,
row-strided slices of the fused projection output, so every block paid three
[tokens, heads, 128] copies.

The sgl-kernel-xpu flash attention kernel reads rows through their stride:
with q/k/v as slices of the fused projection (token stride 13824) the output
is bit-identical to the contiguous inputs. Use the input as-is when it views
as [rows, heads, head_dim] with a unit last-dim stride, heads packed at
head_dim, and 16-byte aligned rows; otherwise copy as before.

FLUX.2-dev, 1024x1024, TP4 on 4x Arc Pro B70, eager: the 144 copies per DiT
forward (8.8 ms profiled) are gone; DiT forward -6 ms (-0.8%).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@AgainstEntropy
AgainstEntropy force-pushed the xpu/flux2-attn-strided-qkv branch from 66df650 to ca46349 Compare September 30, 2026 23:06
@mingfeima mingfeima changed the title [diffusion][xpu] Feed row-strided q/k/v to XPU flash attention without copies [Perf] [XPU] diffusion: Avoid QKV copies in flash attention Oct 16, 2026
@mickqian
mickqian merged commit 327c906 into main Oct 24, 2026
214 of 233 checks passed
@mickqian
mickqian deleted the xpu/flux2-attn-strided-qkv branch October 24, 2026 01:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

diffusion SGLang Diffusion run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants