Skip to content

[diffusion] perf: run MiniMax-H3's QK-norm + RoPE two rows per warp, bit-identical - #43337

Merged
mickqian merged 2 commits into
mainfrom
mick/diffusion-qknorm-rope-half-warp
Oct 10, 2026
Merged

mickqian merged 2 commits into
mainfrom
mick/diffusion-qknorm-rope-half-warp

Conversation

@mickqian

@mickqian mickqian commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

MiniMax-H3 normalizes and rotates q/k with fused_qknorm_rope_warp. With head_dim 128 the kernel gives each warp one (token, head) row, so a lane moves only 8 bytes per row. The kernel is issue-bound, not bandwidth-bound: ncu shows 0.69 IPC per scheduler with 48% of stalls on L1TEX, and it reaches 2.5 TB/s on H200. It runs on every block: in place on the sequential Ulysses path, and once per head group and destination on the pipelined path (#43164).

Modifications

  • fused_qknorm_rope_half_warp: the same kernel with two rows per warp. Each row gets 16 lanes, and each lane moves 16 B of q/k and 16 B of the cos/sin cache.

  • Bit-identical by construction. Each lane covers two lanes of the warp kernel. It keeps their two partial sums, accumulated in the same order, and replays that kernel's xor butterfly:

    • offsets 16..2 become 8..1, applied to each partial;
    • the final xor-1 step becomes part0 + part1.

    The per-element norm and RoPE expressions are unchanged.

  • It is chosen at compile time for head_dim 128, NeoX, the norm rounded before RoPE, a half-width cache, and CUDA. Every other instantiation keeps the warp kernel.

  • _jit_qknorm_rope_module(half_warp=False) keeps the warp kernel for the new test.

Accuracy Tests

  • test_qknorm_rope_out_of_place.py::test_two_rows_per_warp_match_one_row_per_warp (new) compares the two kernels' bytes. It covers:

    • in-place and out-of-place;
    • BF16 and FP16 activations, with BF16, FP16 and FP32 caches;
    • rope_dim 32, 96 and 128;
    • int32 and int64 positions;
    • an odd row count, so the last warp gets one row.

    The file is now registered on 1-gpu-large as well as 4-gpu-b200, so both Hopper and Blackwell check the claim.

  • test_rope.py passes all 1264 cases on H200, including the torch.equal checks against the split QK-norm + sgl_kernel RoPE baseline at H3's shape.

  • Old and new outputs are byte-identical at H3's per-rank shapes (24826 / 18871 / 9436 tokens, in place and per exchange block).

  • End to end on 4×H200, the MP4 md5 is unchanged in every arm: Ref2VA at Ulysses 2 (four runs), and T2VA at Ulysses 2 and 4.

Speed Tests and Profiling

Per call on H200 (the same GPU for every column), 56 + 56 heads in place, or one exchange block on the pipelined path:

tokens path before after
24826 in place, 56 + 56 heads 567.5 µs (2.5 TB/s) 430.5 µs (3.3 TB/s), −24%
24826 exchange block, 4 + 4 heads 51.9 µs 40.5 µs, −22%
18871 in place 430.0 µs 329.1 µs, −23%
18871 exchange block, 4 + 4 heads 40.5 µs 32.5 µs, −20%
9436 in place 218.6 µs 170.0 µs, −22%
9436 exchange block, 2 + 2 heads 15.0 µs 13.0 µs, −13%

Two rows per warp with scalar cos/sin loads already gave 440.6 / 45.4 µs at 24826 tokens. The 16 B cache loads add the rest.

End to end, client latency, 4×H200, 50 steps, default quality. These runs sit on top of #43164 and #43327, and the arms differ only by this PR:

Case before after e2e denoise stage
Ref2VA 5 s, Ulysses 2 (ABBA) 218.71 / 218.97 s 218.47 / 218.70 s −0.12% 212.49 → 212.24 s
T2VA 5 s, Ulysses 2 140.76 s 140.30 s −0.33% 137.01 → 136.55 s
T2VA 5 s, Ulysses 4 71.83 s 71.68 s −0.20% 70.03 → 69.86 s

On the pipelined path the kernel runs once per head group and destination, interleaved with attention. The H200s sit at their 700 W cap during denoise, so part of the freed time comes back as a lower clock.

Checklist

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ✅ Run #37946082549
Latest PR Test (Extra): ❌ Run #37946081478
Latest PR Test (AMD ROCm 10): ❌ Run #37946082392

…bit-identical

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mickqian
mickqian merged commit ff3aa58 into main Oct 10, 2026
177 of 228 checks passed
@mickqian
mickqian deleted the mick/diffusion-qknorm-rope-half-warp branch October 10, 2026 01:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

diffusion SGLang Diffusion jit-kernel run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant