Repository navigation
[diffusion] perf: run MiniMax-H3's QK-norm + RoPE two rows per warp, bit-identical - #43337
Merged
Merged
Conversation
…bit-identical Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mickqian
requested review from
DarkSharpness,
HaiShaw,
HydraQYH,
celve,
niehen6174 and
yuan-luo
as code owners
October 9, 2026 08:47
This was referenced Oct 9, 2026
4 of 5 tasks
This was referenced Oct 10, 2026
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
MiniMax-H3 normalizes and rotates q/k with
fused_qknorm_rope_warp. With head_dim 128 the kernel gives each warp one (token, head) row, so a lane moves only 8 bytes per row. The kernel is issue-bound, not bandwidth-bound: ncu shows 0.69 IPC per scheduler with 48% of stalls on L1TEX, and it reaches 2.5 TB/s on H200. It runs on every block: in place on the sequential Ulysses path, and once per head group and destination on the pipelined path (#43164).Modifications
fused_qknorm_rope_half_warp: the same kernel with two rows per warp. Each row gets 16 lanes, and each lane moves 16 B of q/k and 16 B of the cos/sin cache.Bit-identical by construction. Each lane covers two lanes of the warp kernel. It keeps their two partial sums, accumulated in the same order, and replays that kernel's xor butterfly:
part0 + part1.The per-element norm and RoPE expressions are unchanged.
It is chosen at compile time for head_dim 128, NeoX, the norm rounded before RoPE, a half-width cache, and CUDA. Every other instantiation keeps the warp kernel.
_jit_qknorm_rope_module(half_warp=False)keeps the warp kernel for the new test.Accuracy Tests
test_qknorm_rope_out_of_place.py::test_two_rows_per_warp_match_one_row_per_warp(new) compares the two kernels' bytes. It covers:The file is now registered on
1-gpu-largeas well as4-gpu-b200, so both Hopper and Blackwell check the claim.test_rope.pypasses all 1264 cases on H200, including thetorch.equalchecks against the split QK-norm +sgl_kernelRoPE baseline at H3's shape.Old and new outputs are byte-identical at H3's per-rank shapes (24826 / 18871 / 9436 tokens, in place and per exchange block).
End to end on 4×H200, the MP4 md5 is unchanged in every arm: Ref2VA at Ulysses 2 (four runs), and T2VA at Ulysses 2 and 4.
Speed Tests and Profiling
Per call on H200 (the same GPU for every column), 56 + 56 heads in place, or one exchange block on the pipelined path:
Two rows per warp with scalar cos/sin loads already gave 440.6 / 45.4 µs at 24826 tokens. The 16 B cache loads add the rest.
End to end, client latency, 4×H200, 50 steps, default quality. These runs sit on top of #43164 and #43327, and the arms differ only by this PR:
On the pipelined path the kernel runs once per head group and destination, interleaved with attention. The H200s sit at their 700 W cap during denoise, so part of the freed time comes back as a lower clock.
Checklist
🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ✅ Run #37946082549
Latest PR Test (Extra): ❌ Run #37946081478
Latest PR Test (AMD ROCm 10): ❌ Run #37946082392