Skip to content

perf(mha): Gemma q/k norm + RoPE + KV write through aiter fused_qknorm_idxrqknorm on gfx1250 - #3405

Open
lirui927 wants to merge 4 commits into
ROCm:mainfrom
lirui927:perf/gemma-qknorm-rope-aiter
Open

lirui927 wants to merge 4 commits into
ROCm:mainfrom
lirui927:perf/gemma-qknorm-rope-aiter

Conversation

@lirui927

@lirui927 lirui927 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Depends on ROCm/aiter#8011 and must land after it, with ATOM's aiter floor raised (docs/environment_variables.md: "Needs aiter with #8011"). ATOM calls aiter directly; there is no capability probe.

What it does

The dense layers of MiniMax-M3 (Gemma q/k norm, NeoX RoPE on 64 of 128 dims) run the q/k norm, RoPE and cache write as separate kernels above ATOM_QK_NORM_ROPE_CACHE_FUSION_MAX_TOKENS (8192) tokens: DualRMSNorm, then the rotary, then the cache write. On the gfx1250 family, this PR makes the norm + RoPE one aiter fused_qknorm_idxrqknorm launch: q goes into a new tensor and k is normed and roped in place in qkv, followed by the same cache write. Nothing else changes: decode and prefills up to the threshold keep main's Triton kernel, and there is no new env variable.

_aiter_gemma_norm_rope takes the route when all of these hold:

  • the GPU is in the gfx1250 family (aiter's is_gfx1250_family, so gfx1250-strict counts), which is where the route was measured;
  • the RoPE is NeoX (rotary_emb.is_neox_style), the rotation the op computes;
  • aiter's fused_qknorm_idxrqknorm_supports(...) accepts the call, with the same arguments the call would pass.

Anything else runs main's unfused path: head_dim 256 (Qwen3-Next with the fusion on, Qwen3.5), any other layout the op refuses, other archs. ATOM does not restate the op's limits.

Dependencies

Benefit

Kernel time per call on one MI455X (gfx1250), MiniMax-M3 dense TP4 shape (16 q heads, 1 KV head, head_dim 128, rotary 64), per-token FP8 SHUFFLE cache, page 128.

  • Method: rope_cache with each route forced, 10 calls captured in one HIP graph and replayed 20 times, the arms interleaved over 7 rounds.
  • Values are medians in µs; the min-to-max spread is within 6%.
tokens 8192 16384
main 47.13 (Triton kernel, ≤ threshold) 91.08 (unfused norm + RoPE + cache write)
this PR 47.13 (unchanged) 30.14 (aiter norm + RoPE + cache write)
for reference: unfused vs aiter route at this size 66.09 vs 19.26

End to end, the gain is real but within noise. M3 has 3 dense layers out of 60, so a 16384-token prefill saves about 3 × 61 µs ≈ 0.18 ms of about 220 ms TTFT (≈ 0.1%).

  • The earlier TP4 run (MiniMax-M3-MXFP4 + EAGLE3, FP8 KV, ptpc_fp8, mono off) measured a lone-prefill TTFT at 16384 tokens of 217.7 ms on main and 220.5 ms with the PR.
  • That run used the PR's previous revision, which also had the decode route. Both arms are within run-to-run spread.

Testing

tests/model_ops/test_gemma_qknorm_rope_aiter_gpu.py (GPU and gfx1250-family only, so CI collects and skips it):

  • rope_cache against the Triton kernel at 8192 tokens (no aiter call) and 8193 tokens (one aiter call), FP8 and bf16 caches, pages of 16 and 128. It compares q, k and the dequantized K/V cache.
  • The route declines what it should: head_dim 128 is taken; head_dim 256 (Qwen3-Next, Qwen3.5) and a GPT-J RoPE fall back.

One MI455X, image rocm/fw-bringup:gfx1250-minimax-m3-20261112, aiter main + #7449 + #8011 (f646b7ea):

Not covered

  • No end-to-end gain is measurable; see Benefit.
  • End-to-end accuracy with a bf16 KV cache. The unit test covers the bf16 cache write against the Triton kernel.
  • Prefills between 1024 and 8192 tokens, where the aiter route would also beat the Triton kernel (by up to 28 µs per dense layer at 8192), stay on the Triton kernel. Moving them would need a second threshold.
  • M3 prefill token graphs are not on main, so they were not exercised.
  • The vLLM plugin's MiniMax-M3 attention and the DSpark drafter keep their path.
  • Archs other than the gfx1250 family.

@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Gate checks are published on every PR so required checks do not stay missing
  • ✅ Expensive jobs run only after the PR is approved, or when a matching ci:* label is present
  • ✅ Pre Checkin must pass before approved heavy jobs continue
  • ✅ Without a matching label, expensive jobs also require a relevant path
  • ✅ A matching ci:* label is an explicit request and can open the gate even when paths would otherwise skip it

Path scope:

  • Native ATOM, vLLM, and SGLang use ignore lists: docs/markdown and unrelated plugin or Mesh paths are skipped
  • ATOMesh uses an include list: Mesh code, ATOMesh scripts/workflows/catalogs, OpenAI entrypoints, and Slurm helpers
  • Relevant path alone does not start expensive jobs; approval still opens the automatic path-scoped gate
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, SGLang, and ATOMesh
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests
ci:mesh Run ATOMesh accuracy/benchmark gates for Mesh-related changes

Without a matching ci:* label, heavy jobs are skipped when the path is not relevant or the PR is not approved.
Add labels via the sidebar or gh pr edit 3405 --add-label <label>

lirui927 pushed a commit that referenced this pull request Nov 16, 2026
…m_idxrqknorm on gfx1250 (#3405)

Port of #3405 at 310014b, by @lirui927,
with the prefill split at 512 tokens (the crossover measured with page 128).
Needs ROCm/aiter#8011.

On top of the PR, for the stack's deferred split-K QKV: the decode insert
passes the partials to fused_qknorm_idxrqknorm when the kernel takes them
(aiter_qknorm_takes_partials) and reduces them first otherwise, as before.
lirui927 and others added 2 commits November 16, 2026 14:59
… decode

For a Gemma q/k norm layer with a plain NeoX RoPE and head_dim 128 (the
MiniMax-M3 dense layers), PagedAttentionImpl.rope_cache hands a decode batch
to aiter fused_qknorm_idxrqknorm on gfx1250 instead of
triton_fused_norm_rope_cache: one insert launch writes q and the normed,
roped K and V into the SHUFFLE cache, per-token FP8 or bf16. k and v are
not written back; the decode backends read only the cache.

It needs ROCm/aiter#8011, which lets a layer without index heads insert
into an FP8 cache (earlier versions assert index operands there) and
brings the faster gfx1250 kernel; ATOM reads that off
aiter.FUSED_QKNORM_IDXRQKNORM_SUPPORTS_INDEXLESS_FP8. Without it nothing
changes. The op aborts on arguments it refuses, so
_takes_aiter_gemma_norm_rope checks what the KV pool does not guarantee.

tests/model_ops/test_gemma_qknorm_rope_aiter_gpu.py runs rope_cache against
the Triton kernel (q, the dequantized K/V cache; pages of 16 and 128; CUDA
graph padding rows) and checks the route each batch takes.

Signed-off-by: lirui927 <ruili@amd.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Above ATOM_QK_NORM_ROPE_CACHE_FUSION_MAX_TOKENS, rope_cache runs the Gemma
q/k norm, RoPE and cache write as separate kernels. Where the decode insert
is taken, the norm and RoPE are now one aiter fused_qknorm_idxrqknorm launch
(q into a new tensor, k in place in qkv) instead of DualRMSNorm and the
rotary, ahead of the same cache write. Prefill attention reads k, which the
insert does not write back, so prefill keeps the cache write separate.

Signed-off-by: lirui927 <ruili@amd.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes needed..

end-to-end accuracy with bf16 kv cache? all numbers are fp8

ci Offline inference smoke test failed — not sure yet (low): The captured log is 0 bytes, so nothing can be attributed. The same job passed on the 3 most recent main runs, so main is not the obvious cause, but neither the failure nor a PR link can be shown.

job log is 0 bytes (no content captured)
next: Re-run the job and read its log; if it reproduces, check whether the run reached the gfx1250 path this PR changes. https://github.com/ROCm/ATOM/actions/runs/44001641181/job/133767035815

ci ATOM Test Gate failed — main is already broken the same way (low): The log is 0 bytes. The same gate also failed on 2 of the 3 most recent main runs, which points at the gate/CI rather than this PR, but the failure itself is unreadable.

job log is 0 bytes (no content captured)
next: Treat as pre-existing until a log shows a PR-specific test; no action needed from the author beyond a re-run. https://github.com/ROCm/ATOM/actions/runs/44001641181/job/133768716650

@lirui927
lirui927 requested a review from valarLip November 17, 2026 02:55

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (4)

  1. Wait for ROCm/aiter#8011 to merge, then call the op directly and delete the capability probe. AITER_QKNORM_TAKES_DENSE_LAYERS = getattr(aiter, "FUSED_QKNORM_IDXRQKNORM_SUPPORTS_INDEXLESS_FP8", False) (atom/model_ops/attention_mha.py:46) probes aiter's version, which ATOM must not do. #3162 is a case where a probe misread aiter and M3 failed to start on gfx1250. #8011 is still open, and ATOM CI installs aiter main, so today this route is dead code in CI and the new test always skips. Once #8011 merges, raise ATOM's aiter floor.
  2. Delete the hand copy of the op's argument rules in _takes_aiter_gemma_norm_rope (attention_mha.py:481-519) and get the check from aiter. The docstring says the reason itself: "The op aborts on arguments it refuses instead of raising, so whatever the KV pool does not already guarantee is checked here." The function then restates the op's limits: head_dim 128, rotary_dim % 8, a contiguous [q|k|v], int64 1-D positions, a 5-D SHUFFLE cache, bf16 or per-token FP8, and get_gfx() == "gfx1250", which misses aiter's is_gfx1250_family/gfx1250-strict. Ask #8011 to raise a Python error on unsupported arguments, or to expose a supports(...) query, and keep only the measured "use it on gfx1250" choice in ATOM.
  3. Drop the getattr/hasattr defenses on the rotary module (getattr(rope, "is_neox_style", False), getattr(rope, "mrope_section", None), hasattr(rope, "cos_cache"), lines 512-514). Read the attributes the RoPE class defines, or gate on the RoPE type.
  4. Justify the change end to end, or narrow it to the case that wins. By the PR's own numbers, TPOT moves by -0.4% to +0.7% and 16k TTFT goes 217.7 -> 220.5 ms, all within noise. For that, the PR adds a third route through rope_cache and a new aiter dependency. The one clear kernel win is prefill above 8192 tokens (91 -> 30 µs per dense layer). Either show a measurable end-to-end gain, or keep only that route. Also answer valar-cop's question: accuracy with a bf16 KV cache. Every reported number uses FP8 KV, yet the PR enables the bf16-cache route too.

Reviewed at 58c6715. Depends on unmerged ROCm/aiter#8011. Conflicts textually with #2988 (rope_cache); not stacked on #2949/#3116. Light CI green after re-run; heavy CI not run.

@valar-cop

valar-cop commented Nov 17, 2026 •

Copy link
Copy Markdown

lirui927 and others added 2 commits November 17, 2026 06:42
…easured choice

Drop the FUSED_QKNORM_IDXRQKNORM_SUPPORTS_INDEXLESS_FP8 probe, the copy of
fused_qknorm_idxrqknorm's argument limits (head_dim, rotary_dim, qkv,
positions, cache layout and dtype) and the getattr/hasattr checks on the
rotary module: with ROCm/aiter#8011 the op raises on anything it refuses
instead of aborting. ATOM keeps only its own choices: the route is opt-in
per model (`aiter_gemma_norm_rope`, set by the MiniMax-M3 dense layers it
was measured on), on the gfx1250 family (aiter's is_gfx1250_family, so
gfx1250-strict counts), into its SHUFFLE cache layout. Needs aiter with
#8011.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…her it applies

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lirui927

Copy link
Copy Markdown
Contributor Author

Thanks. All four points are addressed in 75ead45:

  1. The probe is gone, and [HIP] [Perf] fused_qknorm_idxrqknorm: faster on gfx1250, split-K partials as input aiter#8011 no longer adds the flag. This PR lands after #8011 with the aiter floor raised; docs/environment_variables.md says "Needs aiter with #8011".

  2. The hand copy of the op's rules is deleted. #8011 now:

    • raises RuntimeError on any argument it refuses instead of aborting;
    • adds fused_qknorm_idxrqknorm_supports(...), which runs the op's own checks with check_only and does not launch.

    ATOM takes the route on the gfx1250 family (aiter's is_gfx1250_family, so gfx1250-strict counts), for a NeoX RoPE, where supports(...) is True. Otherwise it runs main's path: head_dim 256 (Qwen3-Next, Qwen3.5) falls back, and a test covers that.

  3. The getattr/hasattr checks on the rotary module are gone. The route reads rotary_emb.is_neox_style and rotary_dim, which the RoPE classes define.

  4. Narrowed to the route that clearly wins. The decode insert is removed, and decode keeps main's Triton kernel. Only prefills above ATOM_QK_NORM_ROPE_CACHE_FUSION_MAX_TOKENS (8192) run the norm + RoPE as one aiter launch: 91 → 30 µs per dense layer at 16384 tokens. End to end that is about 0.18 ms of about 220 ms TTFT at 16k, within noise, and the description says so. bf16-KV accuracy end to end is not run; the unit test covers the bf16 cache write against the Triton kernel. The narrowed route doesn't use #8011's index-less fp8 insert fix (it runs in non-insert mode).

Validation on MI455X with aiter main + #7449 + #8011 (f646b7ea):

@lirui927
lirui927 requested a review from valarLip November 17, 2026 07:48

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (2)

  1. Wait for ROCm/aiter#8011 to merge (I am reviewing it now), then rebase and run heavy CI. aiter.fused_qknorm_idxrqknorm_supports (atom/model_ops/attention_mha.py:477) exists only on #8011 (open, head f646b7ea); aiter main has no such symbol. MiniMax-M3 passes its Gemma q_norm to the attention by default, so on the gfx1250 family every M3 prefill above 8192 tokens would raise AttributeError with today's aiter. ATOM CI installs aiter main and the new test is gfx1250-only, so CI cannot catch this. Do not add a hasattr fallback; land after #8011 instead.
  2. Stop paying for the aiter route on layers it declines (minor, can go in the same push). On gfx1250, Qwen3-Next and Qwen3.5 (head_dim 256, NeoX) with the fusion on reach _aiter_gemma_norm_rope on every long prefill, layer by layer. Each call allocates a q that is thrown away (:461), registers a _minimax_m3_cos_sin_cache buffer on their rotary (:470), and calls supports(), which returns False every time. The layer's shapes, dtypes and cache layout never change, so ask supports() once per layer (first long prefill) and remember the answer. Also: the op takes one eps and only q_norm.variance_epsilon is passed (:475). Assert it equals k_norm's, or note that assumption. In the env-var doc row and the test docstring, add that the route also needs the ASM cache layout (_use_asm_cache_layout, :291).

Optional: a CPU test of the routing (monkeypatch is_gfx1250_family, fake supports) would give CI some coverage of _aiter_gemma_norm_rope, which today it never runs.


Re-reviewed at 75ead45. Already fixed: the aiter capability probe is gone, the hand copy of the op's limits is replaced by aiter's fused_qknorm_idxrqknorm_supports, the getattr/hasattr defenses on the rotary are gone (is_gfx1250_family replaces get_gfx() == "gfx1250"), the change is narrowed to the long-prefill route with kernel numbers (91 -> 30 µs per dense layer at 16k), the bf16-cache write is covered against the Triton kernel, and the env doc is updated. Light CI green; heavy CI not run. Still depends on unmerged ROCm/aiter#8011.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants