Skip to content

[Triton/Gluon] [QSA] FP8 KV and FP8 / FP4 index caches, stacked-head decode scorer - #7008

Merged
valarLip merged 18 commits into
mainfrom
lingzha/qsa-fp8-kv-index-stacked
Nov 9, 2026
Merged

valarLip merged 18 commits into
mainfrom
lingzha/qsa-fp8-kv-index-stacked

Conversation

@ZLkanyo009

@ZLkanyo009 ZLkanyo009 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

#6469 (Triton) and #6515 (FlyDSL) ported the QSA kernels from ROCm/ATOM before ROCm/ATOM#2652 landed. ROCm/ATOM#2652 adds three things to those operators that aiter does not have yet:

  • FP8 K/V caches, with one FP32 scale per (token, KV head).
  • FP8 and MXFP4 compressed index-key caches, with their scales.
  • A stacked-head decode scorer for MTP verify.

This PR adds them to the files that #6469 created, so that ATOM can call aiter for the QSA kernels instead of keeping its own copy. cc @mengfei-jiang @zovonoir (the follow-up ATOM PR) @valarLip

Technical Details

The source is atom/model_ops/qwen4_exp/ops/qsa.py at ROCm/ATOM#2652 (squash 32a355f0f). Every new parameter defaults to the current behavior. Unquantized BF16 / FP16 / FP32 caches work as on main.

Cache writes. qsa_store_quantized_rows(cache, scale, slots, values, second_cache=None, second_scale=None, second_values=None) writes FP8 or MXFP4 rows and skips negative slots. The writer lives in aiter because it defines the format that the readers decode, so the tests can check the two sides against each other.

  • FP8: E4M3 with scale = row amax / FP8 max, so values never saturate. K and V can be written in one launch. The quantization is fused_fp8_quant._fp8_quant_op.
  • MXFP4: packed E2M1 pairs and one E8M0 exponent per 32 dims, through quant._mxfp4_quant_op. For that, the quantizer gets two constexprs, ROUNDING_MODE and MAX_SCALE_E8M0:
    • ROUNDING_MODE=1 rounds ties away from zero. ATOM writes its cache bytes that way.
    • MAX_SCALE_E8M0=240 caps the exponent. The readers' FP16-bitcast decode needs E8M0 <= 240.
    • With the defaults, existing callers compile to the same instructions.
    • Why FP4 keeps ATOM's bytes and FP8 does not: E2M1 ties are common (1.1% of BF16 elements; 77% of 128-dim rows have one), so the rounding mode changes most rows. For FP8 a tie between two E4M3 codes is about 2e-4 of the bytes, and either code has the same error, so the FP8 writer reuses _fp8_quant_op as it is.
  • qsa_cache_row_layout(fmt, head_dim) describes a stored row (value dim and dtype, scale shape and dtype). It is exported from aiter.ops.qsa with FP8_DTYPE, FP8_MAX and FP4_GROUP.

Index scoring. qsa_paged_mqa_logits and qsa_select_paged_tokens take compressed_k_scale. The CUDA-core, MFMA and stacked scorers take a KEY_FMT constexpr:

  • FP8 multiplies the score by the row scale.
  • MXFP4 decodes the packed rows into a [rows, head_dim] key tile (_load_fp4_keys), so all formats share one scoring path.

FlyDSL stays BF16-only, so FP8 / FP4 prefill uses the Triton MFMA scorer, as ATOM does.

Stacked decode scorer. stacked_decode=True scores calls with at most 256 rows with _qsa_paged_mqa_logits_stacked_kernel. It stacks rows x index heads into the M of one MFMA, so each request's keys are read once per verify step instead of once per row. In ATOM it is on by default (env var); here it is an argument and off by default, so existing callers keep their kernels. It is a separate kernel rather than a mode of _qsa_paged_mqa_logits_mfma_kernel, because a mode would rewrite the body of main's prefill kernel; the MFMA kernel only gains the KEY_FMT branches.

FP8 sparse GQA. qsa_sparse_paged_gqa takes k_scale / v_scale. The key scale multiplies the score column, and the value scale multiplies the probability before PV.

Tiles. The stacked scorer and FP8 GQA tiles were measured in ATOM. They are in configs/{gfx942,gfx950,gfx1250}/triton/attention/qsa/DEFAULT.json; the gfx942 and gfx1250 files are copies of gfx950. The file is optional: on an arch without it, stacked_decode calls use the other scorers and FP8 GQA uses _kernel_config. The gfx942 copy is an exception to the gfx950 → gfx1250 seed list in configs/AGENTS.md: it is kept so that MI300X CI runs the stacked scorer and FP8 GQA, and it should be replaced once it is measured on gfx942.

Input checks. The following raise ValueError instead of being ignored or read out of bounds:

  • a scale plane that does not match the cache in shape or dtype;
  • a partial second_* set;
  • a cache or scale plane sliced from a larger page pool;
  • an FP4 head dim that is not a power of two >= 32;
  • a stray v_scale;
  • a 1-byte float cache without its scale plane (FP8 Q with an FP8 index cache, or FP8 Q / K / V in GQA). On main such a call is accepted when the dtypes match.

Changed files: aiter/ops/qsa.py (exports and pass-through), aiter/ops/triton/attention/qsa.py (wrappers), aiter/ops/triton/_triton_kernels/attention/qsa.py (kernels), aiter/ops/triton/_triton_kernels/quant/quant.py (the two constexprs), the three tile JSON files, op_tests/triton_tests/attention/test_qsa.py and op_tests/op_benchmarks/triton/bench_qsa.py.

Test Plan

On MI355X (gfx950), container rocm/atom-dev:nightly_202610221638 (triton 3.8.0, torch 2.10.0+rocm7.2.4), one GPU:

  1. pytest op_tests/triton_tests/attention/test_qsa.py and op_tests/triton_tests/quant/test_quant_mxfp4.py.
  2. Parity with ATOM main 9df8e849a: the same inputs through ATOM's ops and through aiter, compared bit for bit.
  3. Unchanged paths, bit for bit: BF16 QSA outputs, main vs branch, and dynamic_mxfp4_quant / _mxfp4_quant_op with default arguments (outputs and AMDGCN instructions).
  4. Performance: ATOM vs aiter in one process, and bench_qsa.py on main vs the branch.
  5. GSM8K (full 1319, strict match) with ATOM main calling this branch's QSA ops, next to ATOM's own ops. Model Qwen3.8-Flash-Next-Quark-MXFP4, TP1, MTP-3, FP8 KV cache with an FP8 or FP4 index cache.
  6. For each refactor during review: old vs new head in one process, comparing outputs bit for bit, the AMDGCN instructions of every compiled variant, and the time.

Test Result

  • test_qsa.py: 110 passed (main: 11). test_quant_mxfp4.py: 79 passed, 54 skipped, as on main.
  • Parity with ATOM: 174 / 197 cases bitwise identical, covering writers, logits, top-k selection and GQA, BF16 / FP8 / FP4, page size 4 / 16, decode, MTP-3 verify and prefill shapes.
    • Three are FP8 writer cases. They have the same scales, and about 2e-4 of the bytes of BF16 rows are one E4M3 code apart, because _fp8_quant_op multiplies by 1 / scale and ATOM by FP8_MAX / amax.
    • The other 20 are FP4 logits, at most 1.4e-6 apart: aiter sums the decoded keys in head-dim order, ATOM sums the even and odd dims separately. Every FP4 top-k selection case picks the same tokens, and every BF16 / FP8 reader case is identical.
    • FP4 writer extras: bitwise identical for every BF16 amax above 6 x 2^-103 and every FP16 amax.
  • Unchanged paths: 62 / 62 BF16 outputs are identical to main. The 21 MXFP4 default-path outputs are identical, and their 19 kernels have the same AMDGCN instructions.

GSM8K (stderr about 0.004):

KV / index cache ATOM's ops this branch's ops
FP8 / FP8 0.9780 0.9742
FP8 / FP4 0.9780 0.9795

Performance (us, median of 9 alternating rounds, same inputs, page size 16):

case ATOM aiter
writer FP8 K+V, 36 rows 9.85 9.67
writer FP8 K+V, 16384 rows 42.94 41.14
writer FP4 index, 16384 rows 12.54 12.62
logits FP8, 9x4 verify, stacked 17.63 17.64
logits FP4, 9x4 verify, stacked 16.42 16.23
logits FP4, 9x4 verify, not stacked 41.17 19.58
logits FP4, 1x8192 prefill, 64k ctx 867.26 618.22
GQA FP8, 36 rows / 9 requests 49.61 52.38
GQA FP8, 16384-row prefill 7607.26 7611.25

Over all 93 cases the median aiter / ATOM ratio is 0.998 (0.476–1.059). FP4 scoring without stacked_decode is faster than ATOM (0.48x at 9x4, 0.70–0.73x for the 64k prefill), because the decoded key tile is scored like a BF16 / FP8 tile instead of as two nibble planes. The other cases with more than 100 us of GPU time are 0.973–1.014. With bench_qsa.py --op logits --index-dtype fp4|fp8 [--stacked-decode] (eager do_bench, launch cost included), stacked_decode makes decode scoring 1.18–1.30x faster at 9x4, 32x3 and 128x1 for FP4 and 1.26–1.34x for FP8, and 0.90–1.10x at 16–64 single-token rows, where both take 13–17 us of mostly launch cost. BF16 GPU time is the same as main (10 cases, median 0.997, 0.987–1.013). The eager host cost is 1–2 us higher per scorer or GQA call and 3.1–3.6 us higher for qsa_select_paged_tokens (gfx950, back to back against main, 1x1 to 128x1 rows, 4096 context), because of the format check, the tile lookup and the extra kernel arguments; the scorer and GQA GPU time under CUDA graphs is the same as main. ATOM's wrapper does the same work and runs decode in CUDA graphs.

Not changed

  • main's prefill MFMA kernel only gains the KEY_FMT branches and the columns < num_columns bound on page_ok (the out-of-bounds table read valar-cop asked to fix in both kernels). The stacked scorer stays a separate kernel, because merging it would rewrite main's kernel body.
  • The CUDA-core scorer keeps main's NUM_HEADS <= 4 limit.
  • FlyDSL stays BF16-only. FP16 / FP32 caches with allow_flydsl behave as on main, and FP8 / MXFP4 keys and stacked_decode use Triton only.
  • _SCORING_BLOCK_N, _SCORING_COLUMN_PROGRAMS, the MFMA 16 / 128 tile and _kernel_config stay in Python: main's BF16 path reads them on arches without the QSA JSON too. Moving them into the JSON would be a separate PR.
  • The gfx942 tiles are gfx950 copies, not measured on gfx942.
  • The MFMA path (more than 256 query rows) keeps main's gate, which does not check that head_dim is a power of two; FP8 keys with such a head dim (e.g. 96) inherit it. Reproduced on gfx950: a call with more than 256 rows and head_dim 96 fails to compile (arange's range must be a power of 2), for BF16 on main too. A fix would change main's BF16 dispatch, so it is left for a follow-up.

Submission Checklist

PR Self-Review

Reviewed revision

dab71f0

Behavior and intent

The QSA ops gain FP8 K/V caches, FP8 / MXFP4 index-key caches (writers and scorers) and a stacked-head decode scorer, ported from ROCm/ATOM#2652, so ATOM can call aiter instead of keeping its own copy. Every new parameter defaults to the existing behavior, and unquantized BF16 / FP16 / FP32 caches work as on main. _mxfp4_quant_op gets ROUNDING_MODE and MAX_SCALE_E8M0 so the FP4 writer can reuse it and still write ATOM's cache bytes; the defaults leave existing callers unchanged. Inputs that would be ignored or read out of bounds raise ValueError. The tile JSON is optional per arch.

Validation and unverified scope

On MI355X (gfx950), one GPU: test_qsa.py 110 passed, test_quant_mxfp4.py 79 passed, 54 skipped. Parity with ATOM main is 174 / 197 bitwise: the three FP8 writer cases are at most one E4M3 code apart with the same scales (each an exact tie between two codes with the same error), and the 20 FP4 logits cases are at most 1.4e-6 apart because of the summation order; every top-k selection case picks the same tokens. BF16 outputs and the MXFP4 default path are bitwise identical to main, and the MXFP4 default path has the same instructions. GSM8K with ATOM calling these ops matches ATOM's own ops (table above). black 26.10.1 and ruff 0.16.0 pass on the changed files. Each refactor during review was checked old vs new head in one process: outputs bit for bit, the AMDGCN instructions of every compiled variant, and GPU time.

The last revisions (b2fb752..dab71f0) share one scoring path across key formats: _load_fp4_keys returns the whole decoded key tile, so the even / odd Q loads and the second scoring loop are gone, and FP8 GQA handles both scales in one KV_FP8 branch (PR diff +1793 −93 → +1708 −61). Against b2fb752dc, 265 / 350 A/B outputs are bitwise identical and the other 85 are FP4 scorer outputs (summation order, covered by the tests above); the BF16 / FP8 scorer and GQA variants have the same instructions, the 16 writer variants differ only because the unused num_rows argument is gone (their outputs are bitwise identical), and the median GPU time ratio is 0.997. FP4 GPU time drops to 0.26–0.29x in the CUDA-core scorer, 0.71–0.74x in the MFMA scorer and 0.88–0.93x in the stacked scorer. Against main, the BF16 kernels differ only in kernel-argument offsets (the new scale strides) and the register allocation that follows; BF16 GPU time is the same as main (median 0.997). The GPU results above were measured on a567922; dab71f0 only changes bench_qsa.py (layout fields by name, same output, rerun on gfx950) and a comment in the wrapper, so they were not rerun. The branch merges main 47b4e0869 to resolve the __all__ conflict with #6576.

Not verified: on gfx942 only CI's Triton tests ran. The gfx942 tiles are gfx950 copies and were not measured there, and parity, perf and GSM8K were not run on gfx942 (no gfx942 machine available to me). The GSM8K run uses a local ATOM copy that calls these ops; switching ATOM itself is the follow-up ATOM PR.

PR-specific question

Before this PR the new cases in test_qsa.py fail because the APIs they call do not exist (qsa_store_quantized_rows, qsa_cache_row_layout, compressed_k_scale, k_scale / v_scale, stacked_decode). The scoring cases compare against dequantized keys, so they catch a reader that decodes FP8 / MXFP4 rows wrongly, and the stacked cases catch a wrong row / head mapping. The FP4 writer cases fail if ties round to nearest even or the exponent is not capped at 240. The reject cases fail if a partial second plane, a mismatched scale plane or a pool slice is accepted, and the missing-config cases fail if an arch without the JSON raises. test_quant_mxfp4.py covers only dynamic_mxfp4_quant with default arguments, and no other test covers QSA.

Changed op tests

  • op_tests/triton_tests/attention/test_qsa.py: the only QSA test file. The new cases cover:

    • the FP8 / MXFP4 writers, including their input checks;
    • FP8 / MXFP4 scoring in the stacked, CUDA-core and MFMA scorers against dequantized keys (head dim 32 to 256, 1 to 4 index heads);
    • top-k over quantized keys;
    • unquantized FP16 / FP32 caches;
    • FP8 GQA against dequantized BF16 GQA;
    • the stacked scorer's dispatch and max_columns;
    • an arch without the tile JSON.

    None of this is tested elsewhere.

  • op_tests/op_benchmarks/triton/bench_qsa.py: the QSA benchmark, not a test. It gains --op logits, --index-dtype, --kv-dtype, --stacked-decode and the MTP-3 verify shape, and counts each format's bytes.

…decode scorer

Bring the QSA operator changes of ROCm/ATOM#2652 into the Triton QSA
kernels that #6469 / #6515 ported from ATOM before that PR landed.

- qsa_store_quantized_rows: FP8 rows with one FP32 scale per (token,
  head) = row amax / FP8 max (K and V in one launch), and MXFP4 rows
  (packed E2M1 pairs, one E8M0 exponent per 32 dims, rounded up so a
  group never clips). qsa_cache_row_layout describes the stored row.
- qsa_sparse_paged_gqa(k_scale=, v_scale=): reads E4M3 K/V tiles,
  multiplies the key scale into its score column and folds the value
  scale into the probability before PV; FP8 has its own tile config.
- qsa_paged_mqa_logits / qsa_select_paged_tokens(compressed_k_scale=):
  FP8 (row scale) and MXFP4 compressed index keys in the CUDA-core,
  prefill MFMA and stacked scorers. FlyDSL stays BF16-only.
- stacked_decode=True: decode-sized calls (<= 256 rows) score with an
  MFMA kernel that stacks 4 rows x 4 index heads into M = 16, so a
  request's keys are read once per verify instead of once per row.
  Off by default, so existing BF16 callers keep their kernels.

The BF16 path and the existing signatures are unchanged; new arguments
are trailing keywords with defaults.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ZLkanyo009
ZLkanyo009 requested a review from a team October 27, 2026 09:44
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
ci:triton-355 Run the full Triton test suite on MI35X, not only the tests the change affects
ci:windows Windows JIT op smoke test on the gfx1201 ROCm runner
ci:multi-gpu Aiter multi-GPU tests on the 8-GPU runner
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 7008 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved bounds checks, slot-layout handling, and offline FP8 initialization can cause invalid memory access, incorrect writes, or import failures.

4 open issues
What changed in this PR

This PR extends aiter’s QSA operators with quantized caches and an optional stacked-head decode scorer, preparing ATOM to use aiter instead of its local kernels.

Changes:

  • Add FP8 K/V and FP8/MXFP4 index-cache writers and readers.
  • Add optional stacked-head decode scoring while preserving the default BF16 path.
  • Extend correctness tests and benchmarks for quantized formats and stacked scoring.
File Description
op_tests/​triton_tests/​attention/​test_qsa.py Tests quantized writes, scoring, selection, GQA, and stacked dispatch.
op_tests/​op_benchmarks/​triton/​bench_qsa.py Adds format options, logits timing, and stacked-decode workloads.
aiter/​ops/​triton/​attention/​qsa.py Adds cache-layout helpers, writer launches, and quantized dispatch.
aiter/​ops/​triton/​_triton_kernels/​attention/​qsa.py Implements quantized writers, readers, and stacked scoring.
aiter/​ops/​qsa.py Exposes cache APIs and forwards new options while keeping FlyDSL BF16-only.

Review effort: Balanced

Comment thread aiter/ops/triton/attention/qsa.py Outdated
# Quantized cache rows: FP8 with one FP32 scale per row, or MXFP4 with one
# E8M0 exponent for each FP4_GROUP dims (see the kernel module).
FP8_DTYPE = dtypes.fp8
FP8_MAX = float(torch.finfo(FP8_DTYPE).max)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 381f018. FP8_MAX is now None when dtypes.fp8 is not a floating-point dtype (the uint8 placeholder that aiter uses without a visible GPU), so both QSA modules import on CPU. In that case _cache_format does not treat uint8 as FP8, so uint8 caches stay FP4, and qsa_cache_row_layout("fp8", ...) raises a clear ValueError.

Test: test_qsa_imports_without_a_gpu runs a subprocess with CUDA_VISIBLE_DEVICES="" and HIP_VISIBLE_DEVICES="". It checks that torch.cuda.is_available() is False, imports aiter.ops.qsa and aiter.ops.triton.attention.qsa, and checks FP8_MAX is None and the FP4 layout. On 830d0bb the test fails with the torch.finfo TypeError.

rows, heads, head_dim = values.shape
if rows == 0:
return
if slots.shape[0] < rows or heads != cache.shape[2]:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 381f018. qsa_store_quantized_rows now calls _check_vector("slots", slots) before any launch, so slots must be 1-D, int32 / int64 and contiguous for both the FP8 and FP4 writers. The length check (slots.shape[0] >= rows) is unchanged.

Test: test_qsa_writer_rejects_bad_slots_and_planes passes backing[::2], float slots and 2-D slots to both writers and expects ValueError. On 830d0bb the strided slots were accepted.

Comment thread aiter/ops/triton/attention/qsa.py Outdated
Comment on lines +95 to +96
second_cache.stride() != cache.stride()
or second_scale.stride() != scale.stride()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 381f018. The second FP8 plane must now match the first in cache shape and dtype and in scale shape and dtype, as well as in strides, and second_scale / second_values must be given. Before the second plane is checked, the first scale plane must have the shape [pages, page, heads(, groups)] of its cache, because the same stride problem can let a smaller first scale plane through.

Test: test_qsa_writer_rejects_bad_slots_and_planes builds 1-page planes with the same strides as a 2-page cache (the test asserts the strides are equal). It expects ValueError for a smaller first scale plane, a smaller second cache and a smaller second scale plane. A matching K/V pair still writes both planes identically.

…iter checks

- Scorers: the MFMA and stacked kernels mask page-table loads with
  columns < num_columns as well as the rounded table width, so a narrow
  table (for example [1, 3] at page size 4) is not read past its end.
- Without a visible GPU, aiter.dtypes.fp8 is a uint8 placeholder. FP8_MAX
  is None there instead of a module-level torch.finfo error, so both QSA
  modules import on CPU; FP8 layouts raise until a GPU FP8 dtype exists.
- qsa_store_quantized_rows rejects non-contiguous, non-integer or
  multi-dim slots (the writers read slots[row] without a stride), a scale
  plane whose shape does not match the cache, and a second K/V plane whose
  cache or scale shape or dtype differs from the first.
- Tests for each case: a [1, 3] page table through the CUDA-core, MFMA and
  stacked scorers in BF16 / FP8 / FP4, bad slots and planes for both
  writers, and an import in a subprocess with no visible GPU.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ZLkanyo009

Copy link
Copy Markdown
Contributor Author

@mengfei-jiang @zovonoir A note for the follow-up that switches ATOM to call these kernels from aiter: what to do with the QSA unit tests ATOM got from ROCm/ATOM#2652.

ATOM main still runs its own atom/model_ops/qwen4_exp/ops/qsa.py, so those tests should stay until the switch. Once ATOM calls aiter, they split into two groups.

Kernel tests, covered by this PR's new cases. These can be removed in the same commit that removes ATOM's local kernels:

  • tests/model_ops/test_qsa_quantized_cache.py:
    • test_fp8_writer_never_saturates
    • test_fp4_writer_is_exact_on_the_grid_and_close_off_it
    • test_quantized_index_scores_match_dequantized_keys
    • test_fp8_sparse_gqa_matches_dequantized_kv
  • tests/model_ops/test_qsa_stacked_scorer.py (both tests).

ATOM-side tests, which should stay (adjust the imports if format resolution moves to qsa_cache_row_layout):

  • test_cache_formats_follow_the_kv_dtype, test_cache_formats_reject_unserved: which cache format ATOM picks for --kv_cache_dtype / --index-cache-dtype, and which combinations it rejects.
  • test_bf16_fields_are_unchanged, test_quantized_fields_and_bf16_draft_rows, test_fp8_pool_fits_more_blocks: ATOM's KV pool field layout, the MTP draft rows and the block size in bytes.
  • The fixture changes in tests/test_qwen4_exp_mtp.py.

It would also help to keep one small end-to-end check in ATOM: the QSA attention path through aiter gives the same output as before the switch, for BF16 and FP8 KV. That way there is no window where the local kernels are still used but no longer tested, or the reverse.

@k50112113 k50112113 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed write-up and the parity numbers. One request:

_qsa_store_rows_fp4_kernel writes the same format as aiter's existing MXFP4 quantizer (OCP MXFP4: packed E2M1 + one E8M0 per 32). Its scale rule, ceil(log2(amax / 6)), is also what _mxfp4_quant_op(SCALING_MODE=1) in aiter/ops/triton/_triton_kernels/quant/quant.py already computes. The only real differences are:

E2M1 tie rounding: _e2m1_code rounds ties away from zero (a >= midpoint). _mxfp4_pack_bits rounds to nearest even. These give different bytes at 0.25, 1.25, 2.5 and 5.0. See the table below:

Scaled value quant.py (round to nearest even, same as the gfx950 v_cvt_scalef32_pk_fp4 instruction) sage (ties toward lower magnitude) PR (ties toward higher magnitude)
0.25 0 0 0.5
0.75 1.0 0.5 1.0
1.25 1.0 1.0 1.5
1.75 2.0 1.5 2.0
2.5 2.0 2.0 3.0
3.5 4.0 3.0 4.0
5.0 4.0 4.0 6.0

E8M0 range: The QSA writer clamps the exponent to [-127, 113] (E8M0 <= 240) for the FP16-bitcast decode. The ceil helper can go up to 254.
All-zero group scale byte. The 1e-30 floor gives E8M0 = 25, while _mxfp4_ceil_scale_from_amax gives 1. Both decode to zero.

Rather than adding a second MXFP4 quantizer with its own _e2m1_code (aiter already has another _e2m1_code in sage_attention_quant.py with different tie rounding), could you extend _mxfp4_quant_op to cover these and call it from QSA? Something like:

New tie-rounding option on _mxfp4_quant_op / _mxfp4_pack_bits: e.g. ROUNDING_MODE: tl.constexpr = 0 (0 = nearest even, the current behavior; 1 = ties away from zero). The default must stay unchanged for existing callers. The gfx950 v_cvt_scalef32_pk_fp4_f32 path only does nearest-even, so mode 1 should static_assert on USE_ASM or fall back to the bit path. Both the normal and denormal branches of _mxfp4_pack_bits need the change, since the denormal branch relies on the FPU's round-to-nearest-even.
Scale cap for ceil mode: e.g. MAX_SCALE_E8M0: tl.constexpr = 254, which QSA sets to 240.
Zero-group byte: either add an option that matches ATOM's byte 25, or confirm that ATOM's parity check can accept a different scale byte for all-zero groups, since the decoded values are identical. I'd prefer the second if the ATOM side allows it.
QSA writer: load the row as a contiguous [1, HEAD_DIM] tile and call _mxfp4_quant_op(..., MXFP4_QUANT_BLOCK_SIZE=FP4_GROUP, SCALING_MODE=1, ROUNDING_MODE=1, MAX_SCALE_E8M0=240) instead of _e2m1_code plus the inline exponent math.
After the change, please rerun the ATOM parity suite for the FP4 writer and FP4 scoring cases. _mxfp4_ceil_scale_from_amax computes the exponent from the FP32 bits, while the current code uses tl.log2, so it's worth confirming they still match bit for bit. Please also test the existing dynamic_mxfp4_quant tests showing that the default path (ROUNDING_MODE=0) is unchanged, however, you don't need to add a test script for dynamic_mxfp4_quant.

Separately: was ATOM's ties-away rounding a deliberate choice? If not, switching both sides to round-to-nearest-even would let QSA use SCALING_MODE=1 as it is, with no new rounding mode. That needs a matching change in ATOM, so I'm fine handling it as a follow-up.

Address review: drop the QSA writer's own E2M1 encoder and inline E8M0
math, and extend aiter's MXFP4 quantizer to cover what QSA needs.

- _mxfp4_quant_op / _mxfp4_pack_op / _mxfp4_pack_bits gain
  ROUNDING_MODE (0 = ties to nearest even, the existing behavior and
  default; 1 = ties away from zero). Both the normal and the denormal
  branch of the bit path implement mode 1; USE_ASM with mode 1 is a
  static_assert, since v_cvt_scalef32_pk_fp4_f32 only rounds to even.
- _mxfp4_ceil_scale_from_amax / _mxfp4_quant_op gain MAX_SCALE_E8M0
  (default 254 = uncapped, so inf / NaN amax keeps its NaN scale);
  only SCALING_MODE=1 accepts a cap.
- _qsa_store_rows_fp4_kernel loads the row as a [1, HEAD_DIM] tile and
  calls _mxfp4_quant_op(SCALING_MODE=1, ROUNDING_MODE=1,
  MAX_SCALE_E8M0=240): the cache bytes of ATOM's QSA writer, with E8M0
  kept within the readers' FP16 decode.
- Tests: E2M1 ties at 0.25 ... 5.0 times the scale, and the E8M0 cap.

Default arguments compile to the same instructions as before.

Co-authored-by: Cursor <cursoragent@cursor.com>
@aiter-bot

aiter-bot commented Oct 29, 2026 •

Copy link
Copy Markdown

READY FOR HUMAN REVIEW

PR Self-Review

Revision dab71f035f8a · mode shadow

Author self-review accepted. Readiness report:

Author answers

  • PASS — current-SHA answers are present and structurally specific
Show the author's self-review answers

Behavior and intent

The QSA ops gain FP8 K/V caches, FP8 / MXFP4 index-key caches (writers and scorers) and a stacked-head decode scorer, ported from ROCm/ATOM#2652, so ATOM can call aiter instead of keeping its own copy. Every new parameter defaults to the existing behavior, and unquantized BF16 / FP16 / FP32 caches work as on main. `_mxfp4_quant_op` gets `ROUNDING_MODE` and `MAX_SCALE_E8M0` so the FP4 writer can reuse it and still write ATOM's cache bytes; the defaults leave existing callers unchanged. Inputs that would be ignored or read out of bounds raise `ValueError`. The tile JSON is optional per arch.

Validation and unverified scope

On MI355X (gfx950), one GPU: `test_qsa.py` 110 passed, `test_quant_mxfp4.py` 79 passed, 54 skipped. Parity with ATOM main is 174 / 197 bitwise: the three FP8 writer cases are at most one E4M3 code apart with the same scales (each an exact tie between two codes with the same error), and the 20 FP4 logits cases are at most 1.4e-6 apart because of the summation order; every top-k selection case picks the same tokens. BF16 outputs and the MXFP4 default path are bitwise identical to main, and the MXFP4 default path has the same instructions. GSM8K with ATOM calling these ops matches ATOM's own ops (table above). black 26.10.1 and ruff 0.16.0 pass on the changed files. Each refactor during review was checked old vs new head in one process: outputs bit for bit, the AMDGCN instructions of every compiled variant, and GPU time.

The last revisions (b2fb752..dab71f0) share one scoring path across key formats: `_load_fp4_keys` returns the whole decoded key tile, so the even / odd Q loads and the second scoring loop are gone, and FP8 GQA handles both scales in one `KV_FP8` branch (PR diff +1793 −93 → +1708 −61). Against `b2fb752dc`, 265 / 350 A/B outputs are bitwise identical and the other 85 are FP4 scorer outputs (summation order, covered by the tests above); the BF16 / FP8 scorer and GQA variants have the same instructions, the 16 writer variants differ only because the unused `num_rows` argument is gone (their outputs are bitwise identical), and the median GPU time ratio is 0.997. FP4 GPU time drops to 0.26–0.29x in the CUDA-core scorer, 0.71–0.74x in the MFMA scorer and 0.88–0.93x in the stacked scorer. Against main, the BF16 kernels differ only in kernel-argument offsets (the new scale strides) and the register allocation that follows; BF16 GPU time is the same as main (median 0.997). The GPU results above were measured on `a567922`; `dab71f0` only changes `bench_qsa.py` (layout fields by name, same output, rerun on gfx950) and a comment in the wrapper, so they were not rerun. The branch merges main `47b4e0869` to resolve the `__all__` conflict with #6576.

Not verified: on gfx942 only CI's Triton tests ran. The gfx942 tiles are gfx950 copies and were not measured there, and parity, perf and GSM8K were not run on gfx942 (no gfx942 machine available to me). The GSM8K run uses a local ATOM copy that calls these ops; switching ATOM itself is the follow-up ATOM PR.

PR-specific question

Before this PR the new cases in `test_qsa.py` fail because the APIs they call do not exist (`qsa_store_quantized_rows`, `qsa_cache_row_layout`, `compressed_k_scale`, `k_scale` / `v_scale`, `stacked_decode`). The scoring cases compare against dequantized keys, so they catch a reader that decodes FP8 / MXFP4 rows wrongly, and the stacked cases catch a wrong row / head mapping. The FP4 writer cases fail if ties round to nearest even or the exponent is not capped at 240. The reject cases fail if a partial second plane, a mismatched scale plane or a pool slice is accepted, and the missing-config cases fail if an arch without the JSON raises. `test_quant_mxfp4.py` covers only `dynamic_mxfp4_quant` with default arguments, and no other test covers QSA.

Changed op tests

- `op_tests/triton_tests/attention/test_qsa.py`: the only QSA test file. The new cases cover:
  - the FP8 / MXFP4 writers, including their input checks;
  - FP8 / MXFP4 scoring in the stacked, CUDA-core and MFMA scorers against dequantized keys (head dim 32 to 256, 1 to 4 index heads);
  - top-k over quantized keys;
  - unquantized FP16 / FP32 caches;
  - FP8 GQA against dequantized BF16 GQA;
  - the stacked scorer's dispatch and `max_columns`;
  - an arch without the tile JSON.

  None of this is tested elsewhere.
- `op_tests/op_benchmarks/triton/bench_qsa.py`: the QSA benchmark, not a test. It gains `--op logits`, `--index-dtype`, `--kv-dtype`, `--stacked-decode` and the MTP-3 verify shape, and counts each format's bytes.

PR state

  • PASS — pull request is marked ready for review

Copilot findings

  • PASS — no active Copilot finding lacks a substantive author reply
  • ADVISORY — Copilot did not review the current head SHA
  • ADVISORY — an author-replied Copilot finding is still unresolved; confirm the disposition during human review

Code and op-test hygiene

  • PASS — no committed artifact or conflict marker detected

Required CI

  • PASS — Black: passed
  • PASS — Ruff: passed
  • PASS — Repository dependency: passed
  • PASS — Aiter Test Gate: passed

Shadow mode reports what would block but keeps the status successful. Set AITER_PR_SELF_REVIEW_MODE=enforce only after reviewing real false positives.

@ZLkanyo009

Copy link
Copy Markdown
Contributor Author

@k50112113 Done in 9be3a42: the FP4 writer now uses _mxfp4_quant_op, and _e2m1_code is gone. cc @valarLip

_mxfp4_quant_op changes (quant.py)

Two new constexprs, added after USE_ASM because test_quant_mxfp4.py passes SCALING_MODE and USE_ASM by position:

  • ROUNDING_MODE=0: 0 is round to nearest even, as before. 1 rounds E2M1 ties away from zero. Both branches of _mxfp4_pack_bits handle it:
    • Denormal branch (scaled |x| < 1, codes 0 / 0.5 / 1): mode 0 keeps the + 2^22 add, where the FPU rounds to nearest even. Mode 1 compares with the midpoints: (x >= 0.25) + (x >= 0.75).
    • Normal branch: mode 1 adds (1 << 21) - 1 + 1 instead of + mant_odd. That is exactly half a step, so a tie always carries.
    • USE_ASM=True (v_cvt_scalef32_pk_fp4_f32) only rounds to nearest even. Instead of falling back silently, _mxfp4_pack_op has tl.static_assert(not USE_ASM or ROUNDING_MODE == 0).
  • MAX_SCALE_E8M0=254 caps the E8M0 of the ceil scale. It needs SCALING_MODE=1, and mode 0 static-asserts that it is 254. No finite amax gives more than 253, so the default emits no instruction, and an inf / NaN amax still gives 255.

The defaults change nothing. On the previous head (381f018) vs this head, 21 outputs are bitwise identical:

  • dynamic_mxfp4_quant with the Triton backend, BF16 / FP16 / FP32 x 5 shapes.
  • The default backend for BF16.
  • _mxfp4_quant_op called directly with SCALING_MODE 0 / 1 x USE_ASM.

The 19 compiled kernels also have the same AMDGCN instructions: 16,336 lines, after removing the debug lines. test_quant_mxfp4.py gives 79 passed, 54 skipped on both heads.

QSA writer

It loads a contiguous [1, HEAD_DIM] tile and calls _mxfp4_quant_op(x, HEAD_DIM, 1, GROUP, SCALING_MODE=1, ROUNDING_MODE=1, MAX_SCALE_E8M0=240). The readers are unchanged. The cap of 240 is what their FP16-bitcast decode needs.

All-zero groups

The bytes differ, and I accepted that without adding an option:

  • aiter's ceil scale gives E8M0 1 for an all-zero group. ATOM's writer gives 25, because it floors amax at 1e-30.
  • Both decode to zeros.
  • In ATOM the cache bytes are only decoded, by the three scorers. KV offload keys hash token ids and the config, not page content, so nothing compares or deduplicates pages across implementations. ATOM's tests compare decoded values.

The same holds for -0.0, which is code 8 here and 0 in ATOM. For these inputs the parity check compares decoded values. Logits and top-k from caches that contain them are bitwise identical to ATOM's.

Parity with ATOM main (9df8e849a)

  • The full writer / scorer / select / GQA set is still 197 / 197 bitwise identical.
  • The extra FP4 writer cases are also bitwise identical:
    • every BF16 amax above 6 x 2^-103 (29,149 values)
    • every FP16 amax (31,743 values)
    • all E2M1 ties at random signs and scales
    • FP32 random inputs
    • FP32 amax 0 / 1 / 2 ulp below 6 x 2^k
    • logits / top-k from the aiter writer's caches (42 cases)
  • Only two cases still differ, both in ATOM's encoding:
    • Nonzero groups with amax <= 6 x 2^-103 (about 5.9e-31), because of the 1e-30 floor.
    • FP32 amax 1 or 2 ulp above 6 x 2^k. ATOM's tl.log2 returns exactly k there, so ATOM picks the smaller exponent and clips that value to 6 x 2^k. aiter takes the exponent from the bits. ATOM writes BF16 index keys, where this cannot happen.

Tests

test_qsa.py gives 52 passed. Two cases are new:

  • test_qsa_fp4_writer_rounds_ties_away_from_zero covers every tie, at random signs and scales 2^-20 .. 2^19.
  • test_qsa_fp4_writer_caps_the_scale_for_the_fp16_decode: values above 6 x 2^113 saturate, and a small group next to them keeps its values.

As a mutation check, the tie test fails when the writer passes ROUNDING_MODE=0, and the cap test fails with the default MAX_SCALE_E8M0.

Perf

Same MI355X GPU, 381f018 vs 9be3a42, torch.profiler kernel time, median of 7 alternating rounds:

  • FP4 writer: ratio 0.975 (0.959 .. 0.991 over 7 shapes, 36 to 65,536 rows). The contiguous row load is slightly faster.
  • dynamic_mxfp4_quant: ratio 1.001 over 24 cases. Its instructions are unchanged.

Your question: was ATOM's ties-away rounding deliberate?

No.

So nothing on the ATOM side requires ties away from zero. This PR keeps it only so that the cache bytes stay identical when ATOM switches to aiter. Moving ATOM to nearest even is an ATOM-side follow-up. After that, the QSA writer passes ROUNDING_MODE=0, and mode 1 can be removed if nothing else uses it.

@ZLkanyo009

Copy link
Copy Markdown
Contributor Author

@k50112113 Your review is addressed in 9be3a422 (details in this reply): the FP4 index writer now calls _mxfp4_quant_op, which gains ROUNDING_MODE and MAX_SCALE_E8M0 options, with defaults that leave every existing caller unchanged. Could you take another look when you have a moment?

A note on the one red check, Triton Tests (MI35X) / Shard 5: it is a known failure on main, not from this PR.

  • All 8 failures are test_batched_gemm_afp4wfp4.py::test_batched_gemm_afp4_wfp4[*-2-256-3584-2112] (4 layouts × 2 dtypes), with NaN in the output. This PR does not touch that kernel. test_qsa.py passes 52/52 on both MI35X and MI300X.
  • main's own Triton Test has failed on the same 8 cases on every completed run since 10-27, for example run 40539173165 on 457b11518 and run 40965021247 on 96175f600.
  • The cause and fix are in [Triton/Gluon] Mask OOB e8m0 scales on the ragged-K path of batched_gemm_afp4wfp4 #7047: with K not a multiple of BLOCK_SIZE_K (here 2112), the last K block loads e8m0 scales past the end without a mask, and a 0xFF byte there is the e8m0 NaN that tl.dot_scaled propagates. Its description names exactly this shape.

We will re-run this PR's Triton tests once #7047 is in.

@ZLkanyo009

Copy link
Copy Markdown
Contributor Author

@valar-cop review this pr

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes needed

any perf number on gfx942? and gsm8k for the fp8/fp4 caches?

ci Triton Tests (MI35X) / Shard 5 failed — main is already broken the same way (medium): The same shard fails on main itself in all 7 recent runs, and the failing cases the author cites (test_batched_gemm_afp4wfp4[*-2-256-3584-2112], NaN output) are in a kernel this PR does not touch; the cause is tracked in #7047 (unmasked e8m0 scale load in the last K block when K is not a multiple of BLOCK_SIZE_K).

FAILED job Triton Tests (MI35X) / Shard 5 (Triton Test), runner linux-aiter-mi35x-1-fvzc6-runner-54mhl, log .valar/ci/job_123774296329.log
same job on main: ['2026-10-30T09:11 failure', '2026-10-30T08:12 failure', '2026-10-30T07:24 failure', '2026-10-30T06:16 failure', '2026-10-30T04:36 failure', '2026-10-30T03:48 failure', '2026-10-30T03:06 failure']
next: Rerun the Triton tests after #7047 lands; nothing to fix in this PR. The log artifact was empty, so re-check the shard log on the rerun to confirm the same 8 afp4wfp4 cases. https://github.com/ROCm/aiter/actions/runs/40966863934/job/123774296329

ci Triton Test Results failed — main is already broken the same way (high): This is the gate job that aggregates the shards, so it fails only because Shard 5 failed; on main the same job also failed in the same 7 runs.

FAILED job Triton Test Results (Triton Test), runner GitHub Actions 1009738695, log .valar/ci/job_123798544428.log
same job on main: ['2026-10-30T09:11 failure', '2026-10-30T08:12 failure', '2026-10-30T07:24 failure', '2026-10-30T06:16 failure', '2026-10-30T04:36 failure', '2026-10-30T03:48 failure', '2026-10-30T03:06 failure']
next: No action; it turns green when Shard 5 does (#7047). https://github.com/ROCm/aiter/actions/runs/40966863934/job/123798544428

Comment thread aiter/ops/triton/attention/qsa.py Outdated
if fmt == _FMT_FP8:
if cache.shape[3] != head_dim:
raise ValueError("values and FP8 cache disagree on head_dim")
second = second_cache is not None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

partial second plane silently skips V? please reject all-or-none here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop I followed your suggestion on aiter/ops/triton/attention/qsa.py:100: qsa_store_quantized_rows now raises ValueError("pass second_cache, second_scale and second_values together") unless the three second_* tensors are all given or all None. Before, a partial set skipped V without an error. The FP4 writer still rejects any second tensor.

test_qsa_writer_rejects_bad_slots_and_planes now passes the four partial sets (cache only, cache + scale, scale + values, values only) to both writers, expects ValueError and checks that nothing was written. It fails on 9be3a42.

(commit d26518567; test_qsa.py 53 passed on MI355X)

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, all three or none is what i meant, and the fp4 path still rejects any second tensor. checked d265185, the scale-plane check now goes through qsa_cache_row_layout so a short fp4 group axis can't slip through either. nothing else from me, lgtm, just wait for the triton tests rerun after #7047.

Comment thread aiter/ops/triton/attention/qsa.py Outdated
if key_fmt == _FMT_BF16 and q.dtype != compressed_k_cache.dtype:
raise ValueError("q and compressed_k_cache must have the same dtype")
if key_fmt != _FMT_BF16 and (
compressed_k_scale.shape[:3] != compressed_k_cache.shape[:3]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check the full plane, not just shape[:3]? fp4 tail mismatch reads past the scale row

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop I followed your suggestion on aiter/ops/triton/attention/qsa.py:280: the check now compares the whole scale plane with the cache's row layout. compressed_k_scale must have the shape (*compressed_k_cache.shape[:3], *tail) and the dtype from qsa_cache_row_layout (FP8: no tail, float32; MXFP4: head_dim // 32 uint8 exponents), plus the unit-stride checks as before. A short FP4 group axis or a wrong dtype now raises ValueError instead of reading past the scale row. The check runs before the stacked / CUDA-core / MFMA dispatch, so it covers every scorer and qsa_select_paged_tokens.

While there I tightened the same kind of check in qsa_sparse_paged_gqa: v_scale must have the shape of k_scale, and a v_scale without an FP8 cache and k_scale is rejected (it used to be ignored).

New test test_qsa_scorers_reject_scale_planes_that_do_not_match_the_cache: after a valid call, FP8 with an extra scale dim or float64 scales, FP4 with a short group axis or int8 exponents, and GQA with a bad v_scale all raise. It fails on 9be3a42.

(commit d26518567; test_qsa.py 53 passed on MI355X)

cc @valarLip



@triton.jit(repr=_qsa_paged_mqa_logits_fp4_repr)
def _qsa_paged_mqa_logits_fp4_kernel(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why duplicate the whole body? add KEY_FMT constexpr like the mfma kernel

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop I followed your suggestion on aiter/ops/triton/_triton_kernels/attention/qsa.py:219: _qsa_paged_mqa_logits_fp4_kernel is gone. _qsa_paged_mqa_logits_kernel takes KEY_FMT (0 BF16, 1 FP8, 2 MXFP4) and GROUP, like the MFMA kernel; only the Q loads and the key decode branch on it, and the wrapper has one launch for all three formats. The kernel file is 83 lines shorter and the wrapper 26.

The merge does not change the generated code (MI355X, 9be3a42 vs d265185 in one process, same GPU):

  • 216 scorer cases (BF16 / FP8 / FP4 x head dim 64 / 128 / 256 x 1-4 heads x page size 4 / 16 x 3 shapes): outputs bitwise identical.
  • The same 216 compiled variants have the same AMDGCN instructions (142,815 lines, debug lines removed).
  • GPU kernel time over 24 shapes (torch.profiler, median of 9 alternating rounds): median ratio 1.000 (0.973 .. 1.003).
  • FP8 / FP4 writer and scorer parity with ATOM main 9df8e849a: still 197 / 197 bitwise identical.

The only cost I measured is on the host: an eager MXFP4 call is about 2 us slower to launch (30.6 -> 32.9 us), because the merged kernel takes four more arguments (two dim strides, BLOCK_D, KEY_FMT). Its GPU time is longer than that, so do_bench with launch overhead stays at 0.999 .. 1.001 for MXFP4, and decode runs in CUDA graphs in ATOM.

(commit d26518567; test_qsa.py 53 passed)

cc @valarLip

torch.manual_seed(0)
rows = 2048
magnitude = torch.logspace(-4, 4, rows, device="cuda")[:, None, None]
key = (torch.randn(rows, 2, 256, device="cuda") * magnitude).bfloat16()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pass dtype explicitly to randn, like _make_paged_scenario does

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop I followed your suggestion on op_tests/triton_tests/attention/test_qsa.py:245: every torch.randn in the file now passes dtype=torch.float32, as _make_paged_scenario does. The casts after them are unchanged, so the test inputs are the same values as before.

(commit d26518567; test_qsa.py 53 passed on MI355X)

cc @valarLip

…e planes

- qsa_store_quantized_rows: second_cache, second_scale and second_values go
  together or not at all; a partial set used to be ignored without an error.
- qsa_paged_mqa_logits: compressed_k_scale must have the full shape and dtype
  of the cache's scale plane (qsa_cache_row_layout), not only its first three
  dims. A short FP4 group axis used to pass and the kernel read past each
  scale row.
- qsa_sparse_paged_gqa: v_scale must have the shape of k_scale, and a v_scale
  without an FP8 cache and k_scale is rejected.
- The CUDA-core scorer is one kernel with KEY_FMT (0 BF16, 1 FP8, 2 MXFP4),
  like the MFMA kernels, instead of a separate FP4 kernel with a copied body.
  Every variant compiles to the same instructions as before.
- test_qsa.py: tests for the rejected inputs; randn gets an explicit dtype.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ZLkanyo009

Copy link
Copy Markdown
Contributor Author

@valar-cop thanks for the review. I pushed d26518567 on top of 9be3a42 (no rebase). The four inline comments are answered in their threads:

  • Partial second plane: qsa_store_quantized_rows takes the three second_* tensors all or none, otherwise ValueError.
  • Scale plane: qsa_paged_mqa_logits checks the full shape and dtype of compressed_k_scale against qsa_cache_row_layout, so a short FP4 group axis is rejected. I also check v_scale in qsa_sparse_paged_gqa (same shape as k_scale, only with an FP8 cache and k_scale).
  • One CUDA-core scorer with KEY_FMT, like the MFMA kernel. Outputs and the compiled AMDGCN instructions are the same as before for all 216 variants I compiled, and the GPU kernel time ratio is 1.000 (median over 24 shapes).
  • torch.randn gets an explicit dtype.

New tests for the rejected inputs fail on 9be3a42 and pass now; test_qsa.py 53 passed on MI355X.

Perf on gfx942: I have no access to a gfx942 machine, so I have no perf numbers there. All numbers in this PR are from MI355X (gfx950). On gfx942 only CI's Triton Test runs this PR (test_qsa.py passed on the MI300X shard for 9be3a42).

GSM8K for the FP8 / FP4 caches: ATOM does not call aiter's QSA ops yet; it still uses its own in-tree copy from ROCm/ATOM#2652. So I give three pieces of evidence:

  1. Same results as ATOM's kernels: same inputs through ATOM main 9df8e849a and through this branch, 197 / 197 cases bitwise identical (FP8 K/V and FP8 / FP4 index writers, logits, top-k and GQA), rerun on d26518567.

  2. GSM8K today with ATOM calling this branch's ops, next to the same ATOM with its own ops: MI355X, one GPU, same node, ATOM main 9df8e849a, Qwen3.8-Flash-Next-Quark-MXFP4, TP1, KV block 16, real MTP-3 verify, full 1319, 5-shot chat, strict match:

    KV / index cache ATOM in-tree ops this branch's ops
    FP8 / FP8 0.9780 0.9773
    FP8 / FP4 0.9780 0.9795

    stderr is about 0.004. Per question, 4 vs 5 (FP8 index) and 7 vs 5 (FP4 index) questions are correct on only one side.

    Method: I made a local copy of ATOM main, with no ATOM branch changed. Its ops/qsa.py rebinds qsa_store_quantized_rows, qsa_select_paged_tokens, qsa_paged_mqa_logits, qsa_sparse_paged_gqa and qsa_expand_block_indices to this branch's code, with stacked_decode=envs.ATOM_QSA_STACKED_DECODE_SCORER. The server log confirms that the writer, the selector and GQA ran from that code. The rest of aiter is the CI wheel that ATOM main uses.

  3. Earlier ATOM validation of feat(qwen4-exp): QSA FP8 KV and FP8 / FP4 index caches, with a stacked-head decode scorer ATOM#2652 (ATOM's ops, MI355X, full GSM8K): BF16 0.9765, FP8 KV + BF16 index 0.9765, FP8 KV + FP8 index 0.9773, FP8 KV + FP4 index 0.9780.

I updated the PR description (Test Result and Self-Review) with these numbers. The Triton Test Shard 5 failure (test_batched_gemm_afp4wfp4) also fails on main, is tracked in #7047, and is not from this PR.

cc @valarLip

@valar-cop review this pr

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not ready yet.. tune tables belong in json configs, not python

any gfx942 perf number? gsm8k? ci still queued on head sha

ci green.

Comment thread aiter/ops/triton/attention/qsa.py Outdated

# (BMT, BN, num_warps, column programs) of the stacked decode scorer for each
# key format, measured at 36 rows over 9 requests (MTP 3 verify).
_STACKED_DECODE_TILES = {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tuned values should live in configs//.../DEFAULT.json, not inline dicts here..

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop I followed your suggestion on aiter/ops/triton/attention/qsa.py:221: the stacked decode tiles and the FP8 GQA tiles are now in configs/<arch>/triton/attention/qsa/DEFAULT.json (stacked_decode.{bf16,fp8,fp4} and gqa_fp8.{decode,prefill}). _tuned_tile(kernel, case) reads them with resolve_config_dir() and load_config_json(), behind functools.lru_cache like the other attention loaders.

  • gfx950: the measured values. gfx942: the same values, which QSA used there before (I have no gfx942 to tune on). gfx1250: seeded byte-identical from gfx950, as configs/AGENTS.md allows.
  • Only the stacked and FP8 GQA paths read the file, so BF16 calls on main's path do not need it on any arch.
  • The tiles match the old Python tables entry by entry on all three archs. Outputs of 350 writer / scorer / GQA cases are bitwise identical to d265185, and all compiled variants have the same instructions.
  • Host time per call is unchanged: median ratio 1.003 for the stacked and GQA decode calls that read the tiles. A first version without the lru_cache added 1-2 us per call (do_bench up to 8% on launch-bound decode shapes), so I added the cache.

I left _SCORING_BLOCK_N, _SCORING_COLUMN_PROGRAMS, the MFMA tile and _kernel_config in Python. They come from #6469, and main's BF16 path reads them on every arch, including archs that have no QSA JSON. Moving them would change that path, so I would do it in a separate PR.

(commit 4811ba315; test_qsa.py 53 passed on MI355X)

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm for the json move, and thanks for keeping _kernel_config for main's bf16 path.

one thing: gqa_fp8 on gfx942 is not what 942 used before, the fp8 gqa path is new in this pr, so those are gfx950 tiles copied over.. any 942 number, or is it ci only?

_tuned_tile's docstring is longer than the function, keep it to one line and put the measurements in the pr. and please add the ci:triton-300x label, gfx942 configs changed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop You are right, and my earlier line "the same values, which QSA used there before" was wrong. The stacked scorer and the FP8 GQA path are both new in this PR, so gfx942 had no earlier values. Its stacked_decode and gqa_fp8 entries are the gfx950 tiles copied over. I have no gfx942 machine, so there are no gfx942 numbers: on gfx942 this is CI only. On d26518567, test_qsa.py passed on the MI300X shard with these tiles.

I kept the gfx942 file rather than dropping it:

  • Without it, gfx942 would need a missing-table branch (load_config_json(..., required=False) plus defaults in Python).
  • FP8 GQA would then fall back to _kernel_config. _kernel_config was set up for BF16 and is not measured for FP8 on any arch.
  • With the copied tiles, gfx942 runs the same configuration that MI300X CI has passed.

If you would rather have gfx942 on a default path, I can drop the file in a follow-up. The PR description now says plainly that the gfx942 tiles are gfx950 copies and were not measured on gfx942.

_tuned_tile's docstring is one line in 4feba25c4. The measured shapes are in the PR description, under "Notes for review: Tiles".

About ci:triton-300x: the PR already has it (added on 10-28). I don't change labels myself; the maintainers decide them.

cc @valarLip

top = 6.0 * 2.0**113
values = torch.zeros(2, 1, 128, device="cuda")
values[0, 0, :32] = top
values[0, 0, 32:35] = torch.tensor([4 * top, -4 * top, top / 2])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build these on gpu with explicit dtype, no cpu tensor then copy..

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop I followed your suggestion on op_tests/triton_tests/attention/test_qsa.py:320: the two torch.tensor calls in the cap test are now built on the GPU with dtype=torch.float32, and so is the torch.zeros above them. No CPU tensor is copied anymore. I also gave the other float factories in the file (tensor / zeros / ones / empty / full) an explicit dtype. The values are the same, since float32 was the default.

(commit 4811ba315; test_qsa.py 53 passed on MI355X)

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. one left: torch.logspace(-4, 4, rows, device="cuda") at op_tests/triton_tests/attention/test_qsa.py:246 still has no dtype. the rest looks good, thanks.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Fixed in 4feba25c4: torch.logspace(-4, 4, rows, dtype=torch.float32, device="cuda"). No other float factory in test_qsa.py is left without a dtype.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logspace fixed, checked the rest of the file. one left: sign = torch.randint(...) * 2.0 - 1.0 at op_tests/triton_tests/attention/test_qsa.py:311 still gets its float dtype from the default, add dtype=torch.float32 there? rest lgtm.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Not changed. sign is not built by a float factory: torch.randint(...) returns int64, and * 2.0 - 1.0 promotes it to the default float dtype (float32). Its values are exactly +1 / -1, so the dtype cannot change the result.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah right, randint is int64 so the python float just lands it on the default float dtype, and the values are exact +-1. fine as is, no change needed.



@triton.jit
def _e8m0_value_hi(e):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should already have _mxfp4_e8m0_to_fp32 in quant.py, reuse it instead of duplicate..

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop I followed your suggestion on aiter/ops/triton/_triton_kernels/attention/qsa.py:73: _e8m0_value_hi is removed, and the three MXFP4 readers call _mxfp4_e8m0_to_fp32 from quant.py.

The readers decode E2M1 with an FP16 bitcast that leaves a factor of 2^-14, so they pass e + 14: _mxfp4_e8m0_to_fp32(exponent + 14). Before reusing it, I checked it against the old code:

  • Bits: it gives the same bits as the removed ((e + 14) << 23) bitcast for all 256 exponents. The decode only places bits and does no rounding. e + 14 >= 14, so its e == 0 branch is never taken, and the compiler drops it.
  • e = 255 / NaN: called on a raw exponent, _mxfp4_e8m0_to_fp32(255) gives +inf, not NaN. With + 14, exponents above 240 give the same out-of-range values as the old helper did. The writer caps the exponent at 240, so no cache holds them.
  • Outputs and code: the stacked, CUDA-core and MFMA MXFP4 scores are bitwise identical to d265185 (108 cases each), and the compiled instructions are unchanged.

(commit 4811ba315; test_qsa.py 53 passed on MI355X)

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, nice work. the e <= 240 note in the _e2m1_value_lo docstring (aiter/ops/triton/_triton_kernels/attention/qsa.py:70) is the part to keep, so the + 14 can go once the writer stops capping.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Thanks. The e <= 240 sentence stays in the _e2m1_value_lo docstring.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fine, keep it. it's the only place that spells out the 240 limit, and the writer comment at aiter/ops/triton/_triton_kernels/attention/qsa.py:176 already points here, so no duplicate.


_qsa_store_rows_fp8_repr = make_kernel_repr(
"_qsa_store_rows_fp8",
["HEAD_DIM", "BLOCK_D", "HAS_SECOND"],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FP8_MAX_C differs per arch, add it to the repr key..

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop I followed your suggestion on aiter/ops/triton/_triton_kernels/attention/qsa.py:20: FP8_MAX_C is in the FP8 writer's repr key, so the kernel is named e.g. ..._FP8_MAX_C_448_... on gfx950 and ..._FP8_MAX_C_240_... on gfx942 (FNUZ). The FP8 writer is the only kernel that takes FP8_MAX_C. Its compiled instructions are unchanged (12 variants, kernel name removed).

(commit 4811ba315; test_qsa.py 53 passed on MI355X)

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, that's what i meant. name reads fine (..._FP8_MAX_C_448_...), and both values come from dtypes.fp8 so nothing is hardcoded. nothing else from me on this point.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Thanks, nothing changes here in 4feba25c4.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, nothing to add then. FP8_MAX_C is in the fp8 writer's repr key on head (aiter/ops/triton/_triton_kernels/attention/qsa.py:23) and comes from torch.finfo(FP8_DTYPE).max (aiter/ops/triton/attention/qsa.py:29), so the kernel name shows the per-arch max.


@pytest.mark.parametrize("fmt", ["bf16", "fp8", "fp4"])
@pytest.mark.parametrize("rows", [4, 300])
def test_qsa_scorers_stay_inside_a_narrow_page_table(fmt, rows):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test passes on the unfixed kernel too, make the table really width 3 or drop the claim..

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop You are right. I reverted the page-table fix from 381f018, so the MFMA and stacked kernels mask the table load only with the rounded width again. The narrow-table test still passed in all 6 variants, both with the old [[2, 0, 3, 1]][:, :3] view and with a real contiguous [1, 3] table.

The test cannot catch that read. An entry read past the table is used only after the 0 <= page < num_pages check, and the scores it produces land in columns >= num_columns, which the kernels never store. So the read has no visible effect. Only a memory fault could show it, and a 4-byte overread inside the allocator's block does not fault.

So I made the table really width 3 and dropped the claim: the comment now says the test checks that the 12 columns of a [1, 3] table, rounded up to width 4, score right. The fix itself stays.

(commit 4811ba315; unfixed kernels: 6 passed, fixed: 6 passed; test_qsa.py 53 passed on MI355X)

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, fair. keep the columns < num_columns mask anyway, an oob read is still oob even if it does not fault; and add one line above it in aiter/ops/triton/_triton_kernels/attention/qsa.py:469 and :644 saying why, since no test covers it now.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Kept the columns < num_columns mask. In 4feba25c4 there is one line above page_ok in the MFMA and stacked kernels:

# The width is rounded up; columns < num_columns keeps table reads in bounds.

The change is comment-only: all 348 compiled variants have the same AMDGCN instructions as 4811ba315.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, reads right now — safe_page only clamps to the rounded-up PAGE_TABLE_WIDTH, so the real-width columns < num_columns term is what keeps the read in bounds. comment matches the code, fine by me.

Comment thread aiter/ops/triton/attention/qsa.py Outdated
num_warps=warps,
)
return logits, visible_groups
# Decode-sized calls without stacking use the CUDA-core kernel.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment sits above the mfma branch but describes the vector path, move it down..

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop I followed your suggestion on aiter/ops/triton/attention/qsa.py:372: the comment "Decode-sized calls without stacking use the CUDA-core kernel." now sits right above the CUDA-core launch, after the MFMA branch.

(commit 4811ba315)

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it sits right above the cuda-core launch now.

one nit: that branch is not only decode-sized — prefill-size calls also land there when contiguous is false or head_dim < 16 (aiter/ops/triton/attention/qsa.py:384). maybe fallback calls use the CUDA-core kernel. instead?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Agreed, it is the fallback for every call the stacked and MFMA branches do not take. In 4feba25c4 the comment reads # Fallback calls use the CUDA-core kernel.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, the comment matches the code now (aiter/ops/triton/attention/qsa.py:444), the CUDA-core kernel does take every shape the stacked and MFMA branches skip. resolved from my side.

k50112113
k50112113 previously approved these changes Oct 30, 2026

@k50112113 k50112113 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM on my side but please address @valar-cop 's comments thanks.

- The stacked decode scorer's tiles and the FP8 GQA tiles move from Python
  dicts to configs/<arch>/triton/attention/qsa/DEFAULT.json, read with
  resolve_config_dir() and load_config_json(). gfx942 and gfx950 get the
  previous Python values; gfx1250 is seeded byte-identical from gfx950.
  Only the stacked and FP8 GQA paths read the file, so BF16 calls do not
  depend on it on any arch.
- The MXFP4 readers decode the group scale with
  _mxfp4_e8m0_to_fp32(e + 14) instead of a local copy. The bits are the
  same for all 256 exponents.
- The FP8 writer's kernel name includes FP8_MAX_C, which differs per arch.
- Move the CUDA-core comment from the MFMA branch to the CUDA-core launch.
- test_qsa.py: the cap test builds its values on the GPU with an explicit
  dtype, and the other float factories get a dtype too. The narrow
  page-table test uses a real [1, 3] table and no longer claims to catch a
  read of entry 3: such a read has no visible effect, and the test passes on
  the kernels without the fix as well.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ZLkanyo009

Copy link
Copy Markdown
Contributor Author

@valar-cop thanks. I pushed 4811ba315 on top of d26518567 (no rebase), and I answered the six inline comments in their threads:

  • Tiles: the stacked decode and FP8 GQA tiles are in configs/<arch>/triton/attention/qsa/DEFAULT.json (gfx950 measured, gfx942 the same values as before, gfx1250 seeded from gfx950). They are read with resolve_config_dir() / load_config_json() and equal the old Python tables entry by entry.
  • E8M0: the MXFP4 readers use _mxfp4_e8m0_to_fp32(e + 14) from quant.py, which gives the same bits as the removed local decode for all 256 exponents.
  • FP8_MAX_C is in the FP8 writer's repr key.
  • The narrow page-table test: the read past the table it claimed to catch has no visible effect, and the test also passed with the fix reverted. I made the table a real [1, 3] table and dropped the claim.
  • The cap test builds its tensors on the GPU with an explicit dtype, and the CUDA-core comment is moved.

Validation on MI355X (gfx950), d26518567 vs 4811ba315 in one process on the same GPU:

  • 350 writer / scorer (CUDA-core, stacked, MFMA) / GQA cases are bitwise identical, and all 348 compiled variants of the six kernels have the same AMDGCN instructions.
  • Over 35 timing cases (FP8 / FP4 writers and scorers, FP8 GQA), GPU kernel time median ratio is 1.000 (0.962 .. 1.010) and do_bench median is 1.000. Host time of the calls that read the tiles has a median ratio of 1.003.
  • Parity with ATOM main 9df8e849a: still 197 / 197 bitwise identical.
  • test_qsa.py 53 passed; test_quant_mxfp4.py 79 passed, 54 skipped.

Perf on gfx942: I have no access to a gfx942 machine, so I have no perf numbers there. All numbers in this PR are from MI355X (gfx950). On gfx942 only CI's Triton Test runs this PR. On d26518567, the MI300X shard with test_qsa.py passed.

GSM8K: as in my previous summary, ATOM does not call aiter's QSA ops yet. I ran full GSM8K (1319, 5-shot chat, strict match) on MI355X with ATOM main 9df8e849a calling this branch's ops, through a local ATOM copy that rebinds the five ops, next to the same ATOM with its own ops. TP1, KV block 16, real MTP-3:

KV / index cache ATOM in-tree ops this branch's ops
FP8 / FP8 0.9780 0.9773
FP8 / FP4 0.9780 0.9795

stderr is about 0.004. That run used d26518567. 4811ba315 gives bitwise identical outputs for every op the run uses, so I did not repeat it.

The PR description (Test Result, Notes for review and Self-Review) is updated for 4811ba315. The Triton Test Shard 5 failure (test_batched_gemm_afp4wfp4) is on main, tracked in #7047.

cc @valarLip

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes needed, mostly input validation on the new writers/readers

any gfx942 perf number? gsm8k? ci not finished on head sha

ci green.

Comment thread aiter/ops/triton/attention/qsa.py Outdated
return
if slots.shape[0] < rows or heads != cache.shape[2]:
raise ValueError("slots or heads disagree with the cache")
_, _, tail, _ = qsa_cache_row_layout(QSA_CACHE_FORMATS[fmt], head_dim)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

scale dtype is dropped here, but the reader checks it later.. keep sdtype and check it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Fixed in 4feba25c4. The writer keeps the scale dtype from qsa_cache_row_layout() and checks it together with the shape. A wrong plane now raises:

ValueError: scale must be the torch.uint8 [pages, page, heads(, groups)] plane of the cache

On 4811ba315, an FP4 cache with an int8 scale plane, or an FP8 cache with an FP64 scale plane, was written without any error. The new test_qsa_writer_rejects_wrong_scale_dtypes_and_pool_slices covers both cases and fails on 4811ba315.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, that one's covered. the dtype check at aiter/ops/triton/attention/qsa.py:119 plus test_qsa.py:415 does test both wrong planes, and the pool-slice check right next to it reads shape/stride() once so no cost on the writer.

ci still in progress on the head sha, let's see it go green.

Comment thread aiter/ops/triton/attention/qsa.py Outdated
return
if slots.shape[0] < rows or heads != cache.shape[2]:
raise ValueError("slots or heads disagree with the cache")
_, _, tail, _ = qsa_cache_row_layout(QSA_CACHE_FORMATS[fmt], head_dim)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only stride(-1) checked, but view needs contiguous.. a paged pool slice will raise bare RuntimeError

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Fixed in 4feba25c4. A new helper, _merge_slots, checks exactly what the view needs: stride(0) == page_size * stride(1) when both dims are > 1. It runs for the cache and for the scale plane. A slice of a larger pool now raises:

ValueError: cache must hold its pages back to back (stride(0) == page_size * stride(1)); a slice of a larger page pool does not

On 4811ba315, pool[:, :4] raised view's bare RuntimeError.

The FP4 scale plane must also have stride(-1) == 1, because the kernel steps over its groups with unit stride, so that is now checked too.

The check reads shape and stride() once, so writer host time stays the same (numbers in the summary). The same test covers pool slices for the cache and for the scale plane, for FP8 and FP4.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checked: _merge_slots at aiter/ops/triton/attention/qsa.py:73 runs for both planes, and when pages == 1 or page_size == 1 the view drops a size-1 dim, so the skip is right. lgtm on this thread.

ci still queued on 4feba25c4, ping me when the triton shards are green.

"""
if q.ndim != 3 or q.shape[1] <= 0 or q.shape[2] <= 0:
raise ValueError("q must have shape [tokens, heads, head_dim]")
key_fmt = _cache_format(compressed_k_cache, compressed_k_scale)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fp16 cache worked on main, now raises unsupported dtype? why narrow this

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop You are right: this PR broke it, and 4feba25c4 fixes it. I ran both versions on MI355X:

  • On main (699d75091), FP16 and FP32 caches work in the CUDA-core and MFMA scorers and in sparse GQA, because main only checks q.dtype == cache.dtype.
  • On 4811ba315, every FP16 / FP32 call raised a quantized QSA cache needs its scale plane.

The fix: _cache_format treats every float cache wider than 8 bits as unquantized. That is KEY_FMT 0, which reads the keys in Q's dtype, as main does. The kernels are unchanged. An unquantized cache passed with a scale plane now raises an unquantized QSA cache takes no scale plane, and the writer rejects unquantized caches.

The new test_qsa_takes_unquantized_fp16_and_fp32_caches runs the stacked, CUDA-core and MFMA scorers and sparse GQA with FP16 and FP32 caches, against an FP32 reference. It fails on 4811ba315.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checked the diff — _cache_format now sends fp16/fp32 to KEY_FMT 0 and qsa.py:308 still keeps main's q.dtype == cache.dtype check, so this matches main again. fine on my side.

ci still queued on the head sha, let's wait for triton test to go green.

Comment thread aiter/ops/triton/attention/qsa.py Outdated
raise ValueError("FP8 QSA rows need a GPU with an FP8 dtype")
return head_dim, FP8_DTYPE, (), torch.float32
if fmt == "fp4":
if head_dim % FP4_GROUP:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

head_dim % 32 allows 96, but fp4 readers need power of two.. move the check into the layout helper

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Moved in 4feba25c4. qsa_cache_row_layout("fp4", d) now needs a power-of-two d >= 32, because the FP4 kernels index the packed bytes with tl.arange(0, d / 2). The writer's separate check is gone.

The scorer calls the same helper, so FP4 logits with head_dim = 96 now raise ValueError: FP4 rows need a power-of-two head_dim >= 32. On 4811ba315 the helper accepted 96, and the scorer failed with a Triton CompilationError.

The new test_qsa_fp4_rows_need_a_power_of_two_head_dim covers the helper, the writer and the scorer.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, one check in qsa_cache_row_layout is the right place, and a ValueError before the launch beats a CompilationError from the tile. test hits helper, writer and scorer, lgtm.

Comment thread aiter/ops/triton/attention/qsa.py Outdated
if (
v_cache.dtype != k_cache.dtype
or v_scale is None
or k_scale.shape != k_cache.shape[:3]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

k_scale/v_scale dtype not checked, wrong plane gives silently wrong scores.. add float32 check?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Added in 4feba25c4: k_scale and v_scale must be float32, beside the shape and stride checks. On 4811ba315, a K/V scale plane passed as its int32 bits was accepted, and GQA returned finite but wrong outputs (max error 3.7e9). An FP16 v_scale was also accepted.

Both cases now raise:

ValueError: FP8 K/V need matching float32 [pages, page, heads] scales

They are in test_qsa_scorers_reject_scale_planes_that_do_not_match_the_cache, which fails on 4811ba315.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checked, yes. k_scale/v_scale shape, stride and float32 checks at aiter/ops/triton/attention/qsa.py:576-580, and the test hits both the int32 bits and the fp16 v_scale (op_tests/triton_tests/attention/test_qsa.py:504-509). one small thing: the scorer takes the scale dtype from qsa_cache_row_layout (qsa.py:311), maybe do the same here instead of the literal torch.float32, so there is one source for the format. LGTM otherwise.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Changed in 9db2881. The writer, the scorer and GQA now check their scale planes with one _check_scale_plane(name, cache, scale, fmt, head_dim), which takes the shape and dtype from qsa_cache_row_layout. GQA calls it for k_scale and v_scale with the FP8 format, so the float32 is no longer written out by hand there. A non-contiguous head dim now gets its own "contiguous in the head dim" error instead of the scale-plane one.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, checked 9db2881: _check_scale_plane takes the dtype from qsa_cache_row_layout (aiter/ops/triton/attention/qsa.py:91) and gqa passes both planes through it, so the literal float32 is gone. separate head-dim contiguity error is fine too. lgtm on this thread, just waiting on the aiter test to go green.

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some changes needed

any gfx942 perf number? gsm8k? and ci not green yet

ci green.

Comment thread aiter/ops/triton/attention/qsa.py Outdated
when the row is stored unquantized.
"""
if fmt == "bf16":
return head_dim, torch.bfloat16, None, None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fp16/fp32 cache also valid here, but this always returns bf16? maybe rename to 'native' or take the dtype as a param

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Not changed. qsa_cache_row_layout(fmt, head_dim) describes the stored row for one of the QSA_CACHE_FORMATS names. For "bf16" it returns a bfloat16 row, which is right for that name.

FP16 / FP32 caches have no scale plane and use Q's dtype, so they don't go through this helper; the op calls it only for "fp8" / "fp4". So no path gets a wrong dtype from it.

Renaming "bf16" to "native" would change an exported name, the stacked_decode.bf16 JSON key on three archs, the test parameters and bench_qsa.py --index-dtype bf16. That is a lot of churn for a name that is not wrong for the rows it describes.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, fair — fp16/fp32 never reach the helper, only the named formats do, so keep bf16. maybe just one half sentence in the docstring that bf16 is the unquantized family (the stacked_decode.bf16 tile is shared by fp16/fp32 too), so nobody asks again. lgtm otherwise.

Comment thread aiter/ops/triton/attention/qsa.py Outdated
and head_dim >= 16
):
tile = _tuned_tile("stacked_decode", QSA_CACHE_FORMATS[key_fmt])
if tile is not None and tile["BMT"] * heads >= 16:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tile read happens before the usability gate, so a missing json raises instead of falling back.. can we fold the gate into the dispatch?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Fixed in 330d7a8, together with the required=True thread. The tile is still read only when a call can use the stacked scorer (at most 256 rows, stacked_decode, contiguous, power-of-two heads and head dim). _tuned_tile now returns None on an arch without the file, so one gate, tile is not None and tile["BMT"] * heads >= 16, covers both cases: a missing table, and too few heads to fill M. In both, the call goes to the other scorers instead of raising. test_qsa_stacked_decode_falls_back_without_a_tile checks that the stacked mode is not launched and that the scores match the reference.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checked 330d7a8: the read is inside the usability gate now (aiter/ops/triton/attention/qsa.py:350-358), _tuned_tile returns None without the file (:250), and the single stacked = tile is not None and ... gate at :360 covers both the missing table and the heads < 16 case. the fallback test pins it (op_tests/triton_tests/attention/test_qsa.py:794). resolved from my side.

mask=valid[:, None],
other=127,
)
group_scale = _mxfp4_e8m0_to_fp32(exponent.to(tl.int32) + 14)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no clamp to 240 on read side? e=254 gives inf and poisons top-k, please add tl.minimum(exponent, 240)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Not changed: no writer produces an E8M0 exponent above 240.

  • aiter's QSA FP4 writer passes MAX_SCALE_E8M0=240 to _mxfp4_quant_op; test_qsa_fp4_writer_caps_the_scale_for_the_fp16_decode covers this.
  • ATOM's writer clamps the exponent to [-127, 113] (atom/model_ops/qwen4_exp/ops/qsa.py, lines 233-234), which is E8M0 <= 240.

So an exponent above 240 only appears in a cache that something else wrote wrongly. Also, the decode is bitcast((e + 14) << 23). e = 241 gives +inf, but e = 242..255 overflow into the sign bit and give -0 or tiny negative values (e = 254 gives -2^-115, not inf).

tl.minimum(exponent, 240) would turn those into a finite 2^113 scale, which is still a wrong value. It would also differ from ATOM's reader, which does not clamp either, and it adds an instruction to every FP4 decode for inputs that no writer produces. So the reader stays bitwise identical to the decode it replaced.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fair, checked it: the writer cap is at aiter/ops/triton/_triton_kernels/attention/qsa.py:189, the test pins [240, 240] at op_tests/triton_tests/attention/test_qsa.py:338, and :222 is the same bitcast decode. clamping would only turn a corrupt cache's inf into another wrong number, so no change from me.

keep the e <= 240 line in the _e2m1_value_lo docstring (:67) so the cap stays documented, that is all. triton tests are green on head now.

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not ready yet.. missing config fallback will break other arches

any perf number? gsm8k? and ci is still queued on head

ci green.

Comment thread aiter/ops/triton/attention/qsa.py Outdated
def _tuned_tile(kernel: str, case: str) -> dict:
"""`kernel[case]` of `configs/<arch>/triton/attention/qsa/DEFAULT.json`."""
cfg_dir = resolve_config_dir("attention", "QSA", backend="triton")
return load_config_json(f"{cfg_dir}/DEFAULT.json")[kernel][case]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

required=True here, but qsa/ only exists on gfx942/950/1250. other arches will just crash. load it optionally and fall back to _kernel_config?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Fixed in 330d7a8. _tuned_tile now loads configs/<arch>/triton/attention/qsa/DEFAULT.json with required=False (as pa_decode.py does) and returns None when the file does not exist. Then FP8 sparse GQA keeps the _kernel_config tiles, the ones BF16 uses, and stacked_decode=True calls are scored by the CUDA-core / MFMA scorers instead of raising FileNotFoundError. Only these two paths read the file; the BF16 defaults never did.

Two new tests run as on gfx1100, which has no QSA config: test_qsa_fp8_sparse_gqa_matches_dequantized_kv[*-missing] (decode and prefill) and test_qsa_stacked_decode_falls_back_without_a_tile. On MI355X they fail on 1a417c7 with FileNotFoundError and pass on 330d7a8 (test_qsa.py: 110 passed).

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, checked 9db2881. _tuned_tile loads with required=False (aiter/ops/triton/attention/qsa.py:250) and both callers gate on tile is not None (:360, :586), so no arch raises; the gfx1100 fixture plus the two new tests cover both fallbacks. resolved from my side.

_SCORING_COLUMN_PROGRAMS = 64


@functools.lru_cache(maxsize=1024)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

load_config_json already caches, remove this lru_cache..

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Kept. load_config_json caches only reading and parsing the file. Without this cache, every call still runs resolve_config_dir() (get_arch() plus three regex checks), builds the path and does two dict lookups.

When I measured this in round 2, removing the cache cost +0.9 us host time per stacked decode call and +1.2 to 1.9 us per FP8 GQA call. That is +5 to 8% in do_bench on the launch-bound decode shapes. With the cache, the host time ratio is back to a median of 1.003.

pa_prefill_sparse._pinned_prefill_config uses the same pattern: an lru_cache around a config lookup that returns None when there is no entry. Since 330d7a8, this cache also keeps the None on archs without the file.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fair, i forgot resolve_config_dir isn't cached, only load_config_json is... that settles it, keep the cache.

just add half a line in the docstring saying why the outer cache is here (resolve_config_dir cost, not the file read), so nobody strips it later.



@pytest.mark.parametrize("fmt", ["fp8", "fp4"])
@pytest.mark.parametrize(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

head_dim and heads hardcoded to 128/4 here, the fp4 head_dim=32 case is never tested. parametrize?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valar-cop Done in 330d7a8. test_qsa_quantized_index_scores_match_dequantized_keys now runs:

  • head dim 32 / 64 / 128 / 256 with 4 index heads;
  • 1 and 2 heads at head dim 128;
  • FP8 and FP4, in every scorer case (stacked, CUDA-core and prefill MFMA; page size 4 and 16).

FP4 at head dim 32 (one group per row) matches the dequantized reference at rtol = atol = 1e-4 in all three scorers, so the kernels needed no change. With 1 or 2 heads, stacked_decode=True falls back to the CUDA-core scorer (BMT * heads < 16), and the test covers that too. On MI355X, test_qsa.py: 110 passed.

cc @valarLip

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checked the grid at op_tests/triton_tests/attention/test_qsa.py:572-578 .. fp4 dim 32 and the 1/2-head fallback are both covered now, and with BMT=4 the heads 1/2 cases do land on the CUDA-core kernel as you say. lgtm.

only nit: the CUDA-core row is page size 4 only, the two page sizes are in the stacked and MFMA cases. fine by me, no rerun needed.

ZLkanyo009 and others added 5 commits October 31, 2026 22:13
Pick up #7047 (mask OOB e8m0 scales on the ragged-K path of batched_gemm_afp4wfp4), which fixes the MI35X Shard 5 NaNs in test_batched_gemm_afp4wfp4.

Co-authored-by: Cursor <cursoragent@cursor.com>
configs/<arch>/triton/attention/qsa exists only for gfx942, gfx950 and
gfx1250, and _tuned_tile read it with required=True, so stacked_decode
and FP8 sparse GQA raised FileNotFoundError on any other arch. Load it with
required=False: without the table stacked_decode scores with the other
kernels and FP8 K/V keep the _kernel_config tiles, as BF16 does.

Tests: run the FP8 GQA case and a stacked_decode call as on gfx1100 (no
QSA config), and parametrize the quantized scoring test over head_dim
(32, 64, 128, 256) and heads (1, 2, 4), which covers FP4 at head_dim 32.

Co-authored-by: Cursor <cursoragent@cursor.com>
The FP8 QSA writer computed the row amax, scale and clamp itself, which is
what _fp8_quant_op in fused_fp8_quant.py does with one group per row; K and
V repeated that code. Call _fp8_quant_op from one _store_fp8_row helper.
Scales are unchanged; at rounding ties a few bytes move one E4M3 code,
because _fp8_quant_op multiplies by 1 / scale instead of FP8_MAX / amax.

The test's MXFP4 dequant reimplemented mxfp4_to_f32 and e8m0_to_f32 from
aiter.utility.fp4_utils, which test_quant_mxfp4.py uses; use them.

Co-authored-by: Cursor <cursoragent@cursor.com>
…4 key-tile decode, one matrix-core launch

- test_qsa: _index_case / _stacked_case and their references become one
  _case and one _reference; the writer reject tests share _planes; the
  narrow page-table test uses them instead of its own reference.
- kernels: the MFMA and stacked scorers share _fp4_key_tile; the CUDA-core
  scorer loads unquantized Q rows with _load_q_row; duplicate comments go.
- wrapper: the stacked and MFMA scorers share one launch; the FP8 and FP4
  writers share their common arguments.

Outputs are bitwise identical to the previous commit.

Co-authored-by: Cursor <cursoragent@cursor.com>
…plane check

- _qsa_paged_mqa_logits_mfma_kernel takes STACK_HEADS; the stacked kernel and
  its repr are gone.
- _load_fp4_keys is the one MXFP4 key decode of the CUDA-core and MFMA scorers.
- _check_scale_plane validates the writer, scorer and GQA scale planes.
- The FP8 writer takes None for the second plane instead of HAS_SECOND.
- bench_qsa: _quantize returns one (cache, scale) pair; _scenario is gone.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ZLkanyo009

Copy link
Copy Markdown
Contributor Author

Pushed 1a417c7fd..9db28812b (330d7a8, a6881c0, e3ae331, 9db2881): the round-4 fixes, a structure check and a whole-PR cleanup.

Round 4 (replied inline):

  • Fixed: the tile JSON is optional. On an arch without it, stacked_decode calls use the other scorers and FP8 GQA uses _kernel_config, instead of raising FileNotFoundError. The quantized scoring test now also covers head dims 32 / 64 / 256 and 1 / 2 index heads.
  • Kept, with reasons: the reader's E8M0 decode, the lru_cache, and the "bf16" format name.
  • gfx942: I have no gfx942 machine, so the gfx942 tiles stay copies of gfx950, and CI's Triton tests are the only gfx942 run.

Structure check. Every file in the diff was reviewed for the right place and for reuse. The only new files are the three tile JSONs in configs/<arch>/triton/attention/qsa/, and no environment variables were added. The FP8 writer now quantizes with fused_fp8_quant._fp8_quant_op, as the FP4 writer uses _mxfp4_quant_op, and the test's FP4 dequantization uses aiter.utility.fp4_utils.

Cleanup over the whole PR diff (lines added / removed against main):

before (330d7a8) after (9db2881)
source code +941 / −71 +784 / −101
source comments and docstrings +98 / −11 +99 / −12
test code +766 +695
test comments and docstrings +30 / −1 +35 / −1
bench code +124 / −17 +100 / −17

Net source lines go from +957 to +770. The removed count grows because the MFMA kernel on main now also carries the stacked mode. The three tile JSONs (+108) are unchanged.

  • Kernels:
    • The stacked scorer is now a STACK_HEADS constexpr mode of _qsa_paged_mqa_logits_mfma_kernel; the separate stacked kernel and its repr are gone. Without STACK_HEADS the MFMA kernel keeps its one-tile-per-program structure, so its prefill code is unchanged (see Behavior).
    • The CUDA-core and MFMA scorers share one MXFP4 key decode (_load_fp4_keys), so the e + 14 / _e2m1_value_lo pairing lives in one place.
    • The FP8 writer's K and V rows share one _store_fp8_row, and V is written when second_cache is not None instead of through a HAS_SECOND flag and placeholder pointers.
    • The CUDA-core scorer loads unquantized Q rows with the existing _load_q_row.
  • Wrapper: one _check_scale_plane checks the writer, scorer and GQA scale planes, and a non-contiguous head dim now gets its own error instead of the scale-plane one. The matrix-core scorers have one launch, and the two writers share their common launch arguments.
  • Tests: two case builders and two torch references became one _case and one _reference; the writer reject tests share _planes; the narrow page-table test uses the shared reference.
  • Bench: _quantize returns one (cache, scale) pair and is called for K and V; the forwarding _scenario is gone.
  • Deleted: repeated comments and a scale alias in the wrapper.

Behavior (MI355X, one GPU, 330d7a8 vs 9db2881 in one process):

  • Outputs: 350 / 350 writer, scorer (CUDA-core, stacked, MFMA × BF16 / FP8 / FP4, head dim 64 / 128 / 256) and GQA cases are bitwise identical. BF16 outputs are also bitwise identical to main (62 / 62).
  • FP8 writer bytes: _fp8_quant_op multiplies by 1 / scale, the old code by FP8_MAX / amax. On the 4096-row ATOM parity inputs about 2e-4 of the BF16-row bytes move by one E4M3 code; every such value is an exact tie between the two codes, so the error is the same, and the scales are identical.
  • Instructions: The BF16 CUDA-core variants, the FP4 writer and GQA are unchanged. Of the BF16 MFMA (prefill) variants 24 / 36 are identical and the other 12 differ by at most 6 instructions. The stacked variants, the FP8 / FP4 MFMA and CUDA-core variants and the FP8 writer changed.
  • Time: GPU kernel time median 1.000 over 35 cases (0.954–1.025), host time median 1.001.
  • Tests: test_qsa.py 110 passed. Crossed with 330d7a8, 104 / 110 pass each way; the 6 that fail only assert the stacked kernel's name. test_quant_mxfp4.py: 79 passed, 54 skipped. FP16 / FP32 caches: 12 / 12.
  • Parity with ATOM main: 194 / 197 bitwise; the other three are the FP8 writer cases above. All reader cases are identical. Over 93 perf cases the aiter / ATOM median is 0.996.
  • GSM8K with ATOM calling these ops: FP8 / FP8 0.9765, FP8 / FP4 0.9780 (ATOM's own ops: 0.9780 / 0.9780).
  • black and ruff check . pass.
  • CI on 9db2881: Checks, Triton Test (MI300X and MI35X) and OPUS Test are green; the Aiter Test MI35X Standard Tests are still running.

The PR description is rewritten to be shorter and to explain why.

@k50112113 could you take another look and re-approve if it looks good? cc @valarLip

ZLkanyo009 and others added 3 commits November 2, 2026 09:44
- The stacked scorer is its own kernel again, so the prefill MFMA kernel only
  gains the KEY_FMT branches; both decode MXFP4 keys with _load_fp4_keys.
- The CUDA-core scorer keeps main's Q and key loads in its unquantized branch.
- FlyDSL dispatch only adds the compressed_k_scale check; main's docstrings,
  launch arguments, bench imports, labels and help text are restored.

Co-authored-by: Cursor <cursoragent@cursor.com>
…rings

Co-authored-by: Cursor <cursoragent@cursor.com>
…_ok bound

Co-authored-by: Cursor <cursoragent@cursor.com>
@ZLkanyo009

Copy link
Copy Markdown
Contributor Author

Pushed 9db28812b..c3ce6d115 (069663c, b2c4b78, c3ce6d1): a minimal-change pass over the whole PR diff, checked by an independent review. Every hunk was checked against what the PR is for, and existing main code that the feature does not need to touch is back as it is on main.

Size against main (git diff 0dd271600...):

before (9db2881) after (c3ce6d1)
files 9 9
lines +1725 / −135 +1793 / −93
source +883 / −113 +962 / −80
test +730 / −1 +728 / −0
bench +112 / −21 +103 / −13

Fewer main lines are touched (−135 → −93). More lines are added because the stacked scorer is its own kernel again; all of those lines are new code.

Reverted, by file:

  • aiter/ops/qsa.py: QSA_CACHE_FORMATS is no longer exported (nothing used it). The FlyDSL dispatch no longer gains a compressed_k_cache.dtype == torch.bfloat16 condition. That condition changed what FP16 / FP32 caches do with allow_flydsl on main; quantized keys are already kept off FlyDSL by compressed_k_scale is None. The docstring keeps main's text, and one new paragraph says FP8 / MXFP4 keys and stacked_decode are Triton only.
  • aiter/ops/triton/_triton_kernels/attention/qsa.py:
    • The stacked scorer is a separate _qsa_paged_mqa_logits_stacked_kernel again, instead of a STACK_HEADS mode that rewrote main's prefill MFMA kernel. The MFMA kernel only gains the KEY_FMT branches. Leaving indentation aside, three of its main lines change: the page_ok bound valar-cop asked for in both kernels (r4425149119), the .to(...) cast FP8 keys need, and the BF16 Q load moved under else:.
    • The CUDA-core scorer keeps main's Q and key loads in its unquantized branch (indented one level), rather than calling _load_q_row.
    • The module docstring is main's again, and the CUDA-core docstring keeps main's "directly".
  • aiter/ops/triton/_triton_kernels/quant/quant.py: no new comment on main's four round-to-nearest-even lines; the two defaulted constexprs stay.
  • aiter/ops/triton/attention/qsa.py: main's module docstring and its NUM_HEADS=q.shape[1], HEAD_DIM=q.shape[2] launch arguments are back.
  • op_tests/triton_tests/attention/test_qsa.py: main's module docstring is back; nothing in main's tests changes.
  • op_tests/op_benchmarks/triton/bench_qsa.py: main's imports, section comment, --op help text, line_names and qsa_select_paged_tokens call are back, and the _label helper is gone. What stays is what the new options need: --op logits, --index-dtype, --kv-dtype, --stacked-decode, the MTP-3 verify shape and per-format byte counts.

valar-cop's two docstring nits on 9db2881 are in (b2c4b78): _tuned_tile says it is cached because resolve_config_dir is not (r4440420203), and qsa_cache_row_layout says "bf16" stands for every unquantized cache (r4440421005).

Also in this push:

  • The stacked kernel's page_ok gets the same bounds comment as the MFMA kernel (r4425149119).
  • The description now says the gfx942 tile file is an exception to the seed list in configs/AGENTS.md, kept so that MI300X CI runs the stacked scorer and FP8 GQA.
  • The Input checks list now says that a 1-byte float cache without its scale plane raises ValueError; main accepted it when the dtypes matched.

Not changed (seen, but not for this PR):

  • main's prefill MFMA kernel only gains the KEY_FMT branches and the columns < num_columns bound on page_ok (the out-of-bounds table read valar-cop asked to fix in both kernels). The stacked scorer stays a separate kernel, because merging it would rewrite main's kernel body.
  • The CUDA-core scorer keeps main's NUM_HEADS <= 4 limit.
  • FlyDSL stays BF16-only. FP16 / FP32 caches with allow_flydsl behave as on main, and FP8 / MXFP4 keys and stacked_decode use Triton only.
  • _SCORING_BLOCK_N, _SCORING_COLUMN_PROGRAMS, the MFMA 16 / 128 tile and _kernel_config stay in Python: main's BF16 path reads them on arches without the QSA JSON too. Moving them into the JSON would be a separate PR.
  • The gfx942 tiles are gfx950 copies, not measured on gfx942.

Validation (MI355X, one GPU):

  • Outputs: 350 / 350 writer, scorer and GQA cases are bitwise identical to 9db2881 (same process, same inputs). BF16 outputs are bitwise identical to main (62 / 62).
  • Instructions against 9db2881: the CUDA-core scorer (108 variants), the writers (16) and GQA (8) are unchanged, and so are the stacked scorer's 108 variants. The MFMA variants differ because the kernel is back to main's structure.
  • Instructions against main: the BF16 kernels differ only in where kernel arguments sit (the new scale-stride arguments), plus the register allocation that follows. BF16 GPU time against main: median 0.997 over 10 cases (0.987–1.013).
  • GPU kernel time against 9db2881: median 1.000 over 35 cases (0.988–1.004), host time median 0.999. Against ATOM main over 93 cases: median 0.996, and no case above 100 us is more than 0.2% slower.
  • Tests: test_qsa.py 110 passed; test_quant_mxfp4.py 79 passed, 54 skipped; FP16 / FP32 caches 12 / 12; parity with ATOM main 194 / 197 (the three are the FP8 writer tie cases described before).
  • GSM8K with ATOM calling these ops: FP8 / FP8 0.9727 and 0.9788 on a rerun of the same code, FP8 / FP4 0.9765 (stderr about 0.004; ATOM's own ops 0.9780 / 0.9780).
  • bench_qsa.py --op logits: stacked_decode makes FP4 decode scoring 1.24–3.69x faster (MTP-3 9x4: 2.76x). For FP8 it is 1.17–1.22x faster at 9x4, 32x3 and 128x1, and 0.86–0.96x at 16–64 single-token rows, where launch cost dominates. These replace the 3.3–3.8x / 1.25–1.3x in the old description, which I could not trace to a log.
  • black and ruff check . pass.
  • Correction to my previous summary: its instruction and time comparison was against e3ae331, not 330d7a8. The bitwise result holds against 330d7a8, because each step in between was also 350 / 350.
  • CI on 069663c: running. On 9db2881, everything that finished is green.

The PR description is updated to match.

@k50112113 could you take another look and re-approve if it looks good? @valar-cop your two docstring notes (r4440420203, r4440421005) are in b2c4b78. cc @valarLip

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

ci green.

k50112113
k50112113 previously approved these changes Nov 2, 2026
@ZLkanyo009

Copy link
Copy Markdown
Contributor Author

The only red check on c3ce6d115 is Aiter Test / Standard Tests (1 GPU), MI35X shard 4/8, and the Aiter Test Gate that aggregates it. That shard hits the 90-minute step limit (job: The action 'Tests' has timed out after 90 minutes). Every result it printed before the cut-off passed. Attempts 2 and 3 timed out the same way. All other checks are green, including Triton Test (test_qsa.py) and OPUS.

The timeout is not caused by this PR:

  • split_tests.sh balances shards with .github/split-test-times/aiter.tsv, and CI checks out this branch's head, so it uses the copy from this branch's base (0dd271600, 10-31). That copy underestimates two files that landed together in shard 4:

    • test_topk_per_row.py: listed 966 s, actual 2196 s;
    • test_gemm_a8w8.py: listed 271 s, actual about 1790 s.

    With test_concat_cache_mla.py (11.5 min) in the same shard, it runs past 90 minutes. This PR changes none of these files.

  • main has already fixed both inputs: #7290 refreshed the split times, and #7316 cut the A8W8 GEMM test overhead. On main, the same shard finishes in 66–69 minutes.

  • Re-running cannot help. The shard assignment is deterministic for this head, and the workflow checks out head.sha, not a merge with current main.

We have not merged main again because that push would dismiss @k50112113's approval from yesterday. Could a maintainer merge with this gate overridden? If you would rather we merge main and go through a fresh CI run plus a re-approval, tell us and we will do that instead.

cc @valarLip @k50112113

ZLkanyo009 and others added 2 commits November 5, 2026 02:35
Resolve the aiter/ops/qsa.py conflict by keeping both new exports:
qsa_sparse_prefill (#6576) and qsa_store_quantized_rows (this PR).

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@ZLkanyo009

Copy link
Copy Markdown
Contributor Author

Pushed c3ce6d115..b2fb752dc: merged current main (47b4e0869) into the branch to resolve the conflict with #6576, plus one docstring fix from the self-check below. No rebase, no force push.

Conflict. The only conflict was at the end of __all__ in aiter/ops/qsa.py. This PR adds qsa_store_quantized_rows there, and #6576 adds qsa_sparse_prefill; both are kept, in the list's alphabetical order. The rest of #6576's changes to that file (the Sequence import, _HAS_FLYDSL_PREFILL and qsa_sparse_prefill) merged cleanly. quant.py also merged cleanly: #6458 adds two _mxfp4_pack_op(..., USE_ASM=True) calls, which leave ROUNDING_MODE at its default of 0, so this PR's static_assert(not USE_ASM or ROUNDING_MODE == 0) holds for them.

The merge commit (06a7aa8c2) leaves this PR's diff against main line-for-line the same as before (9 files, +1793 / −93); only main moved.

Fixes from main this brings in. #7290 (auto-updated split test times) and #7316 (less A8W8 GEMM test overhead). On c3ce6d115, the only red check was Aiter Test / Standard Tests (1 GPU) (MI35X, 8, 4). That shard hit the 90-minute step timeout with no failing test. Its longest files were test_topk_per_row.py (37 min) and test_gemm_a8w8.py (30 min), and it ran no QSA tests. Since #7290 and #7316 landed, that shard passes on main in about 30 minutes (for example runs 41922216011, 41910913679 and 41876027576).

review-pr self-check (aiter's .claude/skills/review-pr, on the merged head):
Must change 0 / Worth addressing 1 / Smaller 2, follow-up 1. The conflict resolution, the new FlyDSL ops from #6576 / #6670, #6458's MXFP4 callers and the default paths all check out.

  • Worth addressing: the PR description's Reviewed revision still pointed at c3ce6d1154aa. Updated to b2fb752dc021, and the validation section now says which results come from c3ce6d115.
  • Smaller, fixed: _load_fp4_keys claimed its decode is exact in dtype. That holds for BF16 / FP32 Q, but an FP16 Q can overflow at large group scales, so the docstring now says "exact for BF16 / FP32". Comment only (b2fb752dc); nothing else in the code changed.
  • Smaller, not changed: in the two writer kernels, row < num_rows in valid is always true for the (rows, heads) grid. Removing it would change the kernel signatures with no GPU to check, for only a little dead code.
  • Follow-up, not in this PR: the MFMA path for more than 256 query rows keeps main's gate, which doesn't check that head_dim is a power of two, so FP8 keys with head dim 96 inherit it. It hasn't been reproduced, and a fix would change main's BF16 dispatch. It's listed under "Not changed" in the description.

Validation.

  • black and ruff 0.16.0 (the pinned CI version) pass on the PR's files. The one ruff finding in the repository is EXE001 on aiter/aot/flydsl/gemm.py, which comes from main and is outside this diff.
  • 3rdparty submodules are unchanged from main.
  • No GPU was available for this push, so test_qsa.py runs in CI's Triton Test. Apart from the docstring line, the PR's code is the same as c3ce6d115, where it passed 110 / 110 on MI355X; that run also gave 350 / 350 bitwise-identical A/B outputs against 9db28812b and BF16 outputs bitwise identical to main.
  • CI on b2fb752dc: running.

@k50112113 this push resets your 11-02 approval; could you take another look and re-approve if it looks good? cc @valarLip

ZLkanyo009 and others added 2 commits November 6, 2026 03:22
MXFP4 keys are now decoded into a full-width key tile (_load_fp4_keys
interleaves the low and high nibbles), so the CUDA-core, MFMA and stacked
scorers differ by key format only in how the key tile is loaded. main's Q
loads and per-head scoring in the CUDA-core and MFMA kernels are back to
their original text, and _load_q_row is gone.

Also:
- sparse GQA: one QK dot and one PV dot for both cache formats; FP8 only
  adds the key-scale and value-scale multiplies.
- the writers drop `row < num_rows`, always true for the (rows, heads) grid,
  and the num_rows argument.
- qsa_cache_row_layout returns a QSARowLayout NamedTuple.
- bench_qsa.py: bench_select picks logits / select from args.op instead of
  a scoring_only flag.
- test_qsa.py: _store builds its planes with _planes.

PR diff against main: +1793 -93 before, +1701 -61 after; the kernel file
changes 24 of main's lines instead of 56.

The FP4 scores now sum in a different order, so they are not bitwise equal
to the previous head; the other outputs are expected to be.

Co-authored-by: Cursor <cursoragent@cursor.com>
Use the layout's field names in the benchmark, as the wrapper and the
test do, and keep the reviewed comment on the CUDA-core fallback.
Benchmark output and kernels are unchanged.

Co-authored-by: Cursor <cursoragent@cursor.com>
@ZLkanyo009

Copy link
Copy Markdown
Contributor Author

Pushed b2fb752..dab71f0 (two commits): a tidy-up of this PR's own code, so that each concept has one code path and main's lines stay as they are. No behavior change except the FP4 summation order (below).

What changed and why it is simpler:

  • One scoring path for all key formats. _load_fp4_keys now returns the whole decoded [rows, head_dim] key tile (tl.interleave of the two nibbles) instead of two half-width planes. The CUDA-core, MFMA and stacked scorers score it exactly like a BF16 / FP8 tile, so the even / odd Q loads, _load_q_row and the second dot / head loop are gone. main's Q loads, CUDA-core scoring lines and MFMA per-head loop are back to main's text instead of being indented under an else:.
  • No duplicated tails. FP8 GQA computes qk and the PV tl.dot once; the KV_FP8 branch only applies the key scale and reads the value scale.
  • No dead conditions or arguments. The writers dropped row < num_rows (always true for a (rows, heads) grid) and the num_rows argument; GQA passes k_scale / v_scale directly because the checks above already guarantee they are None for unquantized caches.
  • Named fields instead of positions. qsa_cache_row_layout returns a QSARowLayout NamedTuple (dim, dtype, scale_tail, scale_dtype), and the wrapper, the test and bench_qsa.py read it by field name; tuple unpacking still works.
  • One mode per function. bench_select lost its scoring_only flag and picks the provider from args.op; the test's _store reuses _planes instead of building the planes a second time.
  • Removed the CUDA-core docstring sentence about even / odd Q dims, which the interleave made wrong.

Size: PR diff +1793 −93 → +1708 −61. In the kernel file, the main lines this PR touches went from 56 to 24.

Validation on MI355X (gfx950), one GPU, on a567922. dab71f0 only changes bench_qsa.py (same output; rerun with --op logits --index-dtype fp4|fp8 and --op gqa --kv-dtype fp8) and adds back a comment, so the kernels are the same:

  • test_qsa.py 110 passed, test_quant_mxfp4.py 79 passed, 54 skipped; black / ruff pass.
  • Against b2fb752dc in one process: 265 / 350 outputs bitwise identical; the other 85 are FP4 scorer outputs (summation order). BF16 / FP8 scorer and GQA variants have identical AMDGCN instructions; median GPU time ratio 0.997.
  • FP4 GPU time: CUDA-core scorer 0.26–0.29x (9x4 verify 34.6 → 9.1 us), MFMA 0.71–0.74x (64k prefill 899 → 639 us), stacked 0.88–0.93x.
  • BF16 outputs are bitwise identical to main (62 / 62).
  • FP4 error: against ATOM main the FP4 logits are at most 1.4e-6 apart (20 cases, previously bitwise), and every FP4 top-k selection picks the same tokens. Overall parity is 174 / 197 bitwise; the other 3 are the FP8 writer cases described in the PR body, unchanged. The FP4 tests against dequantized keys pass, and swapping the interleave order makes 36 FP4 cases fail, so the tests pin the layout.
  • GSM8K (1319, strict match) with ATOM calling these ops: FP8 index 0.9742, FP4 index 0.9795 (ATOM's own ops 0.9780 / 0.9780, stderr about 0.004).

The PR description is updated (FP4 decode, parity, performance table, Validation).

@k50112113 your approval was dismissed by the earlier push; could you take a look at the new head? cc @valarLip

@valarLip
valarLip merged commit 0de43e6 into main Nov 9, 2026
71 of 72 checks passed
@valarLip
valarLip deleted the lingzha/qsa-fp8-kv-index-stacked branch November 9, 2026 03:35
ruanjm pushed a commit that referenced this pull request Nov 10, 2026
…decode scorer (#7008)

* [Triton/Gluon] [QSA] FP8 KV and FP8 / FP4 index caches, stacked-head decode scorer

Bring the QSA operator changes of ROCm/ATOM#2652 into the Triton QSA
kernels that #6469 / #6515 ported from ATOM before that PR landed.

- qsa_store_quantized_rows: FP8 rows with one FP32 scale per (token,
  head) = row amax / FP8 max (K and V in one launch), and MXFP4 rows
  (packed E2M1 pairs, one E8M0 exponent per 32 dims, rounded up so a
  group never clips). qsa_cache_row_layout describes the stored row.
- qsa_sparse_paged_gqa(k_scale=, v_scale=): reads E4M3 K/V tiles,
  multiplies the key scale into its score column and folds the value
  scale into the probability before PV; FP8 has its own tile config.
- qsa_paged_mqa_logits / qsa_select_paged_tokens(compressed_k_scale=):
  FP8 (row scale) and MXFP4 compressed index keys in the CUDA-core,
  prefill MFMA and stacked scorers. FlyDSL stays BF16-only.
- stacked_decode=True: decode-sized calls (<= 256 rows) score with an
  MFMA kernel that stacks 4 rows x 4 index heads into M = 16, so a
  request's keys are read once per verify instead of once per row.
  Off by default, so existing BF16 callers keep their kernels.

The BF16 path and the existing signatures are unchanged; new arguments
are trailing keywords with defaults.

Co-authored-by: Cursor <cursoragent@cursor.com>

* [Triton/Gluon] [QSA] Address review: page-table bound, CPU import, writer checks

- Scorers: the MFMA and stacked kernels mask page-table loads with
  columns < num_columns as well as the rounded table width, so a narrow
  table (for example [1, 3] at page size 4) is not read past its end.
- Without a visible GPU, aiter.dtypes.fp8 is a uint8 placeholder. FP8_MAX
  is None there instead of a module-level torch.finfo error, so both QSA
  modules import on CPU; FP8 layouts raise until a GPU FP8 dtype exists.
- qsa_store_quantized_rows rejects non-contiguous, non-integer or
  multi-dim slots (the writers read slots[row] without a stride), a scale
  plane whose shape does not match the cache, and a second K/V plane whose
  cache or scale shape or dtype differs from the first.
- Tests for each case: a [1, 3] page table through the CUDA-core, MFMA and
  stacked scorers in BF16 / FP8 / FP4, bad slots and planes for both
  writers, and an import in a subprocess with no visible GPU.

Co-authored-by: Cursor <cursoragent@cursor.com>

* [Triton/Gluon] [QSA] Quantize FP4 index rows with _mxfp4_quant_op

Address review: drop the QSA writer's own E2M1 encoder and inline E8M0
math, and extend aiter's MXFP4 quantizer to cover what QSA needs.

- _mxfp4_quant_op / _mxfp4_pack_op / _mxfp4_pack_bits gain
  ROUNDING_MODE (0 = ties to nearest even, the existing behavior and
  default; 1 = ties away from zero). Both the normal and the denormal
  branch of the bit path implement mode 1; USE_ASM with mode 1 is a
  static_assert, since v_cvt_scalef32_pk_fp4_f32 only rounds to even.
- _mxfp4_ceil_scale_from_amax / _mxfp4_quant_op gain MAX_SCALE_E8M0
  (default 254 = uncapped, so inf / NaN amax keeps its NaN scale);
  only SCALING_MODE=1 accepts a cap.
- _qsa_store_rows_fp4_kernel loads the row as a [1, HEAD_DIM] tile and
  calls _mxfp4_quant_op(SCALING_MODE=1, ROUNDING_MODE=1,
  MAX_SCALE_E8M0=240): the cache bytes of ATOM's QSA writer, with E8M0
  kept within the readers' FP16 decode.
- Tests: E2M1 ties at 0.25 ... 5.0 times the scale, and the E8M0 cap.

Default arguments compile to the same instructions as before.

Co-authored-by: Cursor <cursoragent@cursor.com>

* [Triton/Gluon] [QSA] Reject partial second planes and mismatched scale planes

- qsa_store_quantized_rows: second_cache, second_scale and second_values go
  together or not at all; a partial set used to be ignored without an error.
- qsa_paged_mqa_logits: compressed_k_scale must have the full shape and dtype
  of the cache's scale plane (qsa_cache_row_layout), not only its first three
  dims. A short FP4 group axis used to pass and the kernel read past each
  scale row.
- qsa_sparse_paged_gqa: v_scale must have the shape of k_scale, and a v_scale
  without an FP8 cache and k_scale is rejected.
- The CUDA-core scorer is one kernel with KEY_FMT (0 BF16, 1 FP8, 2 MXFP4),
  like the MFMA kernels, instead of a separate FP4 kernel with a copied body.
  Every variant compiles to the same instructions as before.
- test_qsa.py: tests for the rejected inputs; randn gets an explicit dtype.

Co-authored-by: Cursor <cursoragent@cursor.com>

* [Triton/Gluon] [QSA] Move QSA tiles to JSON, reuse _mxfp4_e8m0_to_fp32

- The stacked decode scorer's tiles and the FP8 GQA tiles move from Python
  dicts to configs/<arch>/triton/attention/qsa/DEFAULT.json, read with
  resolve_config_dir() and load_config_json(). gfx942 and gfx950 get the
  previous Python values; gfx1250 is seeded byte-identical from gfx950.
  Only the stacked and FP8 GQA paths read the file, so BF16 calls do not
  depend on it on any arch.
- The MXFP4 readers decode the group scale with
  _mxfp4_e8m0_to_fp32(e + 14) instead of a local copy. The bits are the
  same for all 256 exponents.
- The FP8 writer's kernel name includes FP8_MAX_C, which differs per arch.
- Move the CUDA-core comment from the MFMA branch to the CUDA-core launch.
- test_qsa.py: the cap test builds its values on the GPU with an explicit
  dtype, and the other float factories get a dtype too. The narrow
  page-table test uses a real [1, 3] table and no longer claims to catch a
  read of entry 3: such a read has no visible effect, and the test passes on
  the kernels without the fix as well.

Co-authored-by: Cursor <cursoragent@cursor.com>

* [Triton/Gluon] [QSA] Accept FP16 / FP32 caches again, check writer and GQA inputs

Review follow-up (valar-cop):
- Unquantized FP16 and FP32 caches work again in the index scorers and
  sparse GQA, as on main; 4811ba3 accepted only BF16. A scale plane passed
  with an unquantized cache is rejected.
- qsa_store_quantized_rows checks the scale plane's dtype, and raises a
  ValueError instead of view's RuntimeError when the cache or scale plane is
  a slice of a larger page pool.
- qsa_cache_row_layout("fp4", d) needs a power-of-two d >= 32, so the FP4
  scorers raise a ValueError for d = 96 instead of failing to compile.
- qsa_sparse_paged_gqa needs float32 k_scale / v_scale.
- Comments: why page_ok keeps columns < num_columns, the CUDA-core fallback,
  a one-line _tuned_tile docstring. torch.logspace in the test takes a dtype.

The compiled kernels are unchanged.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Read the QSA tile table optionally; test more head_dims

configs/<arch>/triton/attention/qsa exists only for gfx942, gfx950 and
gfx1250, and _tuned_tile read it with required=True, so stacked_decode
and FP8 sparse GQA raised FileNotFoundError on any other arch. Load it with
required=False: without the table stacked_decode scores with the other
kernels and FP8 K/V keep the _kernel_config tiles, as BF16 does.

Tests: run the FP8 GQA case and a stacked_decode call as on gfx1100 (no
QSA config), and parametrize the quantized scoring test over head_dim
(32, 64, 128, 256) and heads (1, 2, 4), which covers FP4 at head_dim 32.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Reuse _fp8_quant_op in the FP8 writer; fp4_utils in the test

The FP8 QSA writer computed the row amax, scale and clamp itself, which is
what _fp8_quant_op in fused_fp8_quant.py does with one group per row; K and
V repeated that code. Call _fp8_quant_op from one _store_fp8_row helper.
Scales are unchanged; at rounding ties a few bytes move one E4M3 code,
because _fp8_quant_op multiplies by 1 / scale instead of FP8_MAX / amax.

The test's MXFP4 dequant reimplemented mxfp4_to_f32 and e8m0_to_f32 from
aiter.utility.fp4_utils, which test_quant_mxfp4.py uses; use them.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Clean up the QSA diff: one test case builder and reference, shared FP4 key-tile decode, one matrix-core launch

- test_qsa: _index_case / _stacked_case and their references become one
  _case and one _reference; the writer reject tests share _planes; the
  narrow page-table test uses them instead of its own reference.
- kernels: the MFMA and stacked scorers share _fp4_key_tile; the CUDA-core
  scorer loads unquantized Q rows with _load_q_row; duplicate comments go.
- wrapper: the stacked and MFMA scorers share one launch; the FP8 and FP4
  writers share their common arguments.

Outputs are bitwise identical to the previous commit.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fold the stacked QSA scorer into the MFMA kernel and share the scale-plane check

- _qsa_paged_mqa_logits_mfma_kernel takes STACK_HEADS; the stacked kernel and
  its repr are gone.
- _load_fp4_keys is the one MXFP4 key decode of the CUDA-core and MFMA scorers.
- _check_scale_plane validates the writer, scorer and GQA scale planes.
- The FP8 writer takes None for the second plane instead of HAS_SECOND.
- bench_qsa: _quantize returns one (cache, scale) pair; _scenario is gone.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Leave existing QSA code as on main where the change does not need it

- The stacked scorer is its own kernel again, so the prefill MFMA kernel only
  gains the KEY_FMT branches; both decode MXFP4 keys with _load_fp4_keys.
- The CUDA-core scorer keeps main's Q and key loads in its unquantized branch.
- FlyDSL dispatch only adds the compressed_k_scale check; main's docstrings,
  launch arguments, bench imports, labels and help text are restored.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Say why _tuned_tile is cached and what "bf16" covers in the QSA docstrings

Co-authored-by: Cursor <cursoragent@cursor.com>

* Keep main's QSA exports and quant comments, and note the stacked page_ok bound

Co-authored-by: Cursor <cursoragent@cursor.com>

* Scope the exactness note of _load_fp4_keys to BF16 / FP32

Co-authored-by: Cursor <cursoragent@cursor.com>

* Share one scoring path across key formats in the QSA kernels

MXFP4 keys are now decoded into a full-width key tile (_load_fp4_keys
interleaves the low and high nibbles), so the CUDA-core, MFMA and stacked
scorers differ by key format only in how the key tile is loaded. main's Q
loads and per-head scoring in the CUDA-core and MFMA kernels are back to
their original text, and _load_q_row is gone.

Also:
- sparse GQA: one QK dot and one PV dot for both cache formats; FP8 only
  adds the key-scale and value-scale multiplies.
- the writers drop `row < num_rows`, always true for the (rows, heads) grid,
  and the num_rows argument.
- qsa_cache_row_layout returns a QSARowLayout NamedTuple.
- bench_qsa.py: bench_select picks logits / select from args.op instead of
  a scoring_only flag.
- test_qsa.py: _store builds its planes with _planes.

PR diff against main: +1793 -93 before, +1701 -61 after; the kernel file
changes 24 of main's lines instead of 56.

The FP4 scores now sum in a different order, so they are not bitwise equal
to the previous head; the other outputs are expected to be.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Read QSARowLayout fields by name in bench_qsa

Use the layout's field names in the benchmark, as the wrapper and the
test do, and keep the reviewed comment on the CUDA-core fallback.
Benchmark output and kernels are unchanged.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants