Repository navigation
[Triton/Gluon] [QSA] FP8 KV and FP8 / FP4 index caches, stacked-head decode scorer - #7008
Conversation
…decode scorer Bring the QSA operator changes of ROCm/ATOM#2652 into the Triton QSA kernels that #6469 / #6515 ported from ATOM before that PR landed. - qsa_store_quantized_rows: FP8 rows with one FP32 scale per (token, head) = row amax / FP8 max (K and V in one launch), and MXFP4 rows (packed E2M1 pairs, one E8M0 exponent per 32 dims, rounded up so a group never clips). qsa_cache_row_layout describes the stored row. - qsa_sparse_paged_gqa(k_scale=, v_scale=): reads E4M3 K/V tiles, multiplies the key scale into its score column and folds the value scale into the probability before PV; FP8 has its own tile config. - qsa_paged_mqa_logits / qsa_select_paged_tokens(compressed_k_scale=): FP8 (row scale) and MXFP4 compressed index keys in the CUDA-core, prefill MFMA and stacked scorers. FlyDSL stays BF16-only. - stacked_decode=True: decode-sized calls (<= 256 rows) score with an MFMA kernel that stacks 4 rows x 4 index heads into M = 16, so a request's keys are read once per verify instead of once per row. Off by default, so existing BF16 callers keep their kernels. The BF16 path and the existing signatures are unchanged; new arguments are trailing keywords with defaults. Co-authored-by: Cursor <cursoragent@cursor.com>
🏷️ CI GuideRuns automatically on every PR:
Extended tests (opt-in via labels):
One backend per PR: PR title tags & labels: |
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved bounds checks, slot-layout handling, and offline FP8 initialization can cause invalid memory access, incorrect writes, or import failures.
4 open issues
What changed in this PR
This PR extends aiter’s QSA operators with quantized caches and an optional stacked-head decode scorer, preparing ATOM to use aiter instead of its local kernels.
Changes:
- Add FP8 K/V and FP8/MXFP4 index-cache writers and readers.
- Add optional stacked-head decode scoring while preserving the default BF16 path.
- Extend correctness tests and benchmarks for quantized formats and stacked scoring.
| File | Description |
|---|---|
op_tests/triton_tests/attention/test_qsa.py |
Tests quantized writes, scoring, selection, GQA, and stacked dispatch. |
op_tests/op_benchmarks/triton/bench_qsa.py |
Adds format options, logits timing, and stacked-decode workloads. |
aiter/ops/triton/attention/qsa.py |
Adds cache-layout helpers, writer launches, and quantized dispatch. |
aiter/ops/triton/_triton_kernels/attention/qsa.py |
Implements quantized writers, readers, and stacked scoring. |
aiter/ops/qsa.py |
Exposes cache APIs and forwards new options while keeping FlyDSL BF16-only. |
Review effort: Balanced
| # Quantized cache rows: FP8 with one FP32 scale per row, or MXFP4 with one | ||
| # E8M0 exponent for each FP4_GROUP dims (see the kernel module). | ||
| FP8_DTYPE = dtypes.fp8 | ||
| FP8_MAX = float(torch.finfo(FP8_DTYPE).max) |
There was a problem hiding this comment.
Fixed in 381f018. FP8_MAX is now None when dtypes.fp8 is not a floating-point dtype (the uint8 placeholder that aiter uses without a visible GPU), so both QSA modules import on CPU. In that case _cache_format does not treat uint8 as FP8, so uint8 caches stay FP4, and qsa_cache_row_layout("fp8", ...) raises a clear ValueError.
Test: test_qsa_imports_without_a_gpu runs a subprocess with CUDA_VISIBLE_DEVICES="" and HIP_VISIBLE_DEVICES="". It checks that torch.cuda.is_available() is False, imports aiter.ops.qsa and aiter.ops.triton.attention.qsa, and checks FP8_MAX is None and the FP4 layout. On 830d0bb the test fails with the torch.finfo TypeError.
| rows, heads, head_dim = values.shape | ||
| if rows == 0: | ||
| return | ||
| if slots.shape[0] < rows or heads != cache.shape[2]: |
There was a problem hiding this comment.
Fixed in 381f018. qsa_store_quantized_rows now calls _check_vector("slots", slots) before any launch, so slots must be 1-D, int32 / int64 and contiguous for both the FP8 and FP4 writers. The length check (slots.shape[0] >= rows) is unchanged.
Test: test_qsa_writer_rejects_bad_slots_and_planes passes backing[::2], float slots and 2-D slots to both writers and expects ValueError. On 830d0bb the strided slots were accepted.
| second_cache.stride() != cache.stride() | ||
| or second_scale.stride() != scale.stride() |
There was a problem hiding this comment.
Fixed in 381f018. The second FP8 plane must now match the first in cache shape and dtype and in scale shape and dtype, as well as in strides, and second_scale / second_values must be given. Before the second plane is checked, the first scale plane must have the shape [pages, page, heads(, groups)] of its cache, because the same stride problem can let a smaller first scale plane through.
Test: test_qsa_writer_rejects_bad_slots_and_planes builds 1-page planes with the same strides as a 2-page cache (the test asserts the strides are equal). It expects ValueError for a smaller first scale plane, a smaller second cache and a smaller second scale plane. A matching K/V pair still writes both planes identically.
…iter checks - Scorers: the MFMA and stacked kernels mask page-table loads with columns < num_columns as well as the rounded table width, so a narrow table (for example [1, 3] at page size 4) is not read past its end. - Without a visible GPU, aiter.dtypes.fp8 is a uint8 placeholder. FP8_MAX is None there instead of a module-level torch.finfo error, so both QSA modules import on CPU; FP8 layouts raise until a GPU FP8 dtype exists. - qsa_store_quantized_rows rejects non-contiguous, non-integer or multi-dim slots (the writers read slots[row] without a stride), a scale plane whose shape does not match the cache, and a second K/V plane whose cache or scale shape or dtype differs from the first. - Tests for each case: a [1, 3] page table through the CUDA-core, MFMA and stacked scorers in BF16 / FP8 / FP4, bad slots and planes for both writers, and an import in a subprocess with no visible GPU. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@mengfei-jiang @zovonoir A note for the follow-up that switches ATOM to call these kernels from aiter: what to do with the QSA unit tests ATOM got from ROCm/ATOM#2652. ATOM main still runs its own Kernel tests, covered by this PR's new cases. These can be removed in the same commit that removes ATOM's local kernels:
ATOM-side tests, which should stay (adjust the imports if format resolution moves to
It would also help to keep one small end-to-end check in ATOM: the QSA attention path through aiter gives the same output as before the switch, for BF16 and FP8 KV. That way there is no window where the local kernels are still used but no longer tested, or the reverse. |
k50112113
left a comment
There was a problem hiding this comment.
Thanks for the detailed write-up and the parity numbers. One request:
_qsa_store_rows_fp4_kernel writes the same format as aiter's existing MXFP4 quantizer (OCP MXFP4: packed E2M1 + one E8M0 per 32). Its scale rule, ceil(log2(amax / 6)), is also what _mxfp4_quant_op(SCALING_MODE=1) in aiter/ops/triton/_triton_kernels/quant/quant.py already computes. The only real differences are:
E2M1 tie rounding: _e2m1_code rounds ties away from zero (a >= midpoint). _mxfp4_pack_bits rounds to nearest even. These give different bytes at 0.25, 1.25, 2.5 and 5.0. See the table below:
| Scaled value | quant.py (round to nearest even, same as the gfx950 v_cvt_scalef32_pk_fp4 instruction) | sage (ties toward lower magnitude) | PR (ties toward higher magnitude) |
|---|---|---|---|
| 0.25 | 0 | 0 | 0.5 |
| 0.75 | 1.0 | 0.5 | 1.0 |
| 1.25 | 1.0 | 1.0 | 1.5 |
| 1.75 | 2.0 | 1.5 | 2.0 |
| 2.5 | 2.0 | 2.0 | 3.0 |
| 3.5 | 4.0 | 3.0 | 4.0 |
| 5.0 | 4.0 | 4.0 | 6.0 |
E8M0 range: The QSA writer clamps the exponent to [-127, 113] (E8M0 <= 240) for the FP16-bitcast decode. The ceil helper can go up to 254.
All-zero group scale byte. The 1e-30 floor gives E8M0 = 25, while _mxfp4_ceil_scale_from_amax gives 1. Both decode to zero.
Rather than adding a second MXFP4 quantizer with its own _e2m1_code (aiter already has another _e2m1_code in sage_attention_quant.py with different tie rounding), could you extend _mxfp4_quant_op to cover these and call it from QSA? Something like:
New tie-rounding option on _mxfp4_quant_op / _mxfp4_pack_bits: e.g. ROUNDING_MODE: tl.constexpr = 0 (0 = nearest even, the current behavior; 1 = ties away from zero). The default must stay unchanged for existing callers. The gfx950 v_cvt_scalef32_pk_fp4_f32 path only does nearest-even, so mode 1 should static_assert on USE_ASM or fall back to the bit path. Both the normal and denormal branches of _mxfp4_pack_bits need the change, since the denormal branch relies on the FPU's round-to-nearest-even.
Scale cap for ceil mode: e.g. MAX_SCALE_E8M0: tl.constexpr = 254, which QSA sets to 240.
Zero-group byte: either add an option that matches ATOM's byte 25, or confirm that ATOM's parity check can accept a different scale byte for all-zero groups, since the decoded values are identical. I'd prefer the second if the ATOM side allows it.
QSA writer: load the row as a contiguous [1, HEAD_DIM] tile and call _mxfp4_quant_op(..., MXFP4_QUANT_BLOCK_SIZE=FP4_GROUP, SCALING_MODE=1, ROUNDING_MODE=1, MAX_SCALE_E8M0=240) instead of _e2m1_code plus the inline exponent math.
After the change, please rerun the ATOM parity suite for the FP4 writer and FP4 scoring cases. _mxfp4_ceil_scale_from_amax computes the exponent from the FP32 bits, while the current code uses tl.log2, so it's worth confirming they still match bit for bit. Please also test the existing dynamic_mxfp4_quant tests showing that the default path (ROUNDING_MODE=0) is unchanged, however, you don't need to add a test script for dynamic_mxfp4_quant.
Separately: was ATOM's ties-away rounding a deliberate choice? If not, switching both sides to round-to-nearest-even would let QSA use SCALING_MODE=1 as it is, with no new rounding mode. That needs a matching change in ATOM, so I'm fine handling it as a follow-up.
Address review: drop the QSA writer's own E2M1 encoder and inline E8M0 math, and extend aiter's MXFP4 quantizer to cover what QSA needs. - _mxfp4_quant_op / _mxfp4_pack_op / _mxfp4_pack_bits gain ROUNDING_MODE (0 = ties to nearest even, the existing behavior and default; 1 = ties away from zero). Both the normal and the denormal branch of the bit path implement mode 1; USE_ASM with mode 1 is a static_assert, since v_cvt_scalef32_pk_fp4_f32 only rounds to even. - _mxfp4_ceil_scale_from_amax / _mxfp4_quant_op gain MAX_SCALE_E8M0 (default 254 = uncapped, so inf / NaN amax keeps its NaN scale); only SCALING_MODE=1 accepts a cap. - _qsa_store_rows_fp4_kernel loads the row as a [1, HEAD_DIM] tile and calls _mxfp4_quant_op(SCALING_MODE=1, ROUNDING_MODE=1, MAX_SCALE_E8M0=240): the cache bytes of ATOM's QSA writer, with E8M0 kept within the readers' FP16 decode. - Tests: E2M1 ties at 0.25 ... 5.0 times the scale, and the E8M0 cap. Default arguments compile to the same instructions as before. Co-authored-by: Cursor <cursoragent@cursor.com>
READY FOR HUMAN REVIEWPR Self-ReviewRevision Author self-review accepted. Readiness report: Author answers
Show the author's self-review answersBehavior and intentValidation and unverified scopePR-specific questionChanged op testsPR state
Copilot findings
Code and op-test hygiene
Required CI
Shadow mode reports what would block but keeps the status successful. Set |
|
@k50112113 Done in 9be3a42: the FP4 writer now uses
Two new constexprs, added after
The defaults change nothing. On the previous head (381f018) vs this head, 21 outputs are bitwise identical:
The 19 compiled kernels also have the same AMDGCN instructions: 16,336 lines, after removing the debug lines. QSA writer It loads a contiguous All-zero groups The bytes differ, and I accepted that without adding an option:
The same holds for -0.0, which is code 8 here and 0 in ATOM. For these inputs the parity check compares decoded values. Logits and top-k from caches that contain them are bitwise identical to ATOM's. Parity with ATOM main (
Tests
As a mutation check, the tie test fails when the writer passes Perf Same MI355X GPU, 381f018 vs 9be3a42, torch.profiler kernel time, median of 7 alternating rounds:
Your question: was ATOM's ties-away rounding deliberate? No.
So nothing on the ATOM side requires ties away from zero. This PR keeps it only so that the cache bytes stay identical when ATOM switches to aiter. Moving ATOM to nearest even is an ATOM-side follow-up. After that, the QSA writer passes |
|
@k50112113 Your review is addressed in A note on the one red check, Triton Tests (MI35X) / Shard 5: it is a known failure on
We will re-run this PR's Triton tests once #7047 is in. |
|
@valar-cop review this pr |
valar-cop
left a comment
There was a problem hiding this comment.
some changes needed
any perf number on gfx942? and gsm8k for the fp8/fp4 caches?
ci Triton Tests (MI35X) / Shard 5 failed — main is already broken the same way (medium): The same shard fails on main itself in all 7 recent runs, and the failing cases the author cites (test_batched_gemm_afp4wfp4[*-2-256-3584-2112], NaN output) are in a kernel this PR does not touch; the cause is tracked in #7047 (unmasked e8m0 scale load in the last K block when K is not a multiple of BLOCK_SIZE_K).
FAILED job
Triton Tests (MI35X) / Shard 5(Triton Test), runner linux-aiter-mi35x-1-fvzc6-runner-54mhl, log .valar/ci/job_123774296329.log
same job on main: ['2026-10-30T09:11 failure', '2026-10-30T08:12 failure', '2026-10-30T07:24 failure', '2026-10-30T06:16 failure', '2026-10-30T04:36 failure', '2026-10-30T03:48 failure', '2026-10-30T03:06 failure']
next: Rerun the Triton tests after #7047 lands; nothing to fix in this PR. The log artifact was empty, so re-check the shard log on the rerun to confirm the same 8 afp4wfp4 cases. https://github.com/ROCm/aiter/actions/runs/40966863934/job/123774296329
ci Triton Test Results failed — main is already broken the same way (high): This is the gate job that aggregates the shards, so it fails only because Shard 5 failed; on main the same job also failed in the same 7 runs.
FAILED job
Triton Test Results(Triton Test), runner GitHub Actions 1009738695, log .valar/ci/job_123798544428.log
same job on main: ['2026-10-30T09:11 failure', '2026-10-30T08:12 failure', '2026-10-30T07:24 failure', '2026-10-30T06:16 failure', '2026-10-30T04:36 failure', '2026-10-30T03:48 failure', '2026-10-30T03:06 failure']
next: No action; it turns green when Shard 5 does (#7047). https://github.com/ROCm/aiter/actions/runs/40966863934/job/123798544428
| if fmt == _FMT_FP8: | ||
| if cache.shape[3] != head_dim: | ||
| raise ValueError("values and FP8 cache disagree on head_dim") | ||
| second = second_cache is not None |
There was a problem hiding this comment.
partial second plane silently skips V? please reject all-or-none here
There was a problem hiding this comment.
@valar-cop I followed your suggestion on aiter/ops/triton/attention/qsa.py:100: qsa_store_quantized_rows now raises ValueError("pass second_cache, second_scale and second_values together") unless the three second_* tensors are all given or all None. Before, a partial set skipped V without an error. The FP4 writer still rejects any second tensor.
test_qsa_writer_rejects_bad_slots_and_planes now passes the four partial sets (cache only, cache + scale, scale + values, values only) to both writers, expects ValueError and checks that nothing was written. It fails on 9be3a42.
(commit d26518567; test_qsa.py 53 passed on MI355X)
cc @valarLip
There was a problem hiding this comment.
| if key_fmt == _FMT_BF16 and q.dtype != compressed_k_cache.dtype: | ||
| raise ValueError("q and compressed_k_cache must have the same dtype") | ||
| if key_fmt != _FMT_BF16 and ( | ||
| compressed_k_scale.shape[:3] != compressed_k_cache.shape[:3] |
There was a problem hiding this comment.
check the full plane, not just shape[:3]? fp4 tail mismatch reads past the scale row
There was a problem hiding this comment.
@valar-cop I followed your suggestion on aiter/ops/triton/attention/qsa.py:280: the check now compares the whole scale plane with the cache's row layout. compressed_k_scale must have the shape (*compressed_k_cache.shape[:3], *tail) and the dtype from qsa_cache_row_layout (FP8: no tail, float32; MXFP4: head_dim // 32 uint8 exponents), plus the unit-stride checks as before. A short FP4 group axis or a wrong dtype now raises ValueError instead of reading past the scale row. The check runs before the stacked / CUDA-core / MFMA dispatch, so it covers every scorer and qsa_select_paged_tokens.
While there I tightened the same kind of check in qsa_sparse_paged_gqa: v_scale must have the shape of k_scale, and a v_scale without an FP8 cache and k_scale is rejected (it used to be ignored).
New test test_qsa_scorers_reject_scale_planes_that_do_not_match_the_cache: after a valid call, FP8 with an extra scale dim or float64 scales, FP4 with a short group axis or int8 exponents, and GQA with a bad v_scale all raise. It fails on 9be3a42.
(commit d26518567; test_qsa.py 53 passed on MI355X)
cc @valarLip
|
|
||
|
|
||
| @triton.jit(repr=_qsa_paged_mqa_logits_fp4_repr) | ||
| def _qsa_paged_mqa_logits_fp4_kernel( |
There was a problem hiding this comment.
why duplicate the whole body? add KEY_FMT constexpr like the mfma kernel
There was a problem hiding this comment.
@valar-cop I followed your suggestion on aiter/ops/triton/_triton_kernels/attention/qsa.py:219: _qsa_paged_mqa_logits_fp4_kernel is gone. _qsa_paged_mqa_logits_kernel takes KEY_FMT (0 BF16, 1 FP8, 2 MXFP4) and GROUP, like the MFMA kernel; only the Q loads and the key decode branch on it, and the wrapper has one launch for all three formats. The kernel file is 83 lines shorter and the wrapper 26.
The merge does not change the generated code (MI355X, 9be3a42 vs d265185 in one process, same GPU):
- 216 scorer cases (BF16 / FP8 / FP4 x head dim 64 / 128 / 256 x 1-4 heads x page size 4 / 16 x 3 shapes): outputs bitwise identical.
- The same 216 compiled variants have the same AMDGCN instructions (142,815 lines, debug lines removed).
- GPU kernel time over 24 shapes (
torch.profiler, median of 9 alternating rounds): median ratio 1.000 (0.973 .. 1.003). - FP8 / FP4 writer and scorer parity with ATOM main
9df8e849a: still 197 / 197 bitwise identical.
The only cost I measured is on the host: an eager MXFP4 call is about 2 us slower to launch (30.6 -> 32.9 us), because the merged kernel takes four more arguments (two dim strides, BLOCK_D, KEY_FMT). Its GPU time is longer than that, so do_bench with launch overhead stays at 0.999 .. 1.001 for MXFP4, and decode runs in CUDA graphs in ATOM.
(commit d26518567; test_qsa.py 53 passed)
cc @valarLip
| torch.manual_seed(0) | ||
| rows = 2048 | ||
| magnitude = torch.logspace(-4, 4, rows, device="cuda")[:, None, None] | ||
| key = (torch.randn(rows, 2, 256, device="cuda") * magnitude).bfloat16() |
There was a problem hiding this comment.
pass dtype explicitly to randn, like _make_paged_scenario does
There was a problem hiding this comment.
@valar-cop I followed your suggestion on op_tests/triton_tests/attention/test_qsa.py:245: every torch.randn in the file now passes dtype=torch.float32, as _make_paged_scenario does. The casts after them are unchanged, so the test inputs are the same values as before.
(commit d26518567; test_qsa.py 53 passed on MI355X)
cc @valarLip
…e planes - qsa_store_quantized_rows: second_cache, second_scale and second_values go together or not at all; a partial set used to be ignored without an error. - qsa_paged_mqa_logits: compressed_k_scale must have the full shape and dtype of the cache's scale plane (qsa_cache_row_layout), not only its first three dims. A short FP4 group axis used to pass and the kernel read past each scale row. - qsa_sparse_paged_gqa: v_scale must have the shape of k_scale, and a v_scale without an FP8 cache and k_scale is rejected. - The CUDA-core scorer is one kernel with KEY_FMT (0 BF16, 1 FP8, 2 MXFP4), like the MFMA kernels, instead of a separate FP4 kernel with a copied body. Every variant compiles to the same instructions as before. - test_qsa.py: tests for the rejected inputs; randn gets an explicit dtype. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@valar-cop thanks for the review. I pushed
New tests for the rejected inputs fail on 9be3a42 and pass now; Perf on gfx942: I have no access to a gfx942 machine, so I have no perf numbers there. All numbers in this PR are from MI355X (gfx950). On gfx942 only CI's Triton Test runs this PR ( GSM8K for the FP8 / FP4 caches: ATOM does not call aiter's QSA ops yet; it still uses its own in-tree copy from ROCm/ATOM#2652. So I give three pieces of evidence:
I updated the PR description (Test Result and Self-Review) with these numbers. The Triton Test Shard 5 failure ( cc @valarLip @valar-cop review this pr |
valar-cop
left a comment
There was a problem hiding this comment.
not ready yet.. tune tables belong in json configs, not python
any gfx942 perf number? gsm8k? ci still queued on head sha
ci green.
|
|
||
| # (BMT, BN, num_warps, column programs) of the stacked decode scorer for each | ||
| # key format, measured at 36 rows over 9 requests (MTP 3 verify). | ||
| _STACKED_DECODE_TILES = { |
There was a problem hiding this comment.
tuned values should live in configs//.../DEFAULT.json, not inline dicts here..
There was a problem hiding this comment.
@valar-cop I followed your suggestion on aiter/ops/triton/attention/qsa.py:221: the stacked decode tiles and the FP8 GQA tiles are now in configs/<arch>/triton/attention/qsa/DEFAULT.json (stacked_decode.{bf16,fp8,fp4} and gqa_fp8.{decode,prefill}). _tuned_tile(kernel, case) reads them with resolve_config_dir() and load_config_json(), behind functools.lru_cache like the other attention loaders.
- gfx950: the measured values. gfx942: the same values, which QSA used there before (I have no gfx942 to tune on). gfx1250: seeded byte-identical from gfx950, as
configs/AGENTS.mdallows. - Only the stacked and FP8 GQA paths read the file, so BF16 calls on main's path do not need it on any arch.
- The tiles match the old Python tables entry by entry on all three archs. Outputs of 350 writer / scorer / GQA cases are bitwise identical to d265185, and all compiled variants have the same instructions.
- Host time per call is unchanged: median ratio 1.003 for the stacked and GQA decode calls that read the tiles. A first version without the
lru_cacheadded 1-2 us per call (do_bench up to 8% on launch-bound decode shapes), so I added the cache.
I left _SCORING_BLOCK_N, _SCORING_COLUMN_PROGRAMS, the MFMA tile and _kernel_config in Python. They come from #6469, and main's BF16 path reads them on every arch, including archs that have no QSA JSON. Moving them would change that path, so I would do it in a separate PR.
(commit 4811ba315; test_qsa.py 53 passed on MI355X)
cc @valarLip
There was a problem hiding this comment.
lgtm for the json move, and thanks for keeping _kernel_config for main's bf16 path.
one thing: gqa_fp8 on gfx942 is not what 942 used before, the fp8 gqa path is new in this pr, so those are gfx950 tiles copied over.. any 942 number, or is it ci only?
_tuned_tile's docstring is longer than the function, keep it to one line and put the measurements in the pr. and please add the ci:triton-300x label, gfx942 configs changed.
There was a problem hiding this comment.
@valar-cop You are right, and my earlier line "the same values, which QSA used there before" was wrong. The stacked scorer and the FP8 GQA path are both new in this PR, so gfx942 had no earlier values. Its stacked_decode and gqa_fp8 entries are the gfx950 tiles copied over. I have no gfx942 machine, so there are no gfx942 numbers: on gfx942 this is CI only. On d26518567, test_qsa.py passed on the MI300X shard with these tiles.
I kept the gfx942 file rather than dropping it:
- Without it, gfx942 would need a missing-table branch (
load_config_json(..., required=False)plus defaults in Python). - FP8 GQA would then fall back to
_kernel_config._kernel_configwas set up for BF16 and is not measured for FP8 on any arch. - With the copied tiles, gfx942 runs the same configuration that MI300X CI has passed.
If you would rather have gfx942 on a default path, I can drop the file in a follow-up. The PR description now says plainly that the gfx942 tiles are gfx950 copies and were not measured on gfx942.
_tuned_tile's docstring is one line in 4feba25c4. The measured shapes are in the PR description, under "Notes for review: Tiles".
About ci:triton-300x: the PR already has it (added on 10-28). I don't change labels myself; the maintainers decide them.
cc @valarLip
| top = 6.0 * 2.0**113 | ||
| values = torch.zeros(2, 1, 128, device="cuda") | ||
| values[0, 0, :32] = top | ||
| values[0, 0, 32:35] = torch.tensor([4 * top, -4 * top, top / 2]) |
There was a problem hiding this comment.
build these on gpu with explicit dtype, no cpu tensor then copy..
There was a problem hiding this comment.
@valar-cop I followed your suggestion on op_tests/triton_tests/attention/test_qsa.py:320: the two torch.tensor calls in the cap test are now built on the GPU with dtype=torch.float32, and so is the torch.zeros above them. No CPU tensor is copied anymore. I also gave the other float factories in the file (tensor / zeros / ones / empty / full) an explicit dtype. The values are the same, since float32 was the default.
(commit 4811ba315; test_qsa.py 53 passed on MI355X)
cc @valarLip
There was a problem hiding this comment.
LGTM. one left: torch.logspace(-4, 4, rows, device="cuda") at op_tests/triton_tests/attention/test_qsa.py:246 still has no dtype. the rest looks good, thanks.
There was a problem hiding this comment.
@valar-cop Fixed in 4feba25c4: torch.logspace(-4, 4, rows, dtype=torch.float32, device="cuda"). No other float factory in test_qsa.py is left without a dtype.
cc @valarLip
There was a problem hiding this comment.
logspace fixed, checked the rest of the file. one left: sign = torch.randint(...) * 2.0 - 1.0 at op_tests/triton_tests/attention/test_qsa.py:311 still gets its float dtype from the default, add dtype=torch.float32 there? rest lgtm.
There was a problem hiding this comment.
@valar-cop Not changed. sign is not built by a float factory: torch.randint(...) returns int64, and * 2.0 - 1.0 promotes it to the default float dtype (float32). Its values are exactly +1 / -1, so the dtype cannot change the result.
cc @valarLip
There was a problem hiding this comment.
ah right, randint is int64 so the python float just lands it on the default float dtype, and the values are exact +-1. fine as is, no change needed.
|
|
||
|
|
||
| @triton.jit | ||
| def _e8m0_value_hi(e): |
There was a problem hiding this comment.
we should already have _mxfp4_e8m0_to_fp32 in quant.py, reuse it instead of duplicate..
There was a problem hiding this comment.
@valar-cop I followed your suggestion on aiter/ops/triton/_triton_kernels/attention/qsa.py:73: _e8m0_value_hi is removed, and the three MXFP4 readers call _mxfp4_e8m0_to_fp32 from quant.py.
The readers decode E2M1 with an FP16 bitcast that leaves a factor of 2^-14, so they pass e + 14: _mxfp4_e8m0_to_fp32(exponent + 14). Before reusing it, I checked it against the old code:
- Bits: it gives the same bits as the removed
((e + 14) << 23)bitcast for all 256 exponents. The decode only places bits and does no rounding.e + 14 >= 14, so itse == 0branch is never taken, and the compiler drops it. - e = 255 / NaN: called on a raw exponent,
_mxfp4_e8m0_to_fp32(255)gives +inf, not NaN. With+ 14, exponents above 240 give the same out-of-range values as the old helper did. The writer caps the exponent at 240, so no cache holds them. - Outputs and code: the stacked, CUDA-core and MFMA MXFP4 scores are bitwise identical to d265185 (108 cases each), and the compiled instructions are unchanged.
(commit 4811ba315; test_qsa.py 53 passed on MI355X)
cc @valarLip
There was a problem hiding this comment.
lgtm, nice work. the e <= 240 note in the _e2m1_value_lo docstring (aiter/ops/triton/_triton_kernels/attention/qsa.py:70) is the part to keep, so the + 14 can go once the writer stops capping.
There was a problem hiding this comment.
@valar-cop Thanks. The e <= 240 sentence stays in the _e2m1_value_lo docstring.
cc @valarLip
There was a problem hiding this comment.
fine, keep it. it's the only place that spells out the 240 limit, and the writer comment at aiter/ops/triton/_triton_kernels/attention/qsa.py:176 already points here, so no duplicate.
|
|
||
| _qsa_store_rows_fp8_repr = make_kernel_repr( | ||
| "_qsa_store_rows_fp8", | ||
| ["HEAD_DIM", "BLOCK_D", "HAS_SECOND"], |
There was a problem hiding this comment.
FP8_MAX_C differs per arch, add it to the repr key..
There was a problem hiding this comment.
@valar-cop I followed your suggestion on aiter/ops/triton/_triton_kernels/attention/qsa.py:20: FP8_MAX_C is in the FP8 writer's repr key, so the kernel is named e.g. ..._FP8_MAX_C_448_... on gfx950 and ..._FP8_MAX_C_240_... on gfx942 (FNUZ). The FP8 writer is the only kernel that takes FP8_MAX_C. Its compiled instructions are unchanged (12 variants, kernel name removed).
(commit 4811ba315; test_qsa.py 53 passed on MI355X)
cc @valarLip
There was a problem hiding this comment.
ok, that's what i meant. name reads fine (..._FP8_MAX_C_448_...), and both values come from dtypes.fp8 so nothing is hardcoded. nothing else from me on this point.
There was a problem hiding this comment.
@valar-cop Thanks, nothing changes here in 4feba25c4.
cc @valarLip
There was a problem hiding this comment.
ok, nothing to add then. FP8_MAX_C is in the fp8 writer's repr key on head (aiter/ops/triton/_triton_kernels/attention/qsa.py:23) and comes from torch.finfo(FP8_DTYPE).max (aiter/ops/triton/attention/qsa.py:29), so the kernel name shows the per-arch max.
|
|
||
| @pytest.mark.parametrize("fmt", ["bf16", "fp8", "fp4"]) | ||
| @pytest.mark.parametrize("rows", [4, 300]) | ||
| def test_qsa_scorers_stay_inside_a_narrow_page_table(fmt, rows): |
There was a problem hiding this comment.
this test passes on the unfixed kernel too, make the table really width 3 or drop the claim..
There was a problem hiding this comment.
@valar-cop You are right. I reverted the page-table fix from 381f018, so the MFMA and stacked kernels mask the table load only with the rounded width again. The narrow-table test still passed in all 6 variants, both with the old [[2, 0, 3, 1]][:, :3] view and with a real contiguous [1, 3] table.
The test cannot catch that read. An entry read past the table is used only after the 0 <= page < num_pages check, and the scores it produces land in columns >= num_columns, which the kernels never store. So the read has no visible effect. Only a memory fault could show it, and a 4-byte overread inside the allocator's block does not fault.
So I made the table really width 3 and dropped the claim: the comment now says the test checks that the 12 columns of a [1, 3] table, rounded up to width 4, score right. The fix itself stays.
(commit 4811ba315; unfixed kernels: 6 passed, fixed: 6 passed; test_qsa.py 53 passed on MI355X)
cc @valarLip
There was a problem hiding this comment.
ok, fair. keep the columns < num_columns mask anyway, an oob read is still oob even if it does not fault; and add one line above it in aiter/ops/triton/_triton_kernels/attention/qsa.py:469 and :644 saying why, since no test covers it now.
There was a problem hiding this comment.
@valar-cop Kept the columns < num_columns mask. In 4feba25c4 there is one line above page_ok in the MFMA and stacked kernels:
# The width is rounded up; columns < num_columns keeps table reads in bounds.The change is comment-only: all 348 compiled variants have the same AMDGCN instructions as 4811ba315.
cc @valarLip
There was a problem hiding this comment.
ok, reads right now — safe_page only clamps to the rounded-up PAGE_TABLE_WIDTH, so the real-width columns < num_columns term is what keeps the read in bounds. comment matches the code, fine by me.
| num_warps=warps, | ||
| ) | ||
| return logits, visible_groups | ||
| # Decode-sized calls without stacking use the CUDA-core kernel. |
There was a problem hiding this comment.
comment sits above the mfma branch but describes the vector path, move it down..
There was a problem hiding this comment.
@valar-cop I followed your suggestion on aiter/ops/triton/attention/qsa.py:372: the comment "Decode-sized calls without stacking use the CUDA-core kernel." now sits right above the CUDA-core launch, after the MFMA branch.
(commit 4811ba315)
cc @valarLip
There was a problem hiding this comment.
yes, it sits right above the cuda-core launch now.
one nit: that branch is not only decode-sized — prefill-size calls also land there when contiguous is false or head_dim < 16 (aiter/ops/triton/attention/qsa.py:384). maybe fallback calls use the CUDA-core kernel. instead?
There was a problem hiding this comment.
@valar-cop Agreed, it is the fallback for every call the stacked and MFMA branches do not take. In 4feba25c4 the comment reads # Fallback calls use the CUDA-core kernel.
cc @valarLip
There was a problem hiding this comment.
ok, the comment matches the code now (aiter/ops/triton/attention/qsa.py:444), the CUDA-core kernel does take every shape the stacked and MFMA branches skip. resolved from my side.
k50112113
left a comment
There was a problem hiding this comment.
LGTM on my side but please address @valar-cop 's comments thanks.
- The stacked decode scorer's tiles and the FP8 GQA tiles move from Python dicts to configs/<arch>/triton/attention/qsa/DEFAULT.json, read with resolve_config_dir() and load_config_json(). gfx942 and gfx950 get the previous Python values; gfx1250 is seeded byte-identical from gfx950. Only the stacked and FP8 GQA paths read the file, so BF16 calls do not depend on it on any arch. - The MXFP4 readers decode the group scale with _mxfp4_e8m0_to_fp32(e + 14) instead of a local copy. The bits are the same for all 256 exponents. - The FP8 writer's kernel name includes FP8_MAX_C, which differs per arch. - Move the CUDA-core comment from the MFMA branch to the CUDA-core launch. - test_qsa.py: the cap test builds its values on the GPU with an explicit dtype, and the other float factories get a dtype too. The narrow page-table test uses a real [1, 3] table and no longer claims to catch a read of entry 3: such a read has no visible effect, and the test passes on the kernels without the fix as well. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@valar-cop thanks. I pushed
Validation on MI355X (gfx950),
Perf on gfx942: I have no access to a gfx942 machine, so I have no perf numbers there. All numbers in this PR are from MI355X (gfx950). On gfx942 only CI's Triton Test runs this PR. On GSM8K: as in my previous summary, ATOM does not call aiter's QSA ops yet. I ran full GSM8K (1319, 5-shot chat, strict match) on MI355X with ATOM main
stderr is about 0.004. That run used The PR description (Test Result, Notes for review and Self-Review) is updated for cc @valarLip |
valar-cop
left a comment
There was a problem hiding this comment.
some changes needed, mostly input validation on the new writers/readers
any gfx942 perf number? gsm8k? ci not finished on head sha
ci green.
| return | ||
| if slots.shape[0] < rows or heads != cache.shape[2]: | ||
| raise ValueError("slots or heads disagree with the cache") | ||
| _, _, tail, _ = qsa_cache_row_layout(QSA_CACHE_FORMATS[fmt], head_dim) |
There was a problem hiding this comment.
scale dtype is dropped here, but the reader checks it later.. keep sdtype and check it?
There was a problem hiding this comment.
@valar-cop Fixed in 4feba25c4. The writer keeps the scale dtype from qsa_cache_row_layout() and checks it together with the shape. A wrong plane now raises:
ValueError: scale must be the torch.uint8 [pages, page, heads(, groups)] plane of the cache
On 4811ba315, an FP4 cache with an int8 scale plane, or an FP8 cache with an FP64 scale plane, was written without any error. The new test_qsa_writer_rejects_wrong_scale_dtypes_and_pool_slices covers both cases and fails on 4811ba315.
cc @valarLip
There was a problem hiding this comment.
ok, that one's covered. the dtype check at aiter/ops/triton/attention/qsa.py:119 plus test_qsa.py:415 does test both wrong planes, and the pool-slice check right next to it reads shape/stride() once so no cost on the writer.
ci still in progress on the head sha, let's see it go green.
| return | ||
| if slots.shape[0] < rows or heads != cache.shape[2]: | ||
| raise ValueError("slots or heads disagree with the cache") | ||
| _, _, tail, _ = qsa_cache_row_layout(QSA_CACHE_FORMATS[fmt], head_dim) |
There was a problem hiding this comment.
only stride(-1) checked, but view needs contiguous.. a paged pool slice will raise bare RuntimeError
There was a problem hiding this comment.
@valar-cop Fixed in 4feba25c4. A new helper, _merge_slots, checks exactly what the view needs: stride(0) == page_size * stride(1) when both dims are > 1. It runs for the cache and for the scale plane. A slice of a larger pool now raises:
ValueError: cache must hold its pages back to back (stride(0) == page_size * stride(1)); a slice of a larger page pool does not
On 4811ba315, pool[:, :4] raised view's bare RuntimeError.
The FP4 scale plane must also have stride(-1) == 1, because the kernel steps over its groups with unit stride, so that is now checked too.
The check reads shape and stride() once, so writer host time stays the same (numbers in the summary). The same test covers pool slices for the cache and for the scale plane, for FP8 and FP4.
cc @valarLip
There was a problem hiding this comment.
checked: _merge_slots at aiter/ops/triton/attention/qsa.py:73 runs for both planes, and when pages == 1 or page_size == 1 the view drops a size-1 dim, so the skip is right. lgtm on this thread.
ci still queued on 4feba25c4, ping me when the triton shards are green.
| """ | ||
| if q.ndim != 3 or q.shape[1] <= 0 or q.shape[2] <= 0: | ||
| raise ValueError("q must have shape [tokens, heads, head_dim]") | ||
| key_fmt = _cache_format(compressed_k_cache, compressed_k_scale) |
There was a problem hiding this comment.
fp16 cache worked on main, now raises unsupported dtype? why narrow this
There was a problem hiding this comment.
@valar-cop You are right: this PR broke it, and 4feba25c4 fixes it. I ran both versions on MI355X:
- On main (
699d75091), FP16 and FP32 caches work in the CUDA-core and MFMA scorers and in sparse GQA, because main only checksq.dtype == cache.dtype. - On
4811ba315, every FP16 / FP32 call raiseda quantized QSA cache needs its scale plane.
The fix: _cache_format treats every float cache wider than 8 bits as unquantized. That is KEY_FMT 0, which reads the keys in Q's dtype, as main does. The kernels are unchanged. An unquantized cache passed with a scale plane now raises an unquantized QSA cache takes no scale plane, and the writer rejects unquantized caches.
The new test_qsa_takes_unquantized_fp16_and_fp32_caches runs the stacked, CUDA-core and MFMA scorers and sparse GQA with FP16 and FP32 caches, against an FP32 reference. It fails on 4811ba315.
cc @valarLip
There was a problem hiding this comment.
checked the diff — _cache_format now sends fp16/fp32 to KEY_FMT 0 and qsa.py:308 still keeps main's q.dtype == cache.dtype check, so this matches main again. fine on my side.
ci still queued on the head sha, let's wait for triton test to go green.
| raise ValueError("FP8 QSA rows need a GPU with an FP8 dtype") | ||
| return head_dim, FP8_DTYPE, (), torch.float32 | ||
| if fmt == "fp4": | ||
| if head_dim % FP4_GROUP: |
There was a problem hiding this comment.
head_dim % 32 allows 96, but fp4 readers need power of two.. move the check into the layout helper
There was a problem hiding this comment.
@valar-cop Moved in 4feba25c4. qsa_cache_row_layout("fp4", d) now needs a power-of-two d >= 32, because the FP4 kernels index the packed bytes with tl.arange(0, d / 2). The writer's separate check is gone.
The scorer calls the same helper, so FP4 logits with head_dim = 96 now raise ValueError: FP4 rows need a power-of-two head_dim >= 32. On 4811ba315 the helper accepted 96, and the scorer failed with a Triton CompilationError.
The new test_qsa_fp4_rows_need_a_power_of_two_head_dim covers the helper, the writer and the scorer.
cc @valarLip
There was a problem hiding this comment.
ok, one check in qsa_cache_row_layout is the right place, and a ValueError before the launch beats a CompilationError from the tile. test hits helper, writer and scorer, lgtm.
| if ( | ||
| v_cache.dtype != k_cache.dtype | ||
| or v_scale is None | ||
| or k_scale.shape != k_cache.shape[:3] |
There was a problem hiding this comment.
k_scale/v_scale dtype not checked, wrong plane gives silently wrong scores.. add float32 check?
There was a problem hiding this comment.
@valar-cop Added in 4feba25c4: k_scale and v_scale must be float32, beside the shape and stride checks. On 4811ba315, a K/V scale plane passed as its int32 bits was accepted, and GQA returned finite but wrong outputs (max error 3.7e9). An FP16 v_scale was also accepted.
Both cases now raise:
ValueError: FP8 K/V need matching float32 [pages, page, heads] scales
They are in test_qsa_scorers_reject_scale_planes_that_do_not_match_the_cache, which fails on 4811ba315.
cc @valarLip
There was a problem hiding this comment.
checked, yes. k_scale/v_scale shape, stride and float32 checks at aiter/ops/triton/attention/qsa.py:576-580, and the test hits both the int32 bits and the fp16 v_scale (op_tests/triton_tests/attention/test_qsa.py:504-509). one small thing: the scorer takes the scale dtype from qsa_cache_row_layout (qsa.py:311), maybe do the same here instead of the literal torch.float32, so there is one source for the format. LGTM otherwise.
There was a problem hiding this comment.
@valar-cop Changed in 9db2881. The writer, the scorer and GQA now check their scale planes with one _check_scale_plane(name, cache, scale, fmt, head_dim), which takes the shape and dtype from qsa_cache_row_layout. GQA calls it for k_scale and v_scale with the FP8 format, so the float32 is no longer written out by hand there. A non-contiguous head dim now gets its own "contiguous in the head dim" error instead of the scale-plane one.
cc @valarLip
There was a problem hiding this comment.
yep, checked 9db2881: _check_scale_plane takes the dtype from qsa_cache_row_layout (aiter/ops/triton/attention/qsa.py:91) and gqa passes both planes through it, so the literal float32 is gone. separate head-dim contiguity error is fine too. lgtm on this thread, just waiting on the aiter test to go green.
valar-cop
left a comment
There was a problem hiding this comment.
some changes needed
any gfx942 perf number? gsm8k? and ci not green yet
ci green.
| when the row is stored unquantized. | ||
| """ | ||
| if fmt == "bf16": | ||
| return head_dim, torch.bfloat16, None, None |
There was a problem hiding this comment.
fp16/fp32 cache also valid here, but this always returns bf16? maybe rename to 'native' or take the dtype as a param
There was a problem hiding this comment.
@valar-cop Not changed. qsa_cache_row_layout(fmt, head_dim) describes the stored row for one of the QSA_CACHE_FORMATS names. For "bf16" it returns a bfloat16 row, which is right for that name.
FP16 / FP32 caches have no scale plane and use Q's dtype, so they don't go through this helper; the op calls it only for "fp8" / "fp4". So no path gets a wrong dtype from it.
Renaming "bf16" to "native" would change an exported name, the stacked_decode.bf16 JSON key on three archs, the test parameters and bench_qsa.py --index-dtype bf16. That is a lot of churn for a name that is not wrong for the rows it describes.
cc @valarLip
There was a problem hiding this comment.
ok, fair — fp16/fp32 never reach the helper, only the named formats do, so keep bf16. maybe just one half sentence in the docstring that bf16 is the unquantized family (the stacked_decode.bf16 tile is shared by fp16/fp32 too), so nobody asks again. lgtm otherwise.
| and head_dim >= 16 | ||
| ): | ||
| tile = _tuned_tile("stacked_decode", QSA_CACHE_FORMATS[key_fmt]) | ||
| if tile is not None and tile["BMT"] * heads >= 16: |
There was a problem hiding this comment.
tile read happens before the usability gate, so a missing json raises instead of falling back.. can we fold the gate into the dispatch?
There was a problem hiding this comment.
@valar-cop Fixed in 330d7a8, together with the required=True thread. The tile is still read only when a call can use the stacked scorer (at most 256 rows, stacked_decode, contiguous, power-of-two heads and head dim). _tuned_tile now returns None on an arch without the file, so one gate, tile is not None and tile["BMT"] * heads >= 16, covers both cases: a missing table, and too few heads to fill M. In both, the call goes to the other scorers instead of raising. test_qsa_stacked_decode_falls_back_without_a_tile checks that the stacked mode is not launched and that the scores match the reference.
cc @valarLip
There was a problem hiding this comment.
checked 330d7a8: the read is inside the usability gate now (aiter/ops/triton/attention/qsa.py:350-358), _tuned_tile returns None without the file (:250), and the single stacked = tile is not None and ... gate at :360 covers both the missing table and the heads < 16 case. the fallback test pins it (op_tests/triton_tests/attention/test_qsa.py:794). resolved from my side.
| mask=valid[:, None], | ||
| other=127, | ||
| ) | ||
| group_scale = _mxfp4_e8m0_to_fp32(exponent.to(tl.int32) + 14) |
There was a problem hiding this comment.
no clamp to 240 on read side? e=254 gives inf and poisons top-k, please add tl.minimum(exponent, 240)
There was a problem hiding this comment.
@valar-cop Not changed: no writer produces an E8M0 exponent above 240.
- aiter's QSA FP4 writer passes
MAX_SCALE_E8M0=240to_mxfp4_quant_op;test_qsa_fp4_writer_caps_the_scale_for_the_fp16_decodecovers this. - ATOM's writer clamps the exponent to [-127, 113] (
atom/model_ops/qwen4_exp/ops/qsa.py, lines 233-234), which is E8M0 <= 240.
So an exponent above 240 only appears in a cache that something else wrote wrongly. Also, the decode is bitcast((e + 14) << 23). e = 241 gives +inf, but e = 242..255 overflow into the sign bit and give -0 or tiny negative values (e = 254 gives -2^-115, not inf).
tl.minimum(exponent, 240) would turn those into a finite 2^113 scale, which is still a wrong value. It would also differ from ATOM's reader, which does not clamp either, and it adds an instruction to every FP4 decode for inputs that no writer produces. So the reader stays bitwise identical to the decode it replaced.
cc @valarLip
There was a problem hiding this comment.
fair, checked it: the writer cap is at aiter/ops/triton/_triton_kernels/attention/qsa.py:189, the test pins [240, 240] at op_tests/triton_tests/attention/test_qsa.py:338, and :222 is the same bitcast decode. clamping would only turn a corrupt cache's inf into another wrong number, so no change from me.
keep the e <= 240 line in the _e2m1_value_lo docstring (:67) so the cap stays documented, that is all. triton tests are green on head now.
valar-cop
left a comment
There was a problem hiding this comment.
not ready yet.. missing config fallback will break other arches
any perf number? gsm8k? and ci is still queued on head
ci green.
| def _tuned_tile(kernel: str, case: str) -> dict: | ||
| """`kernel[case]` of `configs/<arch>/triton/attention/qsa/DEFAULT.json`.""" | ||
| cfg_dir = resolve_config_dir("attention", "QSA", backend="triton") | ||
| return load_config_json(f"{cfg_dir}/DEFAULT.json")[kernel][case] |
There was a problem hiding this comment.
required=True here, but qsa/ only exists on gfx942/950/1250. other arches will just crash. load it optionally and fall back to _kernel_config?
There was a problem hiding this comment.
@valar-cop Fixed in 330d7a8. _tuned_tile now loads configs/<arch>/triton/attention/qsa/DEFAULT.json with required=False (as pa_decode.py does) and returns None when the file does not exist. Then FP8 sparse GQA keeps the _kernel_config tiles, the ones BF16 uses, and stacked_decode=True calls are scored by the CUDA-core / MFMA scorers instead of raising FileNotFoundError. Only these two paths read the file; the BF16 defaults never did.
Two new tests run as on gfx1100, which has no QSA config: test_qsa_fp8_sparse_gqa_matches_dequantized_kv[*-missing] (decode and prefill) and test_qsa_stacked_decode_falls_back_without_a_tile. On MI355X they fail on 1a417c7 with FileNotFoundError and pass on 330d7a8 (test_qsa.py: 110 passed).
cc @valarLip
There was a problem hiding this comment.
ok, checked 9db2881. _tuned_tile loads with required=False (aiter/ops/triton/attention/qsa.py:250) and both callers gate on tile is not None (:360, :586), so no arch raises; the gfx1100 fixture plus the two new tests cover both fallbacks. resolved from my side.
| _SCORING_COLUMN_PROGRAMS = 64 | ||
|
|
||
|
|
||
| @functools.lru_cache(maxsize=1024) |
There was a problem hiding this comment.
load_config_json already caches, remove this lru_cache..
There was a problem hiding this comment.
@valar-cop Kept. load_config_json caches only reading and parsing the file. Without this cache, every call still runs resolve_config_dir() (get_arch() plus three regex checks), builds the path and does two dict lookups.
When I measured this in round 2, removing the cache cost +0.9 us host time per stacked decode call and +1.2 to 1.9 us per FP8 GQA call. That is +5 to 8% in do_bench on the launch-bound decode shapes. With the cache, the host time ratio is back to a median of 1.003.
pa_prefill_sparse._pinned_prefill_config uses the same pattern: an lru_cache around a config lookup that returns None when there is no entry. Since 330d7a8, this cache also keeps the None on archs without the file.
cc @valarLip
There was a problem hiding this comment.
fair, i forgot resolve_config_dir isn't cached, only load_config_json is... that settles it, keep the cache.
just add half a line in the docstring saying why the outer cache is here (resolve_config_dir cost, not the file read), so nobody strips it later.
|
|
||
|
|
||
| @pytest.mark.parametrize("fmt", ["fp8", "fp4"]) | ||
| @pytest.mark.parametrize( |
There was a problem hiding this comment.
head_dim and heads hardcoded to 128/4 here, the fp4 head_dim=32 case is never tested. parametrize?
There was a problem hiding this comment.
@valar-cop Done in 330d7a8. test_qsa_quantized_index_scores_match_dequantized_keys now runs:
- head dim 32 / 64 / 128 / 256 with 4 index heads;
- 1 and 2 heads at head dim 128;
- FP8 and FP4, in every scorer case (stacked, CUDA-core and prefill MFMA; page size 4 and 16).
FP4 at head dim 32 (one group per row) matches the dequantized reference at rtol = atol = 1e-4 in all three scorers, so the kernels needed no change. With 1 or 2 heads, stacked_decode=True falls back to the CUDA-core scorer (BMT * heads < 16), and the test covers that too. On MI355X, test_qsa.py: 110 passed.
cc @valarLip
There was a problem hiding this comment.
checked the grid at op_tests/triton_tests/attention/test_qsa.py:572-578 .. fp4 dim 32 and the 1/2-head fallback are both covered now, and with BMT=4 the heads 1/2 cases do land on the CUDA-core kernel as you say. lgtm.
only nit: the CUDA-core row is page size 4 only, the two page sizes are in the stacked and MFMA cases. fine by me, no rerun needed.
Pick up #7047 (mask OOB e8m0 scales on the ragged-K path of batched_gemm_afp4wfp4), which fixes the MI35X Shard 5 NaNs in test_batched_gemm_afp4wfp4. Co-authored-by: Cursor <cursoragent@cursor.com>
configs/<arch>/triton/attention/qsa exists only for gfx942, gfx950 and gfx1250, and _tuned_tile read it with required=True, so stacked_decode and FP8 sparse GQA raised FileNotFoundError on any other arch. Load it with required=False: without the table stacked_decode scores with the other kernels and FP8 K/V keep the _kernel_config tiles, as BF16 does. Tests: run the FP8 GQA case and a stacked_decode call as on gfx1100 (no QSA config), and parametrize the quantized scoring test over head_dim (32, 64, 128, 256) and heads (1, 2, 4), which covers FP4 at head_dim 32. Co-authored-by: Cursor <cursoragent@cursor.com>
The FP8 QSA writer computed the row amax, scale and clamp itself, which is what _fp8_quant_op in fused_fp8_quant.py does with one group per row; K and V repeated that code. Call _fp8_quant_op from one _store_fp8_row helper. Scales are unchanged; at rounding ties a few bytes move one E4M3 code, because _fp8_quant_op multiplies by 1 / scale instead of FP8_MAX / amax. The test's MXFP4 dequant reimplemented mxfp4_to_f32 and e8m0_to_f32 from aiter.utility.fp4_utils, which test_quant_mxfp4.py uses; use them. Co-authored-by: Cursor <cursoragent@cursor.com>
…4 key-tile decode, one matrix-core launch - test_qsa: _index_case / _stacked_case and their references become one _case and one _reference; the writer reject tests share _planes; the narrow page-table test uses them instead of its own reference. - kernels: the MFMA and stacked scorers share _fp4_key_tile; the CUDA-core scorer loads unquantized Q rows with _load_q_row; duplicate comments go. - wrapper: the stacked and MFMA scorers share one launch; the FP8 and FP4 writers share their common arguments. Outputs are bitwise identical to the previous commit. Co-authored-by: Cursor <cursoragent@cursor.com>
…plane check - _qsa_paged_mqa_logits_mfma_kernel takes STACK_HEADS; the stacked kernel and its repr are gone. - _load_fp4_keys is the one MXFP4 key decode of the CUDA-core and MFMA scorers. - _check_scale_plane validates the writer, scorer and GQA scale planes. - The FP8 writer takes None for the second plane instead of HAS_SECOND. - bench_qsa: _quantize returns one (cache, scale) pair; _scenario is gone. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Pushed Round 4 (replied inline):
Structure check. Every file in the diff was reviewed for the right place and for reuse. The only new files are the three tile JSONs in Cleanup over the whole PR diff (lines added / removed against main):
Net source lines go from +957 to +770. The removed count grows because the MFMA kernel on main now also carries the stacked mode. The three tile JSONs (+108) are unchanged.
Behavior (MI355X, one GPU, 330d7a8 vs 9db2881 in one process):
The PR description is rewritten to be shorter and to explain why. @k50112113 could you take another look and re-approve if it looks good? cc @valarLip |
- The stacked scorer is its own kernel again, so the prefill MFMA kernel only gains the KEY_FMT branches; both decode MXFP4 keys with _load_fp4_keys. - The CUDA-core scorer keeps main's Q and key loads in its unquantized branch. - FlyDSL dispatch only adds the compressed_k_scale check; main's docstrings, launch arguments, bench imports, labels and help text are restored. Co-authored-by: Cursor <cursoragent@cursor.com>
…rings Co-authored-by: Cursor <cursoragent@cursor.com>
…_ok bound Co-authored-by: Cursor <cursoragent@cursor.com>
|
Pushed Size against main (
Fewer main lines are touched (−135 → −93). More lines are added because the stacked scorer is its own kernel again; all of those lines are new code. Reverted, by file:
valar-cop's two docstring nits on 9db2881 are in (b2c4b78): Also in this push:
Not changed (seen, but not for this PR):
Validation (MI355X, one GPU):
The PR description is updated to match. @k50112113 could you take another look and re-approve if it looks good? @valar-cop your two docstring notes (r4440420203, r4440421005) are in b2c4b78. cc @valarLip |
|
The only red check on The timeout is not caused by this PR:
We have not merged main again because that push would dismiss @k50112113's approval from yesterday. Could a maintainer merge with this gate overridden? If you would rather we merge main and go through a fresh CI run plus a re-approval, tell us and we will do that instead. |
Resolve the aiter/ops/qsa.py conflict by keeping both new exports: qsa_sparse_prefill (#6576) and qsa_store_quantized_rows (this PR). Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
Pushed Conflict. The only conflict was at the end of The merge commit ( Fixes from main this brings in. #7290 (auto-updated split test times) and #7316 (less A8W8 GEMM test overhead). On review-pr self-check (aiter's
Validation.
@k50112113 this push resets your 11-02 approval; could you take another look and re-approve if it looks good? cc @valarLip |
MXFP4 keys are now decoded into a full-width key tile (_load_fp4_keys interleaves the low and high nibbles), so the CUDA-core, MFMA and stacked scorers differ by key format only in how the key tile is loaded. main's Q loads and per-head scoring in the CUDA-core and MFMA kernels are back to their original text, and _load_q_row is gone. Also: - sparse GQA: one QK dot and one PV dot for both cache formats; FP8 only adds the key-scale and value-scale multiplies. - the writers drop `row < num_rows`, always true for the (rows, heads) grid, and the num_rows argument. - qsa_cache_row_layout returns a QSARowLayout NamedTuple. - bench_qsa.py: bench_select picks logits / select from args.op instead of a scoring_only flag. - test_qsa.py: _store builds its planes with _planes. PR diff against main: +1793 -93 before, +1701 -61 after; the kernel file changes 24 of main's lines instead of 56. The FP4 scores now sum in a different order, so they are not bitwise equal to the previous head; the other outputs are expected to be. Co-authored-by: Cursor <cursoragent@cursor.com>
Use the layout's field names in the benchmark, as the wrapper and the test do, and keep the reviewed comment on the CUDA-core fallback. Benchmark output and kernels are unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Pushed What changed and why it is simpler:
Size: PR diff +1793 −93 → +1708 −61. In the kernel file, the main lines this PR touches went from 56 to 24. Validation on MI355X (gfx950), one GPU, on
The PR description is updated (FP4 decode, parity, performance table, Validation). @k50112113 your approval was dismissed by the earlier push; could you take a look at the new head? cc @valarLip |
…decode scorer (#7008) * [Triton/Gluon] [QSA] FP8 KV and FP8 / FP4 index caches, stacked-head decode scorer Bring the QSA operator changes of ROCm/ATOM#2652 into the Triton QSA kernels that #6469 / #6515 ported from ATOM before that PR landed. - qsa_store_quantized_rows: FP8 rows with one FP32 scale per (token, head) = row amax / FP8 max (K and V in one launch), and MXFP4 rows (packed E2M1 pairs, one E8M0 exponent per 32 dims, rounded up so a group never clips). qsa_cache_row_layout describes the stored row. - qsa_sparse_paged_gqa(k_scale=, v_scale=): reads E4M3 K/V tiles, multiplies the key scale into its score column and folds the value scale into the probability before PV; FP8 has its own tile config. - qsa_paged_mqa_logits / qsa_select_paged_tokens(compressed_k_scale=): FP8 (row scale) and MXFP4 compressed index keys in the CUDA-core, prefill MFMA and stacked scorers. FlyDSL stays BF16-only. - stacked_decode=True: decode-sized calls (<= 256 rows) score with an MFMA kernel that stacks 4 rows x 4 index heads into M = 16, so a request's keys are read once per verify instead of once per row. Off by default, so existing BF16 callers keep their kernels. The BF16 path and the existing signatures are unchanged; new arguments are trailing keywords with defaults. Co-authored-by: Cursor <cursoragent@cursor.com> * [Triton/Gluon] [QSA] Address review: page-table bound, CPU import, writer checks - Scorers: the MFMA and stacked kernels mask page-table loads with columns < num_columns as well as the rounded table width, so a narrow table (for example [1, 3] at page size 4) is not read past its end. - Without a visible GPU, aiter.dtypes.fp8 is a uint8 placeholder. FP8_MAX is None there instead of a module-level torch.finfo error, so both QSA modules import on CPU; FP8 layouts raise until a GPU FP8 dtype exists. - qsa_store_quantized_rows rejects non-contiguous, non-integer or multi-dim slots (the writers read slots[row] without a stride), a scale plane whose shape does not match the cache, and a second K/V plane whose cache or scale shape or dtype differs from the first. - Tests for each case: a [1, 3] page table through the CUDA-core, MFMA and stacked scorers in BF16 / FP8 / FP4, bad slots and planes for both writers, and an import in a subprocess with no visible GPU. Co-authored-by: Cursor <cursoragent@cursor.com> * [Triton/Gluon] [QSA] Quantize FP4 index rows with _mxfp4_quant_op Address review: drop the QSA writer's own E2M1 encoder and inline E8M0 math, and extend aiter's MXFP4 quantizer to cover what QSA needs. - _mxfp4_quant_op / _mxfp4_pack_op / _mxfp4_pack_bits gain ROUNDING_MODE (0 = ties to nearest even, the existing behavior and default; 1 = ties away from zero). Both the normal and the denormal branch of the bit path implement mode 1; USE_ASM with mode 1 is a static_assert, since v_cvt_scalef32_pk_fp4_f32 only rounds to even. - _mxfp4_ceil_scale_from_amax / _mxfp4_quant_op gain MAX_SCALE_E8M0 (default 254 = uncapped, so inf / NaN amax keeps its NaN scale); only SCALING_MODE=1 accepts a cap. - _qsa_store_rows_fp4_kernel loads the row as a [1, HEAD_DIM] tile and calls _mxfp4_quant_op(SCALING_MODE=1, ROUNDING_MODE=1, MAX_SCALE_E8M0=240): the cache bytes of ATOM's QSA writer, with E8M0 kept within the readers' FP16 decode. - Tests: E2M1 ties at 0.25 ... 5.0 times the scale, and the E8M0 cap. Default arguments compile to the same instructions as before. Co-authored-by: Cursor <cursoragent@cursor.com> * [Triton/Gluon] [QSA] Reject partial second planes and mismatched scale planes - qsa_store_quantized_rows: second_cache, second_scale and second_values go together or not at all; a partial set used to be ignored without an error. - qsa_paged_mqa_logits: compressed_k_scale must have the full shape and dtype of the cache's scale plane (qsa_cache_row_layout), not only its first three dims. A short FP4 group axis used to pass and the kernel read past each scale row. - qsa_sparse_paged_gqa: v_scale must have the shape of k_scale, and a v_scale without an FP8 cache and k_scale is rejected. - The CUDA-core scorer is one kernel with KEY_FMT (0 BF16, 1 FP8, 2 MXFP4), like the MFMA kernels, instead of a separate FP4 kernel with a copied body. Every variant compiles to the same instructions as before. - test_qsa.py: tests for the rejected inputs; randn gets an explicit dtype. Co-authored-by: Cursor <cursoragent@cursor.com> * [Triton/Gluon] [QSA] Move QSA tiles to JSON, reuse _mxfp4_e8m0_to_fp32 - The stacked decode scorer's tiles and the FP8 GQA tiles move from Python dicts to configs/<arch>/triton/attention/qsa/DEFAULT.json, read with resolve_config_dir() and load_config_json(). gfx942 and gfx950 get the previous Python values; gfx1250 is seeded byte-identical from gfx950. Only the stacked and FP8 GQA paths read the file, so BF16 calls do not depend on it on any arch. - The MXFP4 readers decode the group scale with _mxfp4_e8m0_to_fp32(e + 14) instead of a local copy. The bits are the same for all 256 exponents. - The FP8 writer's kernel name includes FP8_MAX_C, which differs per arch. - Move the CUDA-core comment from the MFMA branch to the CUDA-core launch. - test_qsa.py: the cap test builds its values on the GPU with an explicit dtype, and the other float factories get a dtype too. The narrow page-table test uses a real [1, 3] table and no longer claims to catch a read of entry 3: such a read has no visible effect, and the test passes on the kernels without the fix as well. Co-authored-by: Cursor <cursoragent@cursor.com> * [Triton/Gluon] [QSA] Accept FP16 / FP32 caches again, check writer and GQA inputs Review follow-up (valar-cop): - Unquantized FP16 and FP32 caches work again in the index scorers and sparse GQA, as on main; 4811ba3 accepted only BF16. A scale plane passed with an unquantized cache is rejected. - qsa_store_quantized_rows checks the scale plane's dtype, and raises a ValueError instead of view's RuntimeError when the cache or scale plane is a slice of a larger page pool. - qsa_cache_row_layout("fp4", d) needs a power-of-two d >= 32, so the FP4 scorers raise a ValueError for d = 96 instead of failing to compile. - qsa_sparse_paged_gqa needs float32 k_scale / v_scale. - Comments: why page_ok keeps columns < num_columns, the CUDA-core fallback, a one-line _tuned_tile docstring. torch.logspace in the test takes a dtype. The compiled kernels are unchanged. Co-authored-by: Cursor <cursoragent@cursor.com> * Read the QSA tile table optionally; test more head_dims configs/<arch>/triton/attention/qsa exists only for gfx942, gfx950 and gfx1250, and _tuned_tile read it with required=True, so stacked_decode and FP8 sparse GQA raised FileNotFoundError on any other arch. Load it with required=False: without the table stacked_decode scores with the other kernels and FP8 K/V keep the _kernel_config tiles, as BF16 does. Tests: run the FP8 GQA case and a stacked_decode call as on gfx1100 (no QSA config), and parametrize the quantized scoring test over head_dim (32, 64, 128, 256) and heads (1, 2, 4), which covers FP4 at head_dim 32. Co-authored-by: Cursor <cursoragent@cursor.com> * Reuse _fp8_quant_op in the FP8 writer; fp4_utils in the test The FP8 QSA writer computed the row amax, scale and clamp itself, which is what _fp8_quant_op in fused_fp8_quant.py does with one group per row; K and V repeated that code. Call _fp8_quant_op from one _store_fp8_row helper. Scales are unchanged; at rounding ties a few bytes move one E4M3 code, because _fp8_quant_op multiplies by 1 / scale instead of FP8_MAX / amax. The test's MXFP4 dequant reimplemented mxfp4_to_f32 and e8m0_to_f32 from aiter.utility.fp4_utils, which test_quant_mxfp4.py uses; use them. Co-authored-by: Cursor <cursoragent@cursor.com> * Clean up the QSA diff: one test case builder and reference, shared FP4 key-tile decode, one matrix-core launch - test_qsa: _index_case / _stacked_case and their references become one _case and one _reference; the writer reject tests share _planes; the narrow page-table test uses them instead of its own reference. - kernels: the MFMA and stacked scorers share _fp4_key_tile; the CUDA-core scorer loads unquantized Q rows with _load_q_row; duplicate comments go. - wrapper: the stacked and MFMA scorers share one launch; the FP8 and FP4 writers share their common arguments. Outputs are bitwise identical to the previous commit. Co-authored-by: Cursor <cursoragent@cursor.com> * Fold the stacked QSA scorer into the MFMA kernel and share the scale-plane check - _qsa_paged_mqa_logits_mfma_kernel takes STACK_HEADS; the stacked kernel and its repr are gone. - _load_fp4_keys is the one MXFP4 key decode of the CUDA-core and MFMA scorers. - _check_scale_plane validates the writer, scorer and GQA scale planes. - The FP8 writer takes None for the second plane instead of HAS_SECOND. - bench_qsa: _quantize returns one (cache, scale) pair; _scenario is gone. Co-authored-by: Cursor <cursoragent@cursor.com> * Leave existing QSA code as on main where the change does not need it - The stacked scorer is its own kernel again, so the prefill MFMA kernel only gains the KEY_FMT branches; both decode MXFP4 keys with _load_fp4_keys. - The CUDA-core scorer keeps main's Q and key loads in its unquantized branch. - FlyDSL dispatch only adds the compressed_k_scale check; main's docstrings, launch arguments, bench imports, labels and help text are restored. Co-authored-by: Cursor <cursoragent@cursor.com> * Say why _tuned_tile is cached and what "bf16" covers in the QSA docstrings Co-authored-by: Cursor <cursoragent@cursor.com> * Keep main's QSA exports and quant comments, and note the stacked page_ok bound Co-authored-by: Cursor <cursoragent@cursor.com> * Scope the exactness note of _load_fp4_keys to BF16 / FP32 Co-authored-by: Cursor <cursoragent@cursor.com> * Share one scoring path across key formats in the QSA kernels MXFP4 keys are now decoded into a full-width key tile (_load_fp4_keys interleaves the low and high nibbles), so the CUDA-core, MFMA and stacked scorers differ by key format only in how the key tile is loaded. main's Q loads and per-head scoring in the CUDA-core and MFMA kernels are back to their original text, and _load_q_row is gone. Also: - sparse GQA: one QK dot and one PV dot for both cache formats; FP8 only adds the key-scale and value-scale multiplies. - the writers drop `row < num_rows`, always true for the (rows, heads) grid, and the num_rows argument. - qsa_cache_row_layout returns a QSARowLayout NamedTuple. - bench_qsa.py: bench_select picks logits / select from args.op instead of a scoring_only flag. - test_qsa.py: _store builds its planes with _planes. PR diff against main: +1793 -93 before, +1701 -61 after; the kernel file changes 24 of main's lines instead of 56. The FP4 scores now sum in a different order, so they are not bitwise equal to the previous head; the other outputs are expected to be. Co-authored-by: Cursor <cursoragent@cursor.com> * Read QSARowLayout fields by name in bench_qsa Use the layout's field names in the benchmark, as the wrapper and the test do, and keep the reviewed comment on the CUDA-core fallback. Benchmark output and kernels are unchanged. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>

Motivation
#6469 (Triton) and #6515 (FlyDSL) ported the QSA kernels from ROCm/ATOM before ROCm/ATOM#2652 landed. ROCm/ATOM#2652 adds three things to those operators that aiter does not have yet:
This PR adds them to the files that #6469 created, so that ATOM can call aiter for the QSA kernels instead of keeping its own copy. cc @mengfei-jiang @zovonoir (the follow-up ATOM PR) @valarLip
Technical Details
The source is
atom/model_ops/qwen4_exp/ops/qsa.pyat ROCm/ATOM#2652 (squash32a355f0f). Every new parameter defaults to the current behavior. Unquantized BF16 / FP16 / FP32 caches work as on main.Cache writes.
qsa_store_quantized_rows(cache, scale, slots, values, second_cache=None, second_scale=None, second_values=None)writes FP8 or MXFP4 rows and skips negative slots. The writer lives in aiter because it defines the format that the readers decode, so the tests can check the two sides against each other.fused_fp8_quant._fp8_quant_op.quant._mxfp4_quant_op. For that, the quantizer gets two constexprs,ROUNDING_MODEandMAX_SCALE_E8M0:ROUNDING_MODE=1rounds ties away from zero. ATOM writes its cache bytes that way.MAX_SCALE_E8M0=240caps the exponent. The readers' FP16-bitcast decode needs E8M0 <= 240._fp8_quant_opas it is.qsa_cache_row_layout(fmt, head_dim)describes a stored row (value dim and dtype, scale shape and dtype). It is exported fromaiter.ops.qsawithFP8_DTYPE,FP8_MAXandFP4_GROUP.Index scoring.
qsa_paged_mqa_logitsandqsa_select_paged_tokenstakecompressed_k_scale. The CUDA-core, MFMA and stacked scorers take aKEY_FMTconstexpr:[rows, head_dim]key tile (_load_fp4_keys), so all formats share one scoring path.FlyDSL stays BF16-only, so FP8 / FP4 prefill uses the Triton MFMA scorer, as ATOM does.
Stacked decode scorer.
stacked_decode=Truescores calls with at most 256 rows with_qsa_paged_mqa_logits_stacked_kernel. It stacks rows x index heads into the M of one MFMA, so each request's keys are read once per verify step instead of once per row. In ATOM it is on by default (env var); here it is an argument and off by default, so existing callers keep their kernels. It is a separate kernel rather than a mode of_qsa_paged_mqa_logits_mfma_kernel, because a mode would rewrite the body of main's prefill kernel; the MFMA kernel only gains theKEY_FMTbranches.FP8 sparse GQA.
qsa_sparse_paged_gqatakesk_scale/v_scale. The key scale multiplies the score column, and the value scale multiplies the probability before PV.Tiles. The stacked scorer and FP8 GQA tiles were measured in ATOM. They are in
configs/{gfx942,gfx950,gfx1250}/triton/attention/qsa/DEFAULT.json; the gfx942 and gfx1250 files are copies of gfx950. The file is optional: on an arch without it,stacked_decodecalls use the other scorers and FP8 GQA uses_kernel_config. The gfx942 copy is an exception to the gfx950 → gfx1250 seed list inconfigs/AGENTS.md: it is kept so that MI300X CI runs the stacked scorer and FP8 GQA, and it should be replaced once it is measured on gfx942.Input checks. The following raise
ValueErrorinstead of being ignored or read out of bounds:second_*set;v_scale;Changed files:
aiter/ops/qsa.py(exports and pass-through),aiter/ops/triton/attention/qsa.py(wrappers),aiter/ops/triton/_triton_kernels/attention/qsa.py(kernels),aiter/ops/triton/_triton_kernels/quant/quant.py(the two constexprs), the three tile JSON files,op_tests/triton_tests/attention/test_qsa.pyandop_tests/op_benchmarks/triton/bench_qsa.py.Test Plan
On MI355X (gfx950), container
rocm/atom-dev:nightly_202610221638(triton 3.8.0, torch 2.10.0+rocm7.2.4), one GPU:pytest op_tests/triton_tests/attention/test_qsa.pyandop_tests/triton_tests/quant/test_quant_mxfp4.py.9df8e849a: the same inputs through ATOM's ops and through aiter, compared bit for bit.dynamic_mxfp4_quant/_mxfp4_quant_opwith default arguments (outputs and AMDGCN instructions).bench_qsa.pyon main vs the branch.Qwen3.8-Flash-Next-Quark-MXFP4, TP1, MTP-3, FP8 KV cache with an FP8 or FP4 index cache.Test Result
test_qsa.py: 110 passed (main: 11).test_quant_mxfp4.py: 79 passed, 54 skipped, as on main._fp8_quant_opmultiplies by1 / scaleand ATOM byFP8_MAX / amax.GSM8K (stderr about 0.004):
Performance (us, median of 9 alternating rounds, same inputs, page size 16):
Over all 93 cases the median aiter / ATOM ratio is 0.998 (0.476–1.059). FP4 scoring without
stacked_decodeis faster than ATOM (0.48x at 9x4, 0.70–0.73x for the 64k prefill), because the decoded key tile is scored like a BF16 / FP8 tile instead of as two nibble planes. The other cases with more than 100 us of GPU time are 0.973–1.014. Withbench_qsa.py --op logits --index-dtype fp4|fp8 [--stacked-decode](eagerdo_bench, launch cost included),stacked_decodemakes decode scoring 1.18–1.30x faster at 9x4, 32x3 and 128x1 for FP4 and 1.26–1.34x for FP8, and 0.90–1.10x at 16–64 single-token rows, where both take 13–17 us of mostly launch cost. BF16 GPU time is the same as main (10 cases, median 0.997, 0.987–1.013). The eager host cost is 1–2 us higher per scorer or GQA call and 3.1–3.6 us higher forqsa_select_paged_tokens(gfx950, back to back against main, 1x1 to 128x1 rows, 4096 context), because of the format check, the tile lookup and the extra kernel arguments; the scorer and GQA GPU time under CUDA graphs is the same as main. ATOM's wrapper does the same work and runs decode in CUDA graphs.Not changed
KEY_FMTbranches and thecolumns < num_columnsbound onpage_ok(the out-of-bounds table read valar-cop asked to fix in both kernels). The stacked scorer stays a separate kernel, because merging it would rewrite main's kernel body.NUM_HEADS <= 4limit.allow_flydslbehave as on main, and FP8 / MXFP4 keys andstacked_decodeuse Triton only._SCORING_BLOCK_N,_SCORING_COLUMN_PROGRAMS, the MFMA 16 / 128 tile and_kernel_configstay in Python: main's BF16 path reads them on arches without the QSA JSON too. Moving them into the JSON would be a separate PR.head_dimis a power of two; FP8 keys with such a head dim (e.g. 96) inherit it. Reproduced on gfx950: a call with more than 256 rows and head_dim 96 fails to compile (arange's range must be a power of 2), for BF16 on main too. A fix would change main's BF16 dispatch, so it is left for a follow-up.Submission Checklist
PR Self-Review
Reviewed revision
dab71f0
Behavior and intent
The QSA ops gain FP8 K/V caches, FP8 / MXFP4 index-key caches (writers and scorers) and a stacked-head decode scorer, ported from ROCm/ATOM#2652, so ATOM can call aiter instead of keeping its own copy. Every new parameter defaults to the existing behavior, and unquantized BF16 / FP16 / FP32 caches work as on main.
_mxfp4_quant_opgetsROUNDING_MODEandMAX_SCALE_E8M0so the FP4 writer can reuse it and still write ATOM's cache bytes; the defaults leave existing callers unchanged. Inputs that would be ignored or read out of bounds raiseValueError. The tile JSON is optional per arch.Validation and unverified scope
On MI355X (gfx950), one GPU:
test_qsa.py110 passed,test_quant_mxfp4.py79 passed, 54 skipped. Parity with ATOM main is 174 / 197 bitwise: the three FP8 writer cases are at most one E4M3 code apart with the same scales (each an exact tie between two codes with the same error), and the 20 FP4 logits cases are at most 1.4e-6 apart because of the summation order; every top-k selection case picks the same tokens. BF16 outputs and the MXFP4 default path are bitwise identical to main, and the MXFP4 default path has the same instructions. GSM8K with ATOM calling these ops matches ATOM's own ops (table above). black 26.10.1 and ruff 0.16.0 pass on the changed files. Each refactor during review was checked old vs new head in one process: outputs bit for bit, the AMDGCN instructions of every compiled variant, and GPU time.The last revisions (b2fb752..dab71f0) share one scoring path across key formats:
_load_fp4_keysreturns the whole decoded key tile, so the even / odd Q loads and the second scoring loop are gone, and FP8 GQA handles both scales in oneKV_FP8branch (PR diff +1793 −93 → +1708 −61). Againstb2fb752dc, 265 / 350 A/B outputs are bitwise identical and the other 85 are FP4 scorer outputs (summation order, covered by the tests above); the BF16 / FP8 scorer and GQA variants have the same instructions, the 16 writer variants differ only because the unusednum_rowsargument is gone (their outputs are bitwise identical), and the median GPU time ratio is 0.997. FP4 GPU time drops to 0.26–0.29x in the CUDA-core scorer, 0.71–0.74x in the MFMA scorer and 0.88–0.93x in the stacked scorer. Against main, the BF16 kernels differ only in kernel-argument offsets (the new scale strides) and the register allocation that follows; BF16 GPU time is the same as main (median 0.997). The GPU results above were measured ona567922;dab71f0only changesbench_qsa.py(layout fields by name, same output, rerun on gfx950) and a comment in the wrapper, so they were not rerun. The branch merges main47b4e0869to resolve the__all__conflict with #6576.Not verified: on gfx942 only CI's Triton tests ran. The gfx942 tiles are gfx950 copies and were not measured there, and parity, perf and GSM8K were not run on gfx942 (no gfx942 machine available to me). The GSM8K run uses a local ATOM copy that calls these ops; switching ATOM itself is the follow-up ATOM PR.
PR-specific question
Before this PR the new cases in
test_qsa.pyfail because the APIs they call do not exist (qsa_store_quantized_rows,qsa_cache_row_layout,compressed_k_scale,k_scale/v_scale,stacked_decode). The scoring cases compare against dequantized keys, so they catch a reader that decodes FP8 / MXFP4 rows wrongly, and the stacked cases catch a wrong row / head mapping. The FP4 writer cases fail if ties round to nearest even or the exponent is not capped at 240. The reject cases fail if a partial second plane, a mismatched scale plane or a pool slice is accepted, and the missing-config cases fail if an arch without the JSON raises.test_quant_mxfp4.pycovers onlydynamic_mxfp4_quantwith default arguments, and no other test covers QSA.Changed op tests
op_tests/triton_tests/attention/test_qsa.py: the only QSA test file. The new cases cover:max_columns;None of this is tested elsewhere.
op_tests/op_benchmarks/triton/bench_qsa.py: the QSA benchmark, not a test. It gains--op logits,--index-dtype,--kv-dtype,--stacked-decodeand the MTP-3 verify shape, and counts each format's bytes.