Repository navigation
[FlyDSL] [QSA] Add FP8 K/V to sparse prefill - #5
Open
luocheng25 wants to merge 8 commits into
Open
luocheng25 wants to merge 8 commits into
luocheng25 wants to merge 8 commits into
Conversation
K1 sorted every row except complete causal prefixes, but the QSA prefill indexer already emits each row's blocks in ascending order. Compare each complete column with the next one (ds_bpermute from the next lane, lane 63 from lane 0's next column) and sort only rows with a descent. The written block_indices rows are the same: an ascending row sorts to itself. Ported from PyHIP 97c0f35a (SGLang-captured 12k selections: K1 70 -> 33 us per layer). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ck and direct kernel Port PyHIP fb7ac47b's FP8 (E4M3FNUZ) K/V path to qsa_sparse_prefill, on the shared helpers, runtime softmax scale and low-level primitives module of the BF16 kernels. Entry (API.md): qsa_sparse_prefill takes keyword-only k_scale, v_scale and compute="fp8" after batch_invariant. K/V of dtypes.fp8 come with FP32 scales, both per tensor [1] or both per token and KV head [N, HK] ([P, HK] with kv_slots); values are K8 * k_scale, V8 * v_scale. compute="fp8" runs FP8 MFMA kernels on Q quantized per (query, head) row (amax / 240); compute="bf16" dequantizes the request tokens to BF16 and runs the BF16 path. The entry rejects BF16 K/V with scales, FP8 K/V without them, scales of the wrong dtype, shape, layout mix, contiguity or device, other FP8 dtypes, an unknown compute, scales overlapping out, and FP8 K/V (or a pool) past 2**32 - 2**20 bytes. Workspaces are keyed by the FP8 scale layout too. FP8 direct kernel (sparse_prefill_direct_fp8): every call packs the context into four-token blocks in 16x16x32 FP8 MFMA operand order (V^T per block, per token a scale record), and the same launch quantizes Q. One wave per (row, KV head); the lazy max keeps FP8 probabilities in range (P_BIAS). The packed K/V are [blocks * HK, 1024] bytes: PyHIP passed them as one flat int32-sized dim and failed from a 2**31-byte packed context (struct.error), half the FP8 limit. Past 2 GiB of packed K/V the direct kernel finds a block's per-token scale record by an unsigned shift of its byte offset (PyHIP's arithmetic shift read zero scales there, so those rows came out zero or wrong); the residual merge's LSE offset shifts unsigned too. Plans: FP8 route calibration (platform(fp8), FP8_ROUTES), key-range split of short prefills' long union tasks (K3 order entries with slot and part), core/residual union tiles (CORE_RATIOS: K2 keeps a tile's core when that costs less, K3b in the FP8 pack launch lists each row's residual blocks for the direct merge). BF16 plans compile K2 and K3 without the core/residual work. batch_invariant=True never splits (a split depends on the whole call). FP8 calls run direct only until the FP8 union kernel lands. Shared helpers: one FP8 MFMA (_mfma_fp8, public MMA atom), the lazy rescale branch (_lazy, under _rescale and the FP8 _lazy_max), BF16 output rounding, V scale terms and Q quantization. New primitives with their reasons: empty-asm hold and compiler fence, DPP row broadcast, v_cvt_pk_fp8_f32, bare v_log_f32 and v_rcp_f32. The FP8 kernels have no non-empty inline asm. Validated on MI308X (gfx942, 80 CUs), FlyDSL 0.5.0: every direct-only FP8 layout of PyHIP's FP8 tests (per tensor and per token, compute fp8 and bf16, packed and kv_slots, attention sink, growing scores, exact uniform) equals PyHIP fb7ac47b bit for bit; FP8 K/V of exactly 2**32 - 2**20 bytes (per token, packed, rows on both sides of 2 GiB; and a pool from 2 GiB to the limit) are within the FP8 error bounds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Port PyHIP fb7ac47b's FP8 (E4M3FNUZ) union kernel and route FP8 union plans to it. A task is 128 (query, head) rows of a union tile over its compact block list: four waves of 32 rows run 32x32x16 FP8 MFMAs, QK of tile t, the lazy max and exponentials of tile t - 1 and PV of tile t - 1 interleaved with the K/V^T LDS pipeline (V^T read from the FP8 pack). Split plans store each key range's O, m and l and the part counting last merges them; core/residual plans write each row's log-sum-exp for the direct merge. PyHIP's non-empty inline asm is replaced by FlyDSL 0.5.0's public API, as prototyped by F0 (outputs bit-identical, faster): PV MFMAs through the public MMA atom (LLVM keeps O in VGPRs instead of asm-pinned AGPRs), the lazy rescale by plain multiplies in the shared _lazy_max, no asm s_nop (LLVM's hazard recognizer places the wait states), and the BF16 output scale by two FP32 multiplies (no v_pk_mul_f32). The softmax scale is a kernel argument (SCALE_LOG2), as in the BF16 kernels. quantize_q quantizes Q as compute="fp8" does, for references. Validated on MI308X (gfx942, 80 CUs), FlyDSL 0.5.0: 116 layouts of PyHIP's FP8 tests (per tensor and per token, compute fp8 and bf16, packed and kv_slots, default and always-planned unions, split and core/residual plans, attention sink, growing scores, exact uniform, sharp scores) equal PyHIP fb7ac47b bit for bit with the same plan decisions; union_fp8 has no scratch and one wave per SIMD (VGPR + AGPR 365-451). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 25 FP8 K/V cases to op_tests/test_flydsl_qsa_sparse_prefill.py for the FP8 port of aiter.ops.qsa.qsa_sparse_prefill (k_scale, v_scale, compute). Each runs once, at its own group. Data and bounds follow PyHIP's FP8 tests: K/V spread by 2**U(-1, 1) per (token, KV head) and quantized with scale amax / 240; compute="fp8" output within FP8_TOL (mean 0.06, worst 0.16, norm-relative per (row, head)) of the reference on the dequantized K/V over sample rows, compute="bf16" bit-identical to the BF16 path on the K/V the caller dequantizes. - Pairwise coverage of compute (fp8/bf16), scales (per tensor/per token), paths (union, packed and raw direct), packed or pooled K/V and groups 12/6/3 in eight cases. Pools hold NaN in their unused rows and per token scales, and must give the packed call's bits. - Group 16 (direct through a pool at HK2 and T = 256; union with BQ8 tiles), HK2 at T = 768, T = 1024 over a 132096-token context, the BQ42 tiles of an H3 full prefill, large logits, and the 12000-token serving prompt, timed against the BF16 path on the dequantized K/V and aiter's Triton FP8 GQA. - An attention sink over a flat tail (FP8 probability underflow), K = 0 with power-of-two V scales and zero Q rows (exact output), and the reference on the kernels' own FP8 Q, also for sharp scores. - The 4 GiB - 1 MiB FP8 byte span: packed K/V at the limit, a pool whose pages start at 2 GiB and end at the limit, and one token past it, which compute="fp8" must reject while compute="bf16" reads the same pool. - Graph replay with Q, K/V, scales, indices and slots rewritten in place, scratch growth and softmax scales captured unwarmed; a workspace key that must hold the K/V dtype, scale layout, pool and batch mode; input errors and the entry's signature; batch invariance with batch_invariant=True. routes() reports the FP8 paths too (key-range split, core/residual tiles, skipped K pack) and the cases assert them; the helpers from latest_workspace to scratch_arena are the only code reading prefill internals. run_torch can dequantize FP8 K/V after its gather and compute sampled rows only, and make_pool builds FP8 pools. On MI308X (gfx942) all 25 cases pass with the FP8 kernels of this series, and the earlier rows pass unchanged. Before those kernels existed, the cases were also run against PyHIP fb7ac47b's attention() behind this entry (at the default softmax scale, with batch_invariant emulated): all passed except the packed limit case, where PyHIP's FP8 pack passes its flat packed K/V with int32 dims and fails from a 2**31-byte packed context (struct.error), which this series fixes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The FP8 direct kernel merges the residual blocks of core/residual union tiles into their rows by log-sum-exp, finding each row's union LSE from the row's out byte offset (_merge_o). Past 2 GiB of out that offset is negative as a signed Int32; the merge must shift it unsigned. No case reached it. Add fp8_hk16_prefill_24576_token to op_tests/test_flydsl_qsa_sparse_prefill.py: a 24576-row prefill at HK16 and group 12 (H192, per token scales), whose Q and out take 2.25 GiB, so rows from 21845 on reach past 2 GiB, and whose union tiles keep cores (77 rows past 2 GiB merge residual blocks on MI308X). - It is checked against the reference on the kernels' FP8 Q (FP8_QUERY_CHECKED: FP8 Q rounding takes one of its 192-head vectors past FP8_TOL's worst bound) on every row past 2 GiB (FP8_LARGE_Q_CASES), the full reference taking 27 GiB and 15 s; the sample rows meet none of the merged rows past 2 GiB. - residual_rows() reads the plan's per-row residual block counts; at least one row past 2 GiB must have merged. - run_perftest rotates inputs only when Q is below 1 GiB too, and kv_bytes() counts FP8 cases' Q three times in the free memory a case needs. On MI308X (gfx942) the case passes in about 15 s (fp8_query error mean 0.021, worst 0.061 over the 2731 rows past 2 GiB). With _merge_o patched back to the signed shift at run time it fails: worst 21.13. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extend op_tests/test_flydsl_qsa_sparse_prefill_guards.py to the FP8
kernels, which keep their own low-level primitives: empty asm holding
values (prim.hold) and fencing loads (prim.compiler_fence), DPP row
broadcasts, v_cvt_pk_fp8_f32, bare log2 and reciprocal, and the FP8 union's
one wave per SIMD.
Three more layouts are compiled with IR dumps, FP8_CASES entries at their own
group: fp8_ragged_4_token (union with core/residual tiles, direct with the
residual merge, per token pack), fp8_kv_slots_ragged_4_tensor (the same
through a slot table, per tensor) and
fp8_kv_slots_topk1024_mixed_ctx_132096_token (union split into two key
ranges, T = 1024). Their union, direct and pack kernels must show:
- no compiler error and finite outputs;
- no scratch; union one wave per SIMD with at most 1 VGPR and 48 SGPR
spills, direct two waves and pack eight, both without spills;
- -packed-fp32-ops, waves_per_eu 1 (union) and 2 (direct), no v_pk_*_f32;
- the baseline number of empty asm blocks: union 10 per tensor and 18 per
token ("v"), direct 3 ("~{memory}");
- union MFMAs all v_mfma_f32_32x32x16_fp8_fp8 and direct ones
v_mfma_f32_16x16x32_fp8_fp8, the baseline number, none writing AGPRs;
- every exp2, log2 and reciprocal lowered to llvm.amdgcn.exp2/log/rcp.f32,
the baseline number of each;
- the baseline number of v_cvt_pk_fp8_f32.
Without compiling, the FP8 modules' s_waitcnt arguments must all be 0 and
none may use LdsQueue (they keep no counted wait); the stable-API source
audit already covers them.
On MI308X (gfx942) all 16 guards pass in about 105 s. Patching the FP8
primitives at run time makes them fail as intended: dropping prim.hold or
prim.compiler_fence, log2 through fx.math.log2, reciprocal by division,
each FP8 conversion done twice, each FP8 MFMA done twice, the target
feature dropped, exp2 without afn, and the union's waves_per_eu set to 2
(LLVM keeps one wave, the attribute check fails). Edited copies of
sparse_prefill_union_fp8.py with a counted lgkmcnt or vmcnt, LdsQueue,
flydsl._mlir or inline asm are reported.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With more task entries than one K3 order CTA sorts (THREADS * 16), K3 sorts the task order in chunks, one order CTA each, and each placed its chunk's idle entries (tiles routed direct, padding) right after its runnable ones. The FP8 union kernel ends a persistent CTA's task loop at its first idle entry, so runnable tasks of later chunks behind it never ran: a 66000-row batch of a random-selection and a shared-selection request (H6, HK2; 4160 entries) left 1000 rows of out unwritten, and with batch_invariant=True its second request no longer matched its run alone. Each order CTA of such a plan now also counts the runnable entries of every chunk (all order CTAs compute the same route threshold) and places its runnable entries after those of the earlier chunks and its idle entries after every runnable entry, so every union CTA still meets all of its runnable tasks first. This is a separate K3 variant (CHUNKED): plans with one order CTA (up to 4096 entries, every split plan) compile the same kernel as before. The BF16 union kernel skips idle entries and was not affected. Plans with one order CTA keep identical K3 code (same ISA), task orders and outputs. The batch above now writes every row, and with batch_invariant=True each of its requests equals its run alone in BF16 and FP8. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
K3 sorts the union task entries in order CTAs of 4096 entries each, and every CTA puts its idle entries (tiles routed direct) at the end of its own chunk. The FP8 union kernel ends its persistent loop at the first idle entry, so with more than 4096 tasks and direct tiles in the first chunk it never runs the tasks of the later chunks, and their rows of out are never written. The BF16 union kernel walks every entry and is not affected. Add fp8_hk16_k3_chunks_token: a random request whose 6 tiles route direct (a 127k-token context) ahead of a shared 2560-row request, at HK16 and group 12 (H192): 262 tiles x 16 KV heads = 4192 tasks. On 88b74a9 the 96 tasks of the second chunk (KV head 15 of the last 96 tiles, 960 rows) never run and the row fails; with K3 ordering every chunk in one CTA it passes. It is checked against the FP8-Q reference on every row, like the other H192 case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds FP8 K/V to
aiter.ops.qsa.qsa_sparse_prefillon MI308X (gfx942), ported from PyHIP's FP8 QSA sparse prefill (fb7ac47b).torch.float8_e4m3fnuzwith FP32 dequantization scales, either per token and KV head ([N, HK], or[P, HK]withkv_slots) or per tensor ([1]). Both packed K/V and token pools read throughkv_slotsare supported.compute="fp8"(the default) runs FP8 MFMA kernels. Q is quantized per (row, head) row in the pack launch, and the probabilities in FP8.compute="bf16"dequantizes the request tokens to BF16 and runs the BF16 kernels.Stacked on #4 (the ROCm#6576 follow-ups), and built to the same rules:
flydsl._mliroutsidesparse_prefill_primitives.py;batch_invariant=Truealso covers FP8.In the default mode, the outputs and plan decisions are bitwise identical to PyHIP's on the same inputs.
The port also fixes three large-offset bugs that the PyHIP version still has (see Changes).
Motivation
SGLang serves Qwen3.8-Flash-Next with an FP8 KV cache, using PyHIP's QSA sparse prefill today. In the customer TP2 configuration, the FP8 KV cache holds 89% more tokens (151040 → 285312) with GSM8K unchanged.
Moving this into aiter gives aiter's consumers (SGLang's aiter attention path, ATOM) FP8 QSA prefill without depending on PyHIP. aiter's Triton QSA path (ROCm#7008) already reads FP8 K/V, but it converts them to BF16 in the kernel, which corresponds to
compute="bf16"here.Changes
qsa_sparse_prefill(..., kv_slots=None, batch_invariant=False, k_scale=None, v_scale=None, compute="fp8"). The new arguments are trailing and keyword-only.float8_e4m3fn), scale dtype, device, shape and contiguity, mixed scale layouts, andcomputevalues.sparse_prefill_union_fp8.py(union) andsparse_prefill_direct_fp8.py(direct, plus the K/V pack with Q quantization).v_mfma_f32_32x32x16_fp8_fp8with its AGPR constraints, now the public MFMA atom;s_nophazard padding, now left to LLVM;v_pk_mul_f32, now two FP32 multiplies.fb7ac47b):struct.error) once the packed context reached 2³¹ bytes. It is now a[blocks * HK, 1024]byte scratch.outwere left unwritten, with no error. For example, a 66000-row H6/HK2 batch left 1000 rows unwritten.batch_invariant=Truewith FP8. The split is off (it depends on the batch's task count). The other rules are as in BF16 ([FlyDSL] [QSA] Sparse prefill follow-ups: runtime softmax scale, stable FlyDSL API, batch_invariant #4). core/residual stays, since it is a per-tile decision.Performance (if applicable)
FP8, PyHIP
fb7ac47bvs this PR. MI308X (gfx942, 80 CUs), FlyDSL 0.5.0. Measured with PyHIP'sbench_attention_fp8.pyprotocol: 10 rotating inputs, 128 samples. Medians of 8 interleaved ABBA process pairs; outputs identical in every pair.BF16 default mode, #4 vs this PR: op test timing, 30 rows, medians of 8 ABBA pairs.
The other BF16 rows are within ±1%, with one explained exception: prefill_2048 H3, whose timing is bimodal at the process level.
batch_invariant=Truewith FP8, relative to the default mode on the same tree:SGLang end to end. Qwen3.8-Flash-Next, the customer TP2 configuration (1M context, NEXTN speculative decoding), FP8 target KV cache (
fp8_e4m3) with a BF16 draft KV cache, MI308X. The two arms differ only in the QSA prefill (PyHIP vs this PR), run A, B, B, A. Throughput is in tok/s.Testing
Documentation
Dependencies
Breaking Changes
None. The new arguments are trailing, keyword-only and optional, and BF16 calls are bitwise unchanged.
PR Self-Review
Reviewed revision
43ce3d6bf292bfae4e21466c902c7d2f852a979dBehavior and intent
qsa_sparse_prefillaccepts FP8 K/V with per-token or per-tensor scales, in two compute modes:compute="fp8": FP8 MFMA. The relative output error is a few percent on random data, and is dominated by the FP8 quantization of Q.compute="bf16": BF16 accuracy, at the cost of a temporary BF16 copy of the packed context.Results and routing match PyHIP
fb7ac47b, except that the two 4 GiB bugs are fixed. BF16 calls give the same bits as before and are faster for long-context chunks.Validation and unverified scope
All results below are on MI308X (gfx942, 80 CUs) with FlyDSL 0.5.0.
fb7ac47b(default mode):kv_slots, including 28 split plans and 36 core/residual plans.batch_invariant=True: 86 FP8 batches and 2 graph cases: every request alone is bitwise equal to its rows in the batch. The 37 BF16 batches still pass.python op_tests/test_flydsl_qsa_sparse_prefill.py: 132 rows, all pass (291checkAllclosecomparisons), none skipped. The cold-cache runtime is 395.2 s, against 276.0 s for the 99 rows on main (+43%).fp8_hk16_prefill_24576_tokenrow (Q and out of 2.25 GiB) checks every row past 2 GiB. With the residual merge's unsigned shift reverted at run time, it fails (worst 21.13); PyHIPfb7ac47bfails it the same way.python op_tests/test_flydsl_qsa_sparse_prefill_guards.py: 16 checks pass, 103.2 s, including the FP8 kernel guards. Run-time patches of the FP8 primitives and edited sources make them fail as intended (9 + 5 mutations).kv_slots.compute="bf16"under graph capture was checked once (replay equals eager, 0 differing elements). It is not in the op test.waves_per_eu=1).v_pk_*_f32, and every exp2 lowers tollvm.amdgcn.exp2.f32.compute=fp8, with no fallback.TEST=1(38 requests up to 999648 tokens; 7878 attention checks per rank against the reference on the kernels' FP8 Q) passes. The per-layout mean error is 0.42% on average (1.06% at most), and the worst row is at most 5.84%, against the 5% / 20% bounds.--enable-deterministic-inference, the adapter passesbatch_invariant=True, and GSM8K (first 200 questions) scores 0.980.SGLANG_USE_1STAGE_ALLREDUCE=0: SGLang's 1-stage all-reduce, which it selects for deterministic mode on AMD, produces garbage with either QSA backend (an SGLang issue, reported separately).PR-specific question
What behavior would each changed op test fail on before this PR, and why is that coverage not already provided by an existing canonical test? (the question the self-review rules select for changes under
op_tests/)op_tests/test_flydsl_qsa_sparse_prefill.py(27 FP8 rows). Before this PR the entry took BF16 K/V only, so every FP8 row fails there, and no BF16 row runs the FP8 pack, the FP8 kernels or the scale paths. Beyond that, each special row targets one failure:fp8_hk16_prefill_24576_token) fails with the residual merge's arithmetic shift (worst 21.13), as PyHIPfb7ac47bdoes.fp8_hk16_k3_chunks_token) fails before the K3 fix: rows ofoutare left unwritten.routes()asserts each row's path.op_tests/test_flydsl_qsa_sparse_prefill_guards.py(FP8 guards) fails on regressions that leave outputs correct, or break them only after a FlyDSL or LLVM upgrade:-packed-fp32-opsandwaves_per_euattributes;9 run-time patches and 5 edited sources made them fail as intended.
Changed op tests
op_tests/test_flydsl_qsa_sparse_prefill.py: 27 FP8 rows.kv_slots) × group: all 56 value pairs covered in 8 rows. This is instead of PyHIP's 82 FP8 test items.batch_invariant=True;fp8_hk16_k3_chunks_token: H192/HK16, 2620 rows, 4192 entries), which fails before the K3 fix above.routes()asserts the path each row takes.op_tests/test_flydsl_qsa_sparse_prefill_guards.py: compile-only guards for the FP8 kernels, in addition to the BF16 ones. They compile three FP8 layouts with IR dumps and check the union, direct and pack kernels:-packed-fp32-ops, thewaves_per_euattributes, and nov_pk_*_f32;v_cvt_pk_fp8_f32, and exp2/log2/rcp lowered to the AMDGPU intrinsics;These failures would not show up as wrong outputs in the op test, or only after a FlyDSL or LLVM upgrade.