Repository navigation
[FlyDSL] [QSA] Add FP8 K/V to the sparse prefill - #7
Open
luocheng25 wants to merge 1 commit into
Open
luocheng25 wants to merge 1 commit into
luocheng25 wants to merge 1 commit into
Conversation
luocheng25
force-pushed
the
luocheng/qsa-fp8-v2
branch
from
November 17, 2026 05:10
d9b1e15 to
7a9773f
Compare
luocheng25
force-pushed
the
luocheng/qsa-prefill-followups-v2
branch
from
November 17, 2026 05:10
3911bdc to
03de306
Compare
qsa_sparse_prefill accepts FP8 K/V (E4M3FNUZ, gfx942's dtypes.fp8) with FP32 scales per tensor ([1]) or per token and KV head ([N, HK]; [P, HK] with kv_slots), packed or through kv_slots (new keyword-only k_scale, v_scale and compute): - compute="fp8" (default) runs FP8 MFMA union and direct kernels: the pack launch quantizes Q per (row, head) and packs K/V; probabilities are FP8. - compute="bf16" dequantizes the request tokens to BF16 and runs the BF16 kernels. The FP8 kernels follow the BF16 kernels' rules: a runtime softmax scale, the shared QK/PV/rescale helpers, no non-empty inline asm, and FlyDSL's stable API outside the low-level section of sparse_prefill_common.py, which gains the FP8 primitives. FP8 plans add route calibrations, core/residual union tiles (H12/H6) whose residual blocks the direct kernel merges by log-sum-exp, and, with fewer union tasks than CUs, a split of the longest tasks' key ranges over two CTAs. batch_invariant=True also covers FP8: per-request plans, no split. - The pack stores K token-minor, so each 4-lane quad of a direct K load reads 64 contiguous bytes; FP32-to-BF16 output rounds to nearest even. - Byte offsets past 2 GiB use unsigned shifts (per-token scale records, the residual merge's union LSE), and the FP8 pack buffer is a byte scratch. - With more task entries than one K3 order CTA sorts, K3 places every chunk's runnable entries before all idle ones: the FP8 union kernel ends a CTA's task loop at its first idle entry. - K1 skips the sort of rows whose blocks already ascend. The op test adds 27 FP8 rows: 8 pairwise rows over compute, scale layout, route, input and group, and rows for the byte-span limit, an attention sink, per-token V scale exactness, the FP8-Q reference, large logits, graph replay, the workspace key, entry validation, batch invariance, group 16, BQ42, Q and out past 2 GiB, and K3's chunked task order. Route expectations are asserted on MI308X's 80 CUs only, since plans and routes follow the CU count. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
luocheng25
force-pushed
the
luocheng/qsa-fp8-v2
branch
from
November 17, 2026 06:28
7a9773f to
dca8eb2
Compare
luocheng25
force-pushed
the
luocheng/qsa-prefill-followups-v2
branch
from
November 17, 2026 06:28
03de306 to
41df736
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds FP8 K/V to
aiter.ops.qsa.qsa_sparse_prefillon MI308X (gfx942).torch.float8_e4m3fnuz(gfx942'sdtypes.fp8) with FP32 dequantization scales, either per token and KV head ([N, HK], or[P, HK]withkv_slots) or per tensor ([1]). Both packed K/V and token pools read throughkv_slotsare supported.compute="fp8"(the default) runs FP8 MFMA kernels. Q is quantized per (row, head) in the pack launch, and the probabilities in FP8.compute="bf16"dequantizes the request tokens to BF16 and runs the BF16 kernels.Stacked on #6 (the ROCm#6576 follow-ups), and built to the same rules:
flydsl._mliroutside the low-level section ofsparse_prefill_common.py;batch_invariant=True([FlyDSL] [QSA] Sparse prefill follow-ups: runtime softmax scale, shared helpers, opt-in batch invariance #6's opt-in mode) also covers FP8.Motivation
SGLang serves Qwen3.8-Flash-Next with an FP8 KV cache. In the customer TP2 configuration, the FP8 KV cache holds 89% more tokens (151040 → 285312) with GSM8K unchanged.
aiter's QSA sparse prefill takes BF16 K/V only. This PR gives aiter's consumers (SGLang's aiter attention path, ATOM) FP8 QSA prefill. aiter's Triton QSA path (ROCm#7008) already reads FP8 K/V, but it converts them to BF16 in the kernel, which corresponds to
compute="bf16"here.Changes
qsa_sparse_prefill(..., kv_slots=None, batch_invariant=False, k_scale=None, v_scale=None, compute="fp8"). The new arguments are trailing and keyword-only.float8_e4m3fn), scale dtype, device, shape and contiguity, mixed scale layouts, andcomputevalues.sparse_prefill_union_fp8.py(union) andsparse_prefill_direct_fp8.py(direct, plus the K/V pack with Q quantization).v_cvt_pk_fp8_f32, bare log2 and reciprocal) join the low-level section ofsparse_prefill_common.py.bits + 0x7FFF + lsb, as_pack_bf16does), without asm.[blocks * HK, 1024]byte scratch, so contexts past 2³¹ bytes work.batch_invariant=True, FP8 follows [FlyDSL] [QSA] Sparse prefill follow-ups: runtime softmax scale, shared helpers, opt-in batch invariance #6's per-request rules and does not split, since a split depends on the whole call's load. Core/residual stays: it is a per-tile decision.Performance (if applicable)
GPU time of one
qsa_sparse_prefillcall: the profiler's span from the first kernel's start to the last kernel's end, with the GPU kept busy while the call is queued (host time hidden). MI308X (gfx942, 80 CUs), FlyDSL 0.5.0, one KV head, T = 512; identical CPU-generated inputs. Each tree and mode runs in its own process and repeats the call (warm caches): median of 20 samples per process and of 3 alternating rounds. FP8 K/V are the same BF16 K/V quantized to E4M3FNUZ with scale amax / 240, per tensor or per (token, KV head), withcompute="fp8". Changes are relative to main's BF16.batch_invariant=Truewith FP8 (opt-in), FP8 per tensor relative to the default mode. Short prefills that split their key ranges by default lose the split; batches with a full prefill gain per-request tile rows; short requests below the union planning rows run direct.Testing
Documentation
Dependencies
Breaking Changes
None. The new arguments are trailing, keyword-only and optional, and BF16 calls give the same bits as #6.
PR Self-Review
Reviewed revision
dca8eb2e8956a26d50e7280e935b8c51ab1d0e60Behavior and intent
qsa_sparse_prefillaccepts FP8 K/V with per-token or per-tensor scales, in two compute modes:compute="fp8": FP8 MFMA. The relative output error is a few percent on random data, dominated by the FP8 quantization of Q.compute="bf16": BF16 accuracy, at the cost of a temporary BF16 copy of the request tokens.With
batch_invariant=True, FP8 results are batch-invariant like BF16 ones. BF16 calls give the same bits as #6, and long-context chunks run faster.Validation and unverified scope
MI308X (gfx942, 80 CUs), FlyDSL 0.5.0.
python op_tests/test_flydsl_qsa_sparse_prefill.py: all rows pass, 292checkAllclosecomparisons, none skipped (about 6.5 minutes on a shared host). WithCU_NUM=304(the plans of a 304-CU gfx942 such as MI300X) all rows pass too; route expectations are asserted only on 80 CUs.fp8_batch_invariance_token(H3; a 768-row prefill, a 2048-row chunk after 30000 tokens and a 64-row extend, mixing union tiles and direct rows), every request alone withbatch_invariant=Trueis bitwise equal to its rows in the batch.fp8_hk16_k3_chunks_tokenfails (rows ofoutleft unwritten: 2949120 elements differ between two calls).kv_slots). The 30compute="bf16"outputs are bitwise equal. The 30compute="fp8"outputs differ in 556 of 150M elements (3.7e-6), by at most 2 ulp: the ties the round-to-nearest-even output now rounds differently.batch_invariant=Truecalls interleaved on one stream (4 batch layouts, BF16 and FP8 per token, 3 calls each) give the same bits as each mode alone.fp8_e4m3) and a BF16 draft KV cache:compute="fp8"(default mode), with no fallback and no errors.black --check(26.10.1) andruff check(0.16.0) on the changed files are clean.CU_NUM. They log the tuning warning.PR-specific question
What behavior would each changed op test fail on before this PR, and why is that coverage not already provided by an existing canonical test? (the question the self-review rules select for changes under
op_tests/)op_tests/test_flydsl_qsa_sparse_prefill.py(27 FP8 rows). Before this PR the entry took BF16 K/V only, so every FP8 row fails there, and no BF16 row runs the FP8 pack, the FP8 kernels or the scale paths. Beyond that, each special row targets one failure:fp8_hk16_prefill_24576_token) fails with a signed shift in the residual merge (worst error 21.13).fp8_hk16_k3_chunks_token) fails without the K3 chunk ordering: rows ofoutare left unwritten.routes()asserts the path each row takes.Changed op tests
op_tests/test_flydsl_qsa_sparse_prefill.py: 27 FP8 rows, each run once at its own group.kv_slots) × group: all 56 value pairs in 8 rows.batch_invariant=True: a batch mixing union and direct requests, each bitwise equal to its run alone (by default the first two requests alone split their key ranges);fp8_hk16_k3_chunks_token: H192/HK16, 2620 rows, 4192 entries).