Skip to content

[FlyDSL] [QSA] Add FP8 K/V to the sparse prefill - #7

Open
luocheng25 wants to merge 1 commit into
luocheng/qsa-prefill-followups-v2from
luocheng/qsa-fp8-v2
Open

luocheng25 wants to merge 1 commit into
luocheng/qsa-prefill-followups-v2from
luocheng/qsa-fp8-v2

Conversation

@luocheng25

@luocheng25 luocheng25 commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

Summary

Adds FP8 K/V to aiter.ops.qsa.qsa_sparse_prefill on MI308X (gfx942).

  • Data types. K/V can be torch.float8_e4m3fnuz (gfx942's dtypes.fp8) with FP32 dequantization scales, either per token and KV head ([N, HK], or [P, HK] with kv_slots) or per tensor ([1]). Both packed K/V and token pools read through kv_slots are supported.
  • compute="fp8" (the default) runs FP8 MFMA kernels. Q is quantized per (row, head) in the pack launch, and the probabilities in FP8.
  • compute="bf16" dequantizes the request tokens to BF16 and runs the BF16 kernels.
  • BF16 benefits too. K1 now skips sorting the rows whose blocks already ascend.

Stacked on #6 (the ROCm#6576 follow-ups), and built to the same rules:

Motivation

SGLang serves Qwen3.8-Flash-Next with an FP8 KV cache. In the customer TP2 configuration, the FP8 KV cache holds 89% more tokens (151040 → 285312) with GSM8K unchanged.

aiter's QSA sparse prefill takes BF16 K/V only. This PR gives aiter's consumers (SGLang's aiter attention path, ATOM) FP8 QSA prefill. aiter's Triton QSA path (ROCm#7008) already reads FP8 K/V, but it converts them to BF16 in the kernel, which corresponds to compute="bf16" here.

Changes

  • Entry. qsa_sparse_prefill(..., kv_slots=None, batch_invariant=False, k_scale=None, v_scale=None, compute="fp8"). The new arguments are trailing and keyword-only.
    • FP8 K/V need both scales in the same layout.
    • The FP8 K/V (or pool) byte span must be at most 2³² − 2²⁰.
    • Invalid FP8 arguments are rejected before any launch: dtype (float8_e4m3fn), scale dtype, device, shape and contiguity, mixed scale layouts, and compute values.
    • The workspace cache key includes the FP8 mode and scale layout.
  • Kernels. New sparse_prefill_union_fp8.py (union) and sparse_prefill_direct_fp8.py (direct, plus the K/V pack with Q quantization).
    • Stable FlyDSL operations only: the PV MFMA through the public MFMA atom, the rescale as plain FP32 multiplies, hazard padding left to LLVM. The few FP8 primitives without a stable equivalent (empty-asm holds and fences, DPP row broadcasts, v_cvt_pk_fp8_f32, bare log2 and reciprocal) join the low-level section of sparse_prefill_common.py.
    • The pack stores K token-minor, so each 4-lane quad of a direct K load reads 64 contiguous bytes (FP8 prefills that pack K run 10–13% faster than with a token-major pack).
    • FP32-to-BF16 output conversion rounds to nearest even (bits + 0x7FFF + lsb, as _pack_bf16 does), without asm.
    • Byte offsets past 2 GiB use unsigned shifts (per-token scale records, and the residual merge's union LSE found from the row's out offset), and the FP8 pack buffer is a [blocks * HK, 1024] byte scratch, so contexts past 2³¹ bytes work.
  • Plans and routing.
    • FP8 route calibrations; core/residual union tiles (H12/H6), whose residual blocks the direct kernel merges by log-sum-exp; the pack skips K when every tile runs union whole.
    • With fewer union tasks than CUs (a short prefill leaves CUs idle while its longest causal tiles run), K3 splits the longest tasks' key ranges into two parts on separate CTAs; the part finishing last merges their partial O, m and l.
    • With batch_invariant=True, FP8 follows [FlyDSL] [QSA] Sparse prefill follow-ups: runtime softmax scale, shared helpers, opt-in batch invariance #6's per-request rules and does not split, since a split depends on the whole call's load. Core/residual stays: it is a per-tile decision.
    • With more union task entries than one K3 order CTA sorts (4096), K3 places every chunk's runnable entries before all idle ones, since the FP8 union kernel ends a CTA's task loop at its first idle entry. Plans with one order CTA compile the same K3 as before.
    • K1 skips the sort of rows whose blocks already ascend (complete causal prefixes, the indexer's ascending top-k). This also speeds up BF16 long-context chunks.

Performance (if applicable)

GPU time of one qsa_sparse_prefill call: the profiler's span from the first kernel's start to the last kernel's end, with the GPU kept busy while the call is queued (host time hidden). MI308X (gfx942, 80 CUs), FlyDSL 0.5.0, one KV head, T = 512; identical CPU-generated inputs. Each tree and mode runs in its own process and repeats the call (warm caches): median of 20 samples per process and of 3 alternating rounds. FP8 K/V are the same BF16 K/V quantized to E4M3FNUZ with scale amax / 240, per tensor or per (token, KV head), with compute="fp8". Changes are relative to main's BF16.

Hardware Workload/configuration main BF16 (µs) This PR, BF16 (µs) This PR, FP8 per tensor (µs) This PR, FP8 per token (µs)
MI308X prefill 4096, H12 671.2 672.0 (+0.1%) 395.7 (−41.1%) 435.7 (−35.1%)
MI308X prefill 12000, H12 2467.1 2469.8 (+0.1%) 1509.9 (−38.8%) 1622.0 (−34.3%)
MI308X prefill 12000, H3 1066.3 1063.9 (−0.2%) 697.1 (−34.6%) 774.6 (−27.4%)
MI308X prefill 32768, H6 6567.7 6567.9 (+0.0%) 3560.4 (−45.8%) 3808.5 (−42.0%)
MI308X 1024-row chunk after 131072 tokens, H12 288.9 279.6 (−3.2%) 203.8 (−29.4%) 220.3 (−23.7%)
MI308X 4096-row chunk after 126976 tokens, H6 494.6 473.4 (−4.3%) 352.2 (−28.8%) 382.9 (−22.6%)
MI308X one 256-row extend after 8192 tokens, H3 114.1 113.1 (−0.9%) 89.9 (−21.2%) 94.5 (−17.2%)
MI308X one 256-row extend after 8192 tokens, H12 119.3 116.2 (−2.6%) 81.3 (−31.8%) 90.5 (−24.1%)
MI308X one 128-row extend after 16384 tokens, H6 105.5 105.6 (+0.2%) 78.0 (−26.1%) 80.3 (−23.8%)
MI308X 5 requests (300/1024/0/64/1001 rows), H12 134.0 133.9 (−0.0%) 100.0 (−25.3%) 104.9 (−21.7%)
MI308X 12000-row prefill + 64- and 128-row extends, H3 1385.7 1381.5 (−0.3%) 881.1 (−36.4%) 987.0 (−28.8%)
MI308X two 2048-row prefills, H6 226.5 225.7 (−0.4%) 162.5 (−28.3%) 175.4 (−22.5%)
MI308X 8 × 256-row extends after 2k–16k tokens, H3 142.0 130.7 (−7.9%) 131.8 (−7.2%) 141.9 (−0.1%)
  • FP8 vs BF16. Against this PR's BF16, FP8 takes 20–46% less GPU time per tensor and 16–42% less per token on every layout but the batch of eight short extends, where this PR's BF16 gains from K1's sort skip: there FP8 is +0.8% per tensor and +8.5% per token against this PR's BF16, and −7.2% and −0.1% against main's.
  • BF16 vs main. Outputs are bitwise equal to [FlyDSL] [QSA] Sparse prefill follow-ups: runtime softmax scale, shared helpers, opt-in batch invariance #6. K1's sort skip makes long-context chunks 3–4% and the batch of short extends 8% faster; the other layouts are within −2.6% to +0.2%.
  • Against the previous revision of this PR ([FlyDSL] [QSA] Add FP8 K/V to sparse prefill #5), FP8 in the default mode: the token-minor K pack makes prefills that pack K 10–13% faster (12000 rows at H12, 32768 at H6) and single short extends up to 5% faster; every other layout is within ±1.5%.
  • batch_invariant=True with FP8 (opt-in), FP8 per tensor relative to the default mode. Short prefills that split their key ranges by default lose the split; batches with a full prefill gain per-request tile rows; short requests below the union planning rows run direct.
Hardware Workload/configuration Default (µs) batch_invariant=True (µs) Change
MI308X the single requests above (prefills, chunks, extends) — — −1.5% to +0.8%
MI308X prefill 1024, H3 (split by default) 55.6 62.1 +11.7%
MI308X prefill 1024, H6 (split by default) 56.9 60.6 +6.7%
MI308X prefill 2048, H3 (split by default) 71.8 90.6 +26.1%
MI308X 12000-row prefill + 64- and 128-row extends, H3 881.1 745.5 −15.4%
MI308X two 2048-row prefills, H6 162.5 143.1 −12.0%
MI308X 5 requests (300/1024/0/64/1001 rows), H12 100.0 111.1 +11.1%
MI308X 8 × 256-row extends after 2k–16k tokens, H3 131.8 331.1 +151.3%

Testing

  • Unit tests added or updated
  • Performance benchmarks run
  • Tested on MI250X
  • Tested on MI300X

Documentation

  • Docstrings updated
  • User guide updated
  • Performance guide updated
  • No documentation change is needed, with the reason stated below

Dependencies

  • No new third-party dependencies added
  • New dependencies are listed and justified below

Breaking Changes

None. The new arguments are trailing, keyword-only and optional, and BF16 calls give the same bits as #6.

PR Self-Review

Reviewed revision

dca8eb2e8956a26d50e7280e935b8c51ab1d0e60

Behavior and intent

qsa_sparse_prefill accepts FP8 K/V with per-token or per-tensor scales, in two compute modes:

  • compute="fp8": FP8 MFMA. The relative output error is a few percent on random data, dominated by the FP8 quantization of Q.
  • compute="bf16": BF16 accuracy, at the cost of a temporary BF16 copy of the request tokens.

With batch_invariant=True, FP8 results are batch-invariant like BF16 ones. BF16 calls give the same bits as #6, and long-context chunks run faster.

Validation and unverified scope

MI308X (gfx942, 80 CUs), FlyDSL 0.5.0.

  • python op_tests/test_flydsl_qsa_sparse_prefill.py: all rows pass, 292 checkAllclose comparisons, none skipped (about 6.5 minutes on a shared host). With CU_NUM=304 (the plans of a 304-CU gfx942 such as MI300X) all rows pass too; route expectations are asserted only on 80 CUs.
    • FP8 batch invariance: in fp8_batch_invariance_token (H3; a 768-row prefill, a 2048-row chunk after 30000 tokens and a 64-row extend, mixing union tiles and direct rows), every request alone with batch_invariant=True is bitwise equal to its rows in the batch.
  • Mutation check: with K3's chunked task order switched off at run time, fp8_hk16_k3_chunks_token fails (rows of out left unwritten: 2949120 elements differ between two calls).
  • BF16 against [FlyDSL] [QSA] Sparse prefill follow-ups: runtime softmax scale, shared helpers, opt-in batch invariance #6: 26 outputs (13 layouts, each at the default and a new softmax scale) are bitwise equal.
  • FP8 against the previous revision of this PR ([FlyDSL] [QSA] Add FP8 K/V to sparse prefill #5): 60 outputs (per tensor and per token, compute fp8 and bf16, groups 12/6/3, packed and kv_slots). The 30 compute="bf16" outputs are bitwise equal. The 30 compute="fp8" outputs differ in 556 of 150M elements (3.7e-6), by at most 2 ulp: the ties the round-to-nearest-even output now rounds differently.
  • Mode switching: default and batch_invariant=True calls interleaved on one stream (4 batch layouts, BF16 and FP8 per token, 3 calls each) give the same bits as each mode alone.
  • SGLang end to end. Qwen3.8-Flash-Next in the customer TP2 configuration (1M context, NEXTN speculative decoding, chunked prefill 16384), with an FP8 target KV cache (fp8_e4m3) and a BF16 draft KV cache:
    • GSM8K (1319 questions, 5-shot): 0.9795, accept length 3.132.
    • Logs show that every rank's 12 target QSA layers served FP8 K/V through this entry with compute="fp8" (default mode), with no fallback and no errors.
  • Lint: black --check (26.10.1) and ruff check (0.16.0) on the changed files are clean.
  • Not verified:
    • MI300X and other gfx942 parts (no hardware): only their plans were emulated with CU_NUM. They log the tuning warning.
    • MI250X: the kernels are gfx942-only.
    • SGLang throughput and the in-server reference check at 1M context ran on the previous revision ([FlyDSL] [QSA] Add FP8 K/V to sparse prefill #5); this revision changes only the K pack layout and the output rounding of that FP8 path, and they were not rerun.

PR-specific question

What behavior would each changed op test fail on before this PR, and why is that coverage not already provided by an existing canonical test? (the question the self-review rules select for changes under op_tests/)

  • op_tests/test_flydsl_qsa_sparse_prefill.py (27 FP8 rows). Before this PR the entry took BF16 K/V only, so every FP8 row fails there, and no BF16 row runs the FP8 pack, the FP8 kernels or the scale paths. Beyond that, each special row targets one failure:
    • Q and out past 2 GiB (fp8_hk16_prefill_24576_token) fails with a signed shift in the residual merge (worst error 21.13).
    • More than 4096 union task entries (fp8_hk16_k3_chunks_token) fails without the K3 chunk ordering: rows of out are left unwritten.
    • The byte-span rows fail unless a K/V span at the limit runs and one token over it is rejected.
    • The attention sink over a flat tail fails without the probability bias (E4M3 underflow).
    • The other special rows pin per-token V scale exactness, the FP8-Q reference, large logits, graph replay at an unseen scale, the workspace key, entry validation and the public signature, batch invariance, group 16 and BQ42's 64-bit membership.
    • The 8 pairwise rows cover all 56 value pairs of compute × scale layout × route × input × group. On MI308X's 80 CUs, routes() asserts the path each row takes.

Changed op tests

  • op_tests/test_flydsl_qsa_sparse_prefill.py: 27 FP8 rows, each run once at its own group.
    • Pairwise rows (8). Compute × scale layout × route (union, packed direct; FP8 has no raw direct) × input (packed, kv_slots) × group: all 56 value pairs in 8 rows.
    • Special rows:
      • the byte-span limit, at the limit (packed and pooled) and one token over;
      • an attention sink over a flat tail, which needs the probability bias to avoid E4M3 underflow (union and direct);
      • per-token V scale exactness;
      • the FP8-Q reference with sharp scores;
      • large logits;
      • graph replay with rewritten inputs and an unseen softmax scale;
      • workspace-key separation of FP8 modes;
      • entry validation and the exact public signature;
      • batch invariance with batch_invariant=True: a batch mixing union and direct requests, each bitwise equal to its run alone (by default the first two requests alone split their key ranges);
      • group 16 (direct and union);
      • BQ42 with 64-bit membership;
      • Q and out past 2 GiB, which exercises the residual merge's unsigned offset;
      • more than 4096 union task entries (fp8_hk16_k3_chunks_token: H192/HK16, 2620 rows, 4192 entries).
    • No BF16 row covers any of these, and every FP8 row covers a value pair or a failure no other FP8 row does. Route expectations depend on the CU count, so they are asserted only on 80-CU parts (MI308X); elsewhere the numerical checks still run.

@luocheng25
luocheng25 force-pushed the luocheng/qsa-prefill-followups-v2 branch from 3911bdc to 03de306 Compare November 17, 2026 05:10
qsa_sparse_prefill accepts FP8 K/V (E4M3FNUZ, gfx942's dtypes.fp8) with FP32
scales per tensor ([1]) or per token and KV head ([N, HK]; [P, HK] with
kv_slots), packed or through kv_slots (new keyword-only k_scale, v_scale and
compute):

- compute="fp8" (default) runs FP8 MFMA union and direct kernels: the pack
  launch quantizes Q per (row, head) and packs K/V; probabilities are FP8.
- compute="bf16" dequantizes the request tokens to BF16 and runs the BF16
  kernels.

The FP8 kernels follow the BF16 kernels' rules: a runtime softmax scale, the
shared QK/PV/rescale helpers, no non-empty inline asm, and FlyDSL's stable API
outside the low-level section of sparse_prefill_common.py, which gains the FP8
primitives. FP8 plans add route calibrations, core/residual union tiles
(H12/H6) whose residual blocks the direct kernel merges by log-sum-exp, and,
with fewer union tasks than CUs, a split of the longest tasks' key ranges over
two CTAs. batch_invariant=True also covers FP8: per-request plans, no split.

- The pack stores K token-minor, so each 4-lane quad of a direct K load
  reads 64 contiguous bytes; FP32-to-BF16 output rounds to nearest even.
- Byte offsets past 2 GiB use unsigned shifts (per-token scale records, the
  residual merge's union LSE), and the FP8 pack buffer is a byte scratch.
- With more task entries than one K3 order CTA sorts, K3 places every
  chunk's runnable entries before all idle ones: the FP8 union kernel ends a
  CTA's task loop at its first idle entry.
- K1 skips the sort of rows whose blocks already ascend.

The op test adds 27 FP8 rows: 8 pairwise rows over compute, scale layout,
route, input and group, and rows for the byte-span limit, an attention sink,
per-token V scale exactness, the FP8-Q reference, large logits, graph replay,
the workspace key, entry validation, batch invariance, group 16, BQ42, Q and
out past 2 GiB, and K3's chunked task order. Route expectations are asserted
on MI308X's 80 CUs only, since plans and routes follow the CU count.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@luocheng25
luocheng25 force-pushed the luocheng/qsa-prefill-followups-v2 branch from 03de306 to 41df736 Compare November 17, 2026 06:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant