Skip to content

[FlyDSL] [QSA] Add FP8 K/V to sparse prefill - #5

Open
luocheng25 wants to merge 8 commits into
luocheng/qsa-prefill-followupsfrom
luocheng/qsa-fp8
Open

luocheng25 wants to merge 8 commits into
luocheng/qsa-prefill-followupsfrom
luocheng/qsa-fp8

Conversation

@luocheng25

Copy link
Copy Markdown
Owner

Summary

Adds FP8 K/V to aiter.ops.qsa.qsa_sparse_prefill on MI308X (gfx942), ported from PyHIP's FP8 QSA sparse prefill (fb7ac47b).

  • Data types. K/V can be torch.float8_e4m3fnuz with FP32 dequantization scales, either per token and KV head ([N, HK], or [P, HK] with kv_slots) or per tensor ([1]). Both packed K/V and token pools read through kv_slots are supported.
  • compute="fp8" (the default) runs FP8 MFMA kernels. Q is quantized per (row, head) row in the pack launch, and the probabilities in FP8.
  • compute="bf16" dequantizes the request tokens to BF16 and runs the BF16 kernels.
  • BF16 benefits too. K1 now skips sorting the rows whose blocks already ascend.

Stacked on #4 (the ROCm#6576 follow-ups), and built to the same rules:

  • the softmax scale is a runtime argument;
  • the FP8 kernels reuse the shared QK/PV/rescale helpers;
  • no non-empty inline asm and no flydsl._mlir outside sparse_prefill_primitives.py;
  • batch_invariant=True also covers FP8.

In the default mode, the outputs and plan decisions are bitwise identical to PyHIP's on the same inputs.

The port also fixes three large-offset bugs that the PyHIP version still has (see Changes).

Motivation

SGLang serves Qwen3.8-Flash-Next with an FP8 KV cache, using PyHIP's QSA sparse prefill today. In the customer TP2 configuration, the FP8 KV cache holds 89% more tokens (151040 → 285312) with GSM8K unchanged.

Moving this into aiter gives aiter's consumers (SGLang's aiter attention path, ATOM) FP8 QSA prefill without depending on PyHIP. aiter's Triton QSA path (ROCm#7008) already reads FP8 K/V, but it converts them to BF16 in the kernel, which corresponds to compute="bf16" here.

Changes

  • Entry. qsa_sparse_prefill(..., kv_slots=None, batch_invariant=False, k_scale=None, v_scale=None, compute="fp8"). The new arguments are trailing and keyword-only.
    • FP8 K/V need both scales in the same layout.
    • The FP8 K/V (or pool) byte span must be at most 2³² − 2²⁰.
    • Invalid FP8 arguments are rejected before any launch: dtype (float8_e4m3fn), scale dtype, device, shape and contiguity, mixed scale layouts, and compute values.
    • The workspace cache key includes the FP8 mode and scale layout.
  • Kernels.
    • New sparse_prefill_union_fp8.py (union) and sparse_prefill_direct_fp8.py (direct, plus the K/V pack with Q quantization).
    • Following the [FlyDSL] Add QSA sparse prefill attention for MI308 ROCm/aiter#6576 review, all of PyHIP's non-empty inline asm is replaced by stable FlyDSL operations:
      • the PV v_mfma_f32_32x32x16_fp8_fp8 with its AGPR constraints, now the public MFMA atom;
      • the AGPR rescale, now a plain multiply;
      • the s_nop hazard padding, now left to LLVM;
      • v_pk_mul_f32, now two FP32 multiplies.
    • Outputs are unchanged by these replacements. Union now has no scratch (PyHIP: up to 60 B) and is up to 2.5% faster.
    • The softmax scale is a runtime argument, as in the BF16 kernels.
  • Prepare and routing.
    • FP8 route calibration constants;
    • the split of short prefills across CTAs;
    • core/residual union tiles;
    • the pack gate;
    • K1 skips the sort of rows whose blocks already ascend. This also speeds up BF16 long-context chunks by 2.3–4.5%.
  • Fixes over PyHIP (all still present in PyHIP fb7ac47b):
    1. The FP8 pack buffer was passed to FlyDSL with an int32 shape and failed (struct.error) once the packed context reached 2³¹ bytes. It is now a [blocks * HK, 1024] byte scratch.
    2. The direct kernel derived per-token scale record offsets with an arithmetic right shift of an Int32 byte offset. Beyond 2 GiB of packed K/V, the scales were read as 0 and the affected rows were 0 or wrong. It is now an unsigned shift. The same pattern in the residual merge, which finds each row's union LSE from the row's out byte offset, made rows past 2 GiB of Q/out wrong. It is fixed too, and a new test covers it.
    3. Unrun FP8 union tasks past 4096 task entries.
      • The cause: with more task entries (tiles × KV heads) than one K3 order CTA sorts, K3 sorted the task order in chunks of 4096 and put each chunk's idle entries right after its own runnable ones. The FP8 union kernel ends a persistent CTA's task loop at its first idle entry, so the runnable tasks of later chunks never ran and their rows of out were left unwritten, with no error. For example, a 66000-row H6/HK2 batch left 1000 rows unwritten.
      • When it triggers: more than 4096 entries and idle entries in the first chunk, i.e. above about 40960 rows at H12/HK1, or half that at HK2. A single SGLang chunk (16384 tokens) stays below this.
      • The fix: K3 now places every chunk's runnable entries before all idle ones. This is a separate K3 variant, compiled only for plans with more than one order CTA, so every other plan runs the same K3 code as before.
      • The BF16 union kernel skips idle entries and was not affected, so main and [FlyDSL] [QSA] Sparse prefill follow-ups: runtime softmax scale, stable FlyDSL API, batch_invariant #4 are fine.
  • batch_invariant=True with FP8. The split is off (it depends on the batch's task count). The other rules are as in BF16 ([FlyDSL] [QSA] Sparse prefill follow-ups: runtime softmax scale, stable FlyDSL API, batch_invariant #4). core/residual stays, since it is a per-tile decision.

Performance (if applicable)

FP8, PyHIP fb7ac47b vs this PR. MI308X (gfx942, 80 CUs), FlyDSL 0.5.0. Measured with PyHIP's bench_attention_fp8.py protocol: 10 rotating inputs, 128 samples. Medians of 8 interleaved ABBA process pairs; outputs identical in every pair.

Hardware Workload/configuration Before (PyHIP) After Change
MI308X dense2048 TP2 (H12), per tensor / per token 140.34 / 152.28 µs 139.84 / 151.58 µs −0.36% / −0.46%
MI308X dense2048 TP4 (H6), per tensor / per token 102.88 / 110.56 µs 101.34 / 108.32 µs −1.50% / −2.03%
MI308X dense2048 TP8 (H3), per tensor / per token 78.01 / 82.14 µs 76.08 / 80.82 µs −2.47% / −1.61%
MI308X m12000 TP2, per tensor / per token 1720.18 / 1833.60 µs 1708.07 / 1820.60 µs −0.70% / −0.71%
MI308X m12000 TP4, per tensor / per token 1198.18 / 1308.31 µs 1178.62 / 1279.37 µs −1.63% / −2.21%
MI308X m12000 TP8, per tensor / per token 715.55 / 790.66 µs 703.49 / 782.36 µs −1.69% / −1.05%
MI308X random128k TP2/4/8, per tensor and per token — — −0.03% to +0.13%

BF16 default mode, #4 vs this PR: op test timing, 30 rows, medians of 8 ABBA pairs.

Hardware Workload/configuration Before After Change
MI308X chunk_1024_ctx_132096, H12 / H6 / H3 (K1 sort skip) 289.75 / 142.80 / 142.54 µs 283.17 / 136.33 / 136.16 µs −2.27% / −4.53% / −4.47%
MI308X prefill_4096, H12 676.45 µs 676.58 µs +0.02%
MI308X prefill_12000, H12 2469.57 µs 2469.72 µs +0.01%

The other BF16 rows are within ±1%, with one explained exception: prefill_2048 H3, whose timing is bimodal at the process level.

  • In an independent check, the first 8-pair round measured +1.30% and a second 8-pair round +0.29%. The PR's own runs measured +1.07% and +0.99%.
  • Per-kernel timing of processes in the same timing mode attributes about +0.3% (about 0.4 µs) to the changed K1/K2 prepare kernels.
  • The same K1 change saves 2.3–4.5% on long-context chunks (above).

batch_invariant=True with FP8, relative to the default mode on the same tree:

Hardware Workload/configuration Default batch_invariant=True Change
MI308X dense2048 TP8 (H3, split in the default mode), per tensor / per token 74.98 / 79.83 µs 93.41 / 101.87 µs +24.6% / +27.6%
MI308X Two full prefills, H6 406.34 µs 320.14 µs −21.2%
MI308X Full prefill 12000 + short requests, H3 1130.70 µs 920.22 µs −18.6%
MI308X 8 × 256-row extends, 2k–16k prefix, H3 (direct only) 150.52 µs 445.00 µs +195.6%
MI308X Other single requests (dense2048 TP2/4, m12000, random128k) — — −2.1% to +0.0%

SGLang end to end. Qwen3.8-Flash-Next, the customer TP2 configuration (1M context, NEXTN speculative decoding), FP8 target KV cache (fp8_e4m3) with a BF16 draft KV cache, MI308X. The two arms differ only in the QSA prefill (PyHIP vs this PR), run A, B, B, A. Throughput is in tok/s.

Hardware Workload/configuration Before (PyHIP) After Change
MI308X Customer workload, 256 × 12000/350, C8 462.4 465.5 +0.68%
MI308X Random 12000/350 × 32, C1 / C2 / C4 / C8 177.2 / 237.6 / 289.3 / 330.5 178.0 / 238.0 / 291.4 / 331.4 +0.44% / +0.19% / +0.72% / +0.29%
MI308X GSM8K, 1319 questions, 5-shot (accuracy / accept length) 0.9791 / 3.1351 0.9775 / 3.1388 −0.0015 / +0.0038

Testing

  • Unit tests added or updated
  • Performance benchmarks run
  • Tested on MI250X
  • Tested on MI300X

Documentation

  • Docstrings updated
  • User guide updated
  • Performance guide updated
  • No documentation change is needed, with the reason stated below

Dependencies

  • No new third-party dependencies added
  • New dependencies are listed and justified below

Breaking Changes

None. The new arguments are trailing, keyword-only and optional, and BF16 calls are bitwise unchanged.

PR Self-Review

Reviewed revision

43ce3d6bf292bfae4e21466c902c7d2f852a979d

Behavior and intent

qsa_sparse_prefill accepts FP8 K/V with per-token or per-tensor scales, in two compute modes:

  • compute="fp8": FP8 MFMA. The relative output error is a few percent on random data, and is dominated by the FP8 quantization of Q.
  • compute="bf16": BF16 accuracy, at the cost of a temporary BF16 copy of the packed context.

Results and routing match PyHIP fb7ac47b, except that the two 4 GiB bugs are fixed. BF16 calls give the same bits as before and are faster for long-context chunks.

Validation and unverified scope

All results below are on MI308X (gfx942, 80 CUs) with FlyDSL 0.5.0.

  • Bitwise comparison with PyHIP fb7ac47b (default mode):
    • 116 cases, all byte-identical with the same plan decisions. They cover PyHIP's FP8 test layouts, per tensor and per token, compute fp8 and bf16, packed and kv_slots, including 28 split plans and 36 core/residual plans.
    • The 9 benchmark jobs are also identical.
  • BF16 default mode vs [FlyDSL] [QSA] Sparse prefill follow-ups: runtime softmax scale, stable FlyDSL API, batch_invariant #4: 195 cases and 238 outputs byte-identical. The union and direct kernels are byte-identical in ISA.
  • batch_invariant=True: 86 FP8 batches and 2 graph cases: every request alone is bitwise equal to its rows in the batch. The 37 BF16 batches still pass.
  • python op_tests/test_flydsl_qsa_sparse_prefill.py: 132 rows, all pass (291 checkAllclose comparisons), none skipped. The cold-cache runtime is 395.2 s, against 276.0 s for the 99 rows on main (+43%).
    • The new fp8_hk16_prefill_24576_token row (Q and out of 2.25 GiB) checks every row past 2 GiB. With the residual merge's unsigned shift reverted at run time, it fails (worst 21.13); PyHIP fb7ac47b fails it the same way.
  • python op_tests/test_flydsl_qsa_sparse_prefill_guards.py: 16 checks pass, 103.2 s, including the FP8 kernel guards. Run-time patches of the FP8 primitives and edited sources make them fail as intended (9 + 5 mutations).
  • Independent bitwise check against PyHIP: 47 outputs and their plan decisions are byte-identical: 25 single requests and 22 batches; per tensor and per token; compute fp8 and bf16; packed and kv_slots.
  • Independent FP8 timing: 8 shapes, 8 ABBA pairs, −2.20% to +0.44% against PyHIP.
  • compute="bf16" under graph capture was checked once (replay equals eager, 0 differing elements). It is not in the op test.
  • Resources:
    • union_fp8: no scratch in all 24 variants; 1 wave/SIMD by design (waves_per_eu=1).
    • direct_fp8: no spill, 2 waves/SIMD.
    • All FP8 kernels: no v_pk_*_f32, and every exp2 lowers to llvm.amdgcn.exp2.f32.
  • SGLang end to end, on this PR's code:
    • Accuracy and throughput as in the table above. Logs show that every rank's 12 target QSA layers served FP8 K/V through this entry with compute=fp8, with no fallback.
    • With BF16 KV, the same A/B comparison gave GSM8K 0.9775 vs 0.9783 and throughput within −0.98% to +0.26%.
    • 1M-context TEST=1 (38 requests up to 999648 tokens; 7878 attention checks per rank against the reference on the kernels' FP8 Q) passes. The per-layout mean error is 0.42% on average (1.06% at most), and the worst row is at most 5.84%, against the 5% / 20% bounds.
    • With --enable-deterministic-inference, the adapter passes batch_invariant=True, and GSM8K (first 200 questions) scores 0.980.
      • This needs SGLANG_USE_1STAGE_ALLREDUCE=0: SGLang's 1-stage all-reduce, which it selects for deterministic mode on AMD, produces garbage with either QSA backend (an SGLang issue, reported separately).
  • Not verified:
    • other gfx942 parts such as MI300X (no hardware; they get the tuning warning);
    • MI250X (gfx942-only kernels).

PR-specific question

What behavior would each changed op test fail on before this PR, and why is that coverage not already provided by an existing canonical test? (the question the self-review rules select for changes under op_tests/)

  • op_tests/test_flydsl_qsa_sparse_prefill.py (27 FP8 rows). Before this PR the entry took BF16 K/V only, so every FP8 row fails there, and no BF16 row runs the FP8 pack, the FP8 kernels or the scale paths. Beyond that, each special row targets one failure:

    • Q and out past 2 GiB (fp8_hk16_prefill_24576_token) fails with the residual merge's arithmetic shift (worst 21.13), as PyHIP fb7ac47b does.
    • More than 4096 union task entries (fp8_hk16_k3_chunks_token) fails before the K3 fix: rows of out are left unwritten.
    • The byte-span rows fail unless a K/V span at the limit runs and one token over it is rejected.
    • The attention sink over a flat tail fails without the probability bias (E4M3 underflow).
    • The other special rows pin per-token V scale exactness, the fp8_query reference, large logits, graph replay at an unseen scale, the workspace key, entry validation and the public signature, batch invariance, group 16 and BQ42's 64-bit membership.
    • The 8 pairwise rows cover all 56 value pairs of compute × scale layout × route × input × group, in place of PyHIP's 82 FP8 test items. routes() asserts each row's path.
  • op_tests/test_flydsl_qsa_sparse_prefill_guards.py (FP8 guards) fails on regressions that leave outputs correct, or break them only after a FlyDSL or LLVM upgrade:

    • scratch, spills or occupancy;
    • the -packed-fp32-ops and waves_per_eu attributes;
    • extra empty asm, or FP8 MFMAs writing AGPRs;
    • conversions or exp2/log2/rcp not lowered to the AMDGPU intrinsics;
    • counted waits in the FP8 modules.

    9 run-time patches and 5 edited sources made them fail as intended.

Changed op tests

  • op_tests/test_flydsl_qsa_sparse_prefill.py: 27 FP8 rows.

    • Pairwise rows (8). Compute × scale layout × route (union, packed direct; FP8 has no raw direct) × input (packed, kv_slots) × group: all 56 value pairs covered in 8 rows. This is instead of PyHIP's 82 FP8 test items.
    • Special rows:
      • the byte-span limit, at the limit and one token over (P8);
      • an attention sink over a flat tail, which needs the probability bias to avoid E4M3 underflow;
      • per-token V scale exactness;
      • the fp8_query reference;
      • large logits;
      • graph replay with rewritten inputs and an unseen softmax scale;
      • workspace-key separation of FP8 modes;
      • entry validation and the exact public signature;
      • batch invariance with batch_invariant=True;
      • group 16 (direct and union);
      • BQ42 with 64-bit membership;
      • Q and out past 2 GiB, which exercises the residual merge's unsigned offset fixed here;
      • more than 4096 union task entries (fp8_hk16_k3_chunks_token: H192/HK16, 2620 rows, 4192 entries), which fails before the K3 fix above.
    • Every row maps to a feature or a review item of [FlyDSL] Add QSA sparse prefill attention for MI308 ROCm/aiter#6576, and routes() asserts the path each row takes.
  • op_tests/test_flydsl_qsa_sparse_prefill_guards.py: compile-only guards for the FP8 kernels, in addition to the BF16 ones. They compile three FP8 layouts with IR dumps and check the union, direct and pack kernels:

    • no compiler error and finite outputs;
    • no scratch; occupancy (union one wave per SIMD with bounded spills; direct two waves and pack eight, without spills);
    • -packed-fp32-ops, the waves_per_eu attributes, and no v_pk_*_f32;
    • the baseline numbers of empty asm blocks, FP8 MFMAs (none writing AGPRs), v_cvt_pk_fp8_f32, and exp2/log2/rcp lowered to the AMDGPU intrinsics;
    • in the FP8 modules, no counted waits.

    These failures would not show up as wrong outputs in the op test, or only after a FlyDSL or LLVM upgrade.

luocheng25 and others added 8 commits November 13, 2026 06:33
K1 sorted every row except complete causal prefixes, but the QSA prefill
indexer already emits each row's blocks in ascending order. Compare each
complete column with the next one (ds_bpermute from the next lane, lane 63
from lane 0's next column) and sort only rows with a descent. The written
block_indices rows are the same: an ascending row sorts to itself.

Ported from PyHIP 97c0f35a (SGLang-captured 12k selections: K1 70 -> 33 us
per layer).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ck and direct kernel

Port PyHIP fb7ac47b's FP8 (E4M3FNUZ) K/V path to qsa_sparse_prefill, on the shared
helpers, runtime softmax scale and low-level primitives module of the BF16 kernels.

Entry (API.md): qsa_sparse_prefill takes keyword-only k_scale, v_scale and
compute="fp8" after batch_invariant. K/V of dtypes.fp8 come with FP32 scales, both
per tensor [1] or both per token and KV head [N, HK] ([P, HK] with kv_slots);
values are K8 * k_scale, V8 * v_scale. compute="fp8" runs FP8 MFMA kernels on Q
quantized per (query, head) row (amax / 240); compute="bf16" dequantizes the
request tokens to BF16 and runs the BF16 path. The entry rejects BF16 K/V with
scales, FP8 K/V without them, scales of the wrong dtype, shape, layout mix,
contiguity or device, other FP8 dtypes, an unknown compute, scales overlapping
out, and FP8 K/V (or a pool) past 2**32 - 2**20 bytes. Workspaces are keyed by
the FP8 scale layout too.

FP8 direct kernel (sparse_prefill_direct_fp8): every call packs the context into
four-token blocks in 16x16x32 FP8 MFMA operand order (V^T per block, per token a
scale record), and the same launch quantizes Q. One wave per (row, KV head); the
lazy max keeps FP8 probabilities in range (P_BIAS). The packed K/V are
[blocks * HK, 1024] bytes: PyHIP passed them as one flat int32-sized dim and
failed from a 2**31-byte packed context (struct.error), half the FP8 limit.
Past 2 GiB of packed K/V the direct kernel finds a block's per-token scale record
by an unsigned shift of its byte offset (PyHIP's arithmetic shift read zero
scales there, so those rows came out zero or wrong); the residual merge's LSE
offset shifts unsigned too.

Plans: FP8 route calibration (platform(fp8), FP8_ROUTES), key-range split of short
prefills' long union tasks (K3 order entries with slot and part), core/residual
union tiles (CORE_RATIOS: K2 keeps a tile's core when that costs less, K3b in the
FP8 pack launch lists each row's residual blocks for the direct merge). BF16 plans
compile K2 and K3 without the core/residual work. batch_invariant=True never
splits (a split depends on the whole call). FP8 calls run direct only until the
FP8 union kernel lands.

Shared helpers: one FP8 MFMA (_mfma_fp8, public MMA atom), the lazy rescale
branch (_lazy, under _rescale and the FP8 _lazy_max), BF16 output rounding, V
scale terms and Q quantization. New primitives with their reasons: empty-asm
hold and compiler fence, DPP row broadcast, v_cvt_pk_fp8_f32, bare v_log_f32 and
v_rcp_f32. The FP8 kernels have no non-empty inline asm.

Validated on MI308X (gfx942, 80 CUs), FlyDSL 0.5.0: every direct-only FP8 layout
of PyHIP's FP8 tests (per tensor and per token, compute fp8 and bf16, packed and
kv_slots, attention sink, growing scores, exact uniform) equals PyHIP fb7ac47b
bit for bit; FP8 K/V of exactly 2**32 - 2**20 bytes (per token, packed, rows on
both sides of 2 GiB; and a pool from 2 GiB to the limit) are within the FP8
error bounds.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Port PyHIP fb7ac47b's FP8 (E4M3FNUZ) union kernel and route FP8 union plans to
it. A task is 128 (query, head) rows of a union tile over its compact block list:
four waves of 32 rows run 32x32x16 FP8 MFMAs, QK of tile t, the lazy max and
exponentials of tile t - 1 and PV of tile t - 1 interleaved with the K/V^T LDS
pipeline (V^T read from the FP8 pack). Split plans store each key range's O, m and
l and the part counting last merges them; core/residual plans write each row's
log-sum-exp for the direct merge.

PyHIP's non-empty inline asm is replaced by FlyDSL 0.5.0's public API, as
prototyped by F0 (outputs bit-identical, faster): PV MFMAs through the public MMA
atom (LLVM keeps O in VGPRs instead of asm-pinned AGPRs), the lazy rescale by
plain multiplies in the shared _lazy_max, no asm s_nop (LLVM's hazard recognizer
places the wait states), and the BF16 output scale by two FP32 multiplies (no
v_pk_mul_f32). The softmax scale is a kernel argument (SCALE_LOG2), as in the BF16
kernels. quantize_q quantizes Q as compute="fp8" does, for references.

Validated on MI308X (gfx942, 80 CUs), FlyDSL 0.5.0: 116 layouts of PyHIP's FP8
tests (per tensor and per token, compute fp8 and bf16, packed and kv_slots,
default and always-planned unions, split and core/residual plans, attention
sink, growing scores, exact uniform, sharp scores) equal PyHIP fb7ac47b bit for
bit with the same plan decisions; union_fp8 has no scratch and one wave per SIMD
(VGPR + AGPR 365-451).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 25 FP8 K/V cases to op_tests/test_flydsl_qsa_sparse_prefill.py for the
FP8 port of aiter.ops.qsa.qsa_sparse_prefill (k_scale, v_scale, compute).
Each runs once, at its own group. Data and bounds follow PyHIP's FP8 tests:
K/V spread by 2**U(-1, 1) per (token, KV head) and quantized with scale
amax / 240; compute="fp8" output within FP8_TOL (mean 0.06, worst 0.16,
norm-relative per (row, head)) of the reference on the dequantized K/V over
sample rows, compute="bf16" bit-identical to the BF16 path on the K/V the
caller dequantizes.

- Pairwise coverage of compute (fp8/bf16), scales (per tensor/per token),
  paths (union, packed and raw direct), packed or pooled K/V and groups
  12/6/3 in eight cases. Pools hold NaN in their unused rows and per token
  scales, and must give the packed call's bits.
- Group 16 (direct through a pool at HK2 and T = 256; union with BQ8 tiles),
  HK2 at T = 768, T = 1024 over a 132096-token context, the BQ42 tiles of an
  H3 full prefill, large logits, and the 12000-token serving prompt, timed
  against the BF16 path on the dequantized K/V and aiter's Triton FP8 GQA.
- An attention sink over a flat tail (FP8 probability underflow), K = 0 with
  power-of-two V scales and zero Q rows (exact output), and the reference on
  the kernels' own FP8 Q, also for sharp scores.
- The 4 GiB - 1 MiB FP8 byte span: packed K/V at the limit, a pool whose
  pages start at 2 GiB and end at the limit, and one token past it, which
  compute="fp8" must reject while compute="bf16" reads the same pool.
- Graph replay with Q, K/V, scales, indices and slots rewritten in place,
  scratch growth and softmax scales captured unwarmed; a workspace key that
  must hold the K/V dtype, scale layout, pool and batch mode; input errors
  and the entry's signature; batch invariance with batch_invariant=True.

routes() reports the FP8 paths too (key-range split, core/residual tiles,
skipped K pack) and the cases assert them; the helpers from latest_workspace
to scratch_arena are the only code reading prefill internals. run_torch can
dequantize FP8 K/V after its gather and compute sampled rows only, and
make_pool builds FP8 pools.

On MI308X (gfx942) all 25 cases pass with the FP8 kernels of this series,
and the earlier rows pass unchanged. Before those kernels existed, the cases
were also run against PyHIP fb7ac47b's attention() behind this entry (at the
default softmax scale, with batch_invariant emulated): all passed except the
packed limit case, where PyHIP's FP8 pack passes its flat packed K/V with
int32 dims and fails from a 2**31-byte packed context (struct.error), which
this series fixes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The FP8 direct kernel merges the residual blocks of core/residual union
tiles into their rows by log-sum-exp, finding each row's union LSE from the
row's out byte offset (_merge_o). Past 2 GiB of out that offset is negative
as a signed Int32; the merge must shift it unsigned. No case reached it.

Add fp8_hk16_prefill_24576_token to op_tests/test_flydsl_qsa_sparse_prefill.py:
a 24576-row prefill at HK16 and group 12 (H192, per token scales), whose Q and
out take 2.25 GiB, so rows from 21845 on reach past 2 GiB, and whose union
tiles keep cores (77 rows past 2 GiB merge residual blocks on MI308X).

- It is checked against the reference on the kernels' FP8 Q
  (FP8_QUERY_CHECKED: FP8 Q rounding takes one of its 192-head vectors past
  FP8_TOL's worst bound) on every row past 2 GiB (FP8_LARGE_Q_CASES), the
  full reference taking 27 GiB and 15 s; the sample rows meet none of the
  merged rows past 2 GiB.
- residual_rows() reads the plan's per-row residual block counts; at least
  one row past 2 GiB must have merged.
- run_perftest rotates inputs only when Q is below 1 GiB too, and kv_bytes()
  counts FP8 cases' Q three times in the free memory a case needs.

On MI308X (gfx942) the case passes in about 15 s (fp8_query error mean
0.021, worst 0.061 over the 2731 rows past 2 GiB). With _merge_o patched back
to the signed shift at run time it fails: worst 21.13.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extend op_tests/test_flydsl_qsa_sparse_prefill_guards.py to the FP8
kernels, which keep their own low-level primitives: empty asm holding
values (prim.hold) and fencing loads (prim.compiler_fence), DPP row
broadcasts, v_cvt_pk_fp8_f32, bare log2 and reciprocal, and the FP8 union's
one wave per SIMD.

Three more layouts are compiled with IR dumps, FP8_CASES entries at their own
group: fp8_ragged_4_token (union with core/residual tiles, direct with the
residual merge, per token pack), fp8_kv_slots_ragged_4_tensor (the same
through a slot table, per tensor) and
fp8_kv_slots_topk1024_mixed_ctx_132096_token (union split into two key
ranges, T = 1024). Their union, direct and pack kernels must show:

- no compiler error and finite outputs;
- no scratch; union one wave per SIMD with at most 1 VGPR and 48 SGPR
  spills, direct two waves and pack eight, both without spills;
- -packed-fp32-ops, waves_per_eu 1 (union) and 2 (direct), no v_pk_*_f32;
- the baseline number of empty asm blocks: union 10 per tensor and 18 per
  token ("v"), direct 3 ("~{memory}");
- union MFMAs all v_mfma_f32_32x32x16_fp8_fp8 and direct ones
  v_mfma_f32_16x16x32_fp8_fp8, the baseline number, none writing AGPRs;
- every exp2, log2 and reciprocal lowered to llvm.amdgcn.exp2/log/rcp.f32,
  the baseline number of each;
- the baseline number of v_cvt_pk_fp8_f32.

Without compiling, the FP8 modules' s_waitcnt arguments must all be 0 and
none may use LdsQueue (they keep no counted wait); the stable-API source
audit already covers them.

On MI308X (gfx942) all 16 guards pass in about 105 s. Patching the FP8
primitives at run time makes them fail as intended: dropping prim.hold or
prim.compiler_fence, log2 through fx.math.log2, reciprocal by division,
each FP8 conversion done twice, each FP8 MFMA done twice, the target
feature dropped, exp2 without afn, and the union's waves_per_eu set to 2
(LLVM keeps one wave, the attribute check fails). Edited copies of
sparse_prefill_union_fp8.py with a counted lgkmcnt or vmcnt, LdsQueue,
flydsl._mlir or inline asm are reported.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With more task entries than one K3 order CTA sorts (THREADS * 16), K3
sorts the task order in chunks, one order CTA each, and each placed its
chunk's idle entries (tiles routed direct, padding) right after its
runnable ones. The FP8 union kernel ends a persistent CTA's task loop at
its first idle entry, so runnable tasks of later chunks behind it never
ran: a 66000-row batch of a random-selection and a shared-selection
request (H6, HK2; 4160 entries) left 1000 rows of out unwritten, and with
batch_invariant=True its second request no longer matched its run alone.

Each order CTA of such a plan now also counts the runnable entries of
every chunk (all order CTAs compute the same route threshold) and places
its runnable entries after those of the earlier chunks and its idle
entries after every runnable entry, so every union CTA still meets all
of its runnable tasks first. This is a separate K3 variant (CHUNKED):
plans with one order CTA (up to 4096 entries, every split plan) compile
the same kernel as before. The BF16 union kernel skips idle entries and
was not affected.

Plans with one order CTA keep identical K3 code (same ISA), task orders
and outputs. The batch above now writes every row, and with
batch_invariant=True each of its requests equals its run alone in BF16
and FP8.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
K3 sorts the union task entries in order CTAs of 4096 entries each, and every
CTA puts its idle entries (tiles routed direct) at the end of its own chunk.
The FP8 union kernel ends its persistent loop at the first idle entry, so with
more than 4096 tasks and direct tiles in the first chunk it never runs the
tasks of the later chunks, and their rows of out are never written. The BF16
union kernel walks every entry and is not affected.

Add fp8_hk16_k3_chunks_token: a random request whose 6 tiles route direct (a
127k-token context) ahead of a shared 2560-row request, at HK16 and group 12
(H192): 262 tiles x 16 KV heads = 4192 tasks. On 88b74a9 the 96 tasks of the
second chunk (KV head 15 of the last 96 tiles, 960 rows) never run and the row
fails; with K3 ordering every chunk in one CTA it passes. It is checked
against the FP8-Q reference on every row, like the other H192 case.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant