Skip to content

[FlyDSL] [QSA] Optimize the sparse prefill kernels on gfx942 - #3

Open
luocheng25 wants to merge 1 commit into
luocheng/qsa-fp8from
luocheng/qsa-opt
Open

luocheng25 wants to merge 1 commit into
luocheng/qsa-fp8from
luocheng/qsa-opt

Conversation

@luocheng25

@luocheng25 luocheng25 commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

Speeds up the FlyDSL QSA sparse prefill (aiter.ops.qsa.qsa_sparse_prefill) on MI308X (gfx942), for BF16 and FP8 K/V. The signature and the documented contract are unchanged.

  • Whole call vs the base (same GPU, same session):
    • FP8 K/V is 16–27% faster and BF16 K/V 5–35% faster over twelve benchmark configurations.
    • The SGLang-shaped case (12000-token prompt through kv_slots, TP2, per-tensor FP8) drops from 1709.7 to 1251.5 µs (−26.8%).
    • On 12 captured SGLang layers, measured directly against the base, FP8 K/V is 18–35% faster and BF16 K/V 4–27% faster. The captured workloads are a 12000-token prompt, and 16384-token chunks after 64k- and 1M-token prefixes.
  • Kernel efficiency. 27 of the 48 union/direct kernel items that take ≥ 10% of a call now run at ≥ 80% of their measured bound (base: 2).
  • Accuracy.
    • BF16 stays within the 2e-2 tolerance.
    • FP8 errors stay close to the base's. On captured Qwen3.8 selections, the mean error changes by −0.5% to +0.2%.
    • Over all audited FP8 cases, the mean error rises by at most 2.6% and the worst-row error by at most 3.2%. The single exception is one exact-reference vector (see Validation).
    • batch_invariant=True turns every new split off.
  • Plan reuse. Layouts that fix every selection (each request spans ≤ 4T + 3 tokens) reuse their plan while the stream's scratch still holds it. K1 and K2 are skipped, and so is K3 unless a slot table needs remapping. Such calls do not read indices.

This PR targets luocheng/qsa-fp8 (#5, head 43ce3d6bf), the QSA FP8 port that follows ROCm#6576 and #4, so its diff holds only these changes; "the base" below means that branch. The changes keep to the rules of ROCm#6576:

  • the softmax scale stays a runtime argument;
  • there is no non-empty inline asm, and no flydsl._mlir outside sparse_prefill_primitives.py;
  • batch_invariant=True covers every new path;
  • guard tests pin the low-level code.

Motivation

SGLang serves Qwen3.8-Flash-Next prefill through this op, one call per QSA layer. With the base, the kernels ran well below what MI308X sustains (measured with the same tooling as the tables below):

  • FP8 direct: 53–64% of the L2→CU bandwidth. Its K pack order cost 22–25 cache-tag lookups per 1 KiB wave load instead of 16, and a chunk's loads all issued at once.
  • FP8 union: 33–63% of the FP8 MFMA peak. One wave per SIMD ran softmax and MFMAs in series.
  • BF16 union: 31–77%. Each 16-row wave read every K/V operand from LDS for itself.
  • Prepare chain. K1–K3 rebuilt the plan on every layer, even when the layout fixes the selection.

Only 2 of the 48 must-reach kernel items reached 80%.

Changes

  • Prepare chain (K1–K3) and planner (sparse_prefill_prepare.py, entry):
    • Plan reuse. When each request spans ≤ 4T + 3 tokens, the layout determines indices. The plan is then reused while the stream's scratch still holds it:
      • K1 and K2 are skipped. With a slot table, K3 still runs to map the plan's blocks to pool rows from kv_slots; split plans only re-zero their counters.
      • Graph capture records the full preparation.
      • Such plans take no core/residual tiles.
      • SGLang's layers of a short prefill share a layout: each later call of a 2048-token prefill was 10–17% faster when this was added. The benchmark below never reuses a plan.
    • Wide tiles. BQ21/BQ42 tiles (H6/H3, one KV head) are now chosen by the batch's rows, so chunks after a prefix and batches use them too. Such layouts were 8–20% faster when this was added.
    • K2: dense windows up to 8× the dense limit with the rows' block ids kept in registers, a row-overlap probe that sends random-looking tiles to a cheaper bound, 512-thread CTAs for long ranges, and a per-plan unroll. A 16k chunk after a 64k prefix at TP8 goes from 108 to 89 µs.
    • K3: an LDS counting sort; largest-differencing balance of the deal's last rounds; four-round plans re-dealt round by round, which puts TP8 CTAs within 1% of the even load (decided per launch, without a specialization of its own); wide-tile masks shared by two CTAs.
    • Routing: per-task overheads per union kernel and refit FP8 route terms. Every step to fewer union tiles must save route_margin (5%), because direct rows run further from the FP8-Q reference.
    • Long contexts. Beyond 128k tokens, the lagging side is weighted +20% per doubling, through FP32 kernel arguments.
      • That side is K2's direct budget at FP8 H3 and BF16 H3/H6, and K3's union step otherwise.
      • Plans up to 128k decide bit for bit as before.
    • Also: scratch views are 4 KiB-aligned. K1, K3 and the new persistent and listed launches now pass prim.kernel_attrs(), like the other kernels did in the base; K2 and the two BF16 K/V packs stay exempt.
  • FP8 union (sparse_prefill_union_fp8.py):
    • Two waves per SIMD (base: one). Waves w and w + 4 share 32 rows; each scores half of every 64-key tile and accumulates half of the dims.
    • Overlapped stages. Between CTA barriers, one wave's MFMA stage (QK and PV, at issue priority) overlaps its sibling's VALU stage (softmax, masks, K/V staging, loads).
    • Interleaved VALU. exp2, conversions and row sums interleave with the MFMAs. The per-tensor row sum runs on v_smfmac_f32_16x16x64_fp8_fp8, over the FP8 probabilities PV multiplies.
    • Task boundaries. The next task's loads issue while the current one drains. A boundary costs ~11k cycles, down from 14–17k.
    • Low-level. prim.LgkmLedger counts LDS reads in flight across barriers. One LLVM option, set in the primitives, keeps uniform branches unstructurized in every FP8 union variant.
    • Result. 256 VGPRs and no scratch. The SGLang-shaped case goes from 703.6 to 499.2 µs.
  • FP8 direct and K/V pack (sparse_prefill_direct_fp8.py):
    • Pack layout. The operand-order pack is the base's; what changes is the order inside it. K's 16-byte column groups are now token-minor ((g·4 + token)·16; the base: (token·4 + g)·16), so a wave's 1 KiB load covers whole 64-byte lines. Per-token scale records hold one 16-byte entry per token (base: one 16-byte array per term).
    • Loads. The next chunk's loads issue in three groups interleaved with the chunk's MFMAs, not all at the chunk start. Only the peeled last chunk masks keys.
    • Plain plans. They run four rows per workgroup, heaviest first. When the union takes every tile, they exit on K3's direct flag, which cuts the empty launch from 15.5 to 4.3 µs.
    • Persistent full-topk plans. Full-topk plain plans run one 8-wave workgroup per CU.
      • Its waves keep pace and sweep their sorted blocks in step, keeping K/V in L2.
      • Items come from a branch-free queue: lane 0 issues the buffer atomic and the other lanes are out of range.
    • Stores and dispatch. Persistent and listed rows write contiguous 1 KiB stores, staged in LDS. The pack dispatches its latency-bound residual-list CTAs first.
    • Row sums add the FP8-decoded probabilities that PV multiplies.
    • Result. L2→CU utilization goes from 53–64% to 80–88%. The SGLang-shaped case goes from 823.8 to 592.4 µs.
  • BF16 (sparse_prefill_union.py, sparse_prefill_direct_packed.py, sparse_prefill_direct.py):
    • Union. It runs four waves of 32 rows with 32x32x8 MFMAs (base: eight waves of 16 rows with 16x16x16). This halves the LDS reads per MFMA.
    • Key-range splits. Short prefills split their longest union tasks into two key ranges. Each wave counts its rows with one lane-0 buffer atomic, and the last wave merges.
    • Packed direct.
      • It splits the call's last items into key-tile parts.
      • Full-topk plans that fill a wave per SIMD run persistent, with the same queue and the same outputs.
  • Shared (sparse_prefill_common.py, sparse_prefill_primitives.py):
    • Persistent helpers. The persistent pacing and queue helpers live here. Each queue is a workspace tensor that its launch's last wave zeroes.

    • New primitives, each with its reason in the module docstring: smfmac_fp8, LgkmLedger (replaces LdsQueue), uniform_branch_options, buffer_atomic, dpp_banks, fp8_pair, store_lds, buffer_load_x3, lane_id.

    • One QK/PV helper per MFMA shape, on purpose, since each shape has its own fragment layout:

      • common._qk/_pv: BF16 16x16x16, used by both BF16 direct kernels;
      • union._qk32/_pv32: BF16 32x32x8;
      • direct_fp8._qk_half_fp8/_pv16: FP8 16x16x32;
      • the FP8 union's M-stage: 32x32x16.

      The FP8 MFMA call, common._mfma_fp8, is shared.

Performance (if applicable)

Whole qsa_sparse_prefill call (K1–K3, pack and attention kernels).

  • Units: µs, given as BF16 / FP8 per-tensor / FP8 per-token K/V.
  • Setup: MI308X (gfx942, 80 CUs), FlyDSL 0.5.0, Peak TOPS Limiter (PTL) enabled.
  • Shapes: T = 512, D = 256, one KV head. TP2/4/8 means 12/6/3 Q heads.
  • Protocol (as in the base):
    • 10 rotating input buffers and 128 samples;
    • the three K/V types interleaved, so no plan is reused;
    • outputs bitwise equal across buffers;
    • both trees on the same GPU in the same session.
Hardware Workload/configuration Before (base) After Change
MI308X 2048-token prefill, full causal selection, TP2 195.6 / 139.8 / 151.6 172.1 / 112.3 / 125.2 −12.0 / −19.7 / −17.4%
MI308X same, TP4 134.9 / 101.6 / 108.7 109.0 / 80.5 / 89.4 −19.2 / −20.8 / −17.8%
MI308X same, TP8 130.2 / 76.1 / 80.6 84.6 / 62.7 / 67.6 −35.0 / −17.6 / −16.0%
MI308X 12000-token prompt, synthetic indexer-like selection, TP2 2476.8 / 1706.6 / 1819.7 2336.2 / 1241.5 / 1357.7 −5.7 / −27.3 / −25.4%
MI308X same, TP4 1853.5 / 1178.7 / 1280.3 1654.9 / 856.8 / 970.4 −10.7 / −27.3 / −24.2%
MI308X same, TP8 1068.6 / 702.0 / 779.6 928.5 / 539.8 / 621.2 −13.1 / −23.1 / −20.3%
MI308X same prompt through kv_slots (285312-row pool, random 64-token pages; per-tensor FP8 with unit scales, as SGLang stores it), TP2 2463.0 / 1709.7 / 1834.0 2338.7 / 1251.5 / 1375.3 −5.0 / −26.8 / −25.0%
MI308X same, TP4 1841.0 / 1190.0 / 1309.8 1664.1 / 867.9 / 986.4 −9.6 / −27.1 / −24.7%
MI308X same, TP8 1057.9 / 711.2 / 795.0 935.0 / 545.5 / 633.4 −11.6 / −23.3 / −20.3%
MI308X 4096-token chunk after a 126976-token prefix, random selection, TP2 1494.5 / 895.7 / 925.0 1377.3 / 697.8 / 733.7 −7.8 / −22.1 / −20.7%
MI308X same, TP4 1474.9 / 869.4 / 899.9 1352.0 / 682.9 / 717.2 −8.3 / −21.5 / −20.3%
MI308X same, TP8 1466.2 / 853.2 / 883.7 1340.5 / 671.8 / 705.5 −8.6 / −21.3 / −20.2%

Kernel utilization is max(compute, memory), against peaks measured on this machine:

Resource Measured peak
FP8 MFMA 32x32x16 / 16x16x32 577.4 / 565.8 TFLOPS
BF16 MFMA 32x32x8 / 16x16x16 290.8 / 284.7 TFLOPS
L2→CU 9.43 TB/s
MALL 9.39 TB/s
HBM 4.35 TB/s
LDS 18.83 TB/s
  • Compute counts only the attention MFMAs (QK and PV, padding included). The per-tensor row-sum v_smfmac is excluded.
  • Sources: bytes come from PMC counters and times from kernel traces.
  • Must-reach items are kernels taking ≥ 10% of a call.
  • Near 80%: items within one point of 80% are medians of 5 interleaved runs.
Kernel (bound) Items ≥ 80% This PR Base
FP8 direct (L2→CU) 14 / 14 80.3–87.9% 53.4–63.6%
FP8 union (FP8 MFMA) 3 / 18 40.1–83.0% 33.0–63.2%
BF16 union (BF16 MFMA) 6 / 9 52.6–84.7% 30.8–77.2%
BF16 packed direct (L2→CU) 4 / 7 74.1–88.2% 68.1–87.8%

The 21 items below 80%:

  • 2048-token prefill. FP8 union: 6 items, 40.1–59.8%. BF16 union: 3 items, 52.6–66.2%.
    • There are too few tasks per CU: 13–21 MFMA steps at TP4/TP8.
    • The launch's fixed cost is 13–17% of the ideal time.
    • A calibrated makespan model keeps these items below 80% within the register and LDS budget. At TP4/TP8, even a 100%-efficient steady state falls short.
  • Per-token FP8 union. 6 items, 62.6–69.7%.
    • Each score element needs three VALU instead of one, two of them DPP.
    • gfx942 has no VOP3 DPP, and LLVM does not fold DPP moves into v_fmac_f32.
    • Algebraic rewrites exceed the FP8 error budget.
  • Per-tensor FP8 union at TP2. 2 items, 74.7% and 75.7%.
    • With ~15 tasks per CTA, the task boundary weighs most.
    • A fused task stream to hide it does not fit the register and LDS budget.
  • Per-tensor FP8 union at TP4 through kv_slots. 1 item at 79.1%, 0.9 points short.
  • BF16 packed direct, random selections over 128k tokens. 3 items, 74.1–75.7%.
    • The L2 hit rate is already at a synchronized-scan microbenchmark's level (~38.7%).
    • The rest is the scan's time structure. The practical ceiling is ≈ 78–80%.

Captured SGLang layers, the base vs this PR.

  • Workload: 12 QSA layers of Qwen3.8-Flash-Next captured from SGLang (the indexer's selections, random Q/K/V).
  • Units: each figure is the whole call summed over the 12 layers, in ms, as BF16 / FP8 per-tensor / FP8 per-token K/V.
  • Protocol: both trees on the same GPU, alternating.
    • FP8: medians of 8 ABBA pairs.
    • BF16: medians of 4 pairs, with clock-checked rounds.
    • Every run passed the gate, and none was voided.
Hardware Workload/configuration Before (base) After Change
MI308X 12000-token prompt, 12 captured layers, TP2 31.37 / 19.38 / 20.87 29.24 / 14.30 / 15.93 −6.8 / −26.2 / −23.7%
MI308X same, TP4 21.96 / 12.38 / 13.46 19.13 / 9.23 / 10.52 −12.9 / −25.4 / −21.9%
MI308X same, TP8 12.53 / 8.07 / 9.03 10.79 / 6.08 / 7.09 −13.9 / −24.7 / −21.4%
MI308X 16384-token chunk after a 65536-token prefix, TP2 55.59 / 38.33 / 40.77 53.02 / 28.30 / 30.14 −4.6 / −26.2 / −26.1%
MI308X same, TP4 50.88 / 32.83 / 35.98 43.19 / 23.18 / 26.38 −15.1 / −29.4 / −26.7%
MI308X same, TP8 40.92 / 25.34 / 28.56 29.85 / 16.42 / 19.12 −27.1 / −35.2 / −33.0%
MI308X 16384-token chunk after a 983040-token prefix (1M context), TP2 74.07 / 49.26 / 52.28 70.93 / 39.87 / 42.45 −4.2 / −19.0 / −18.8%
MI308X same, TP4 69.31 / 45.58 / 48.93 61.28 / 36.93 / 40.10 −11.6 / −19.0 / −18.1%
MI308X same, TP8 60.48 / 40.19 / 44.03 46.74 / 29.85 / 34.31 −22.7 / −25.7 / −22.1%
  • Through kv_slots: with unit per-tensor scales (how SGLang stores FP8 K/V), the 12000-token prompt is 25.6% / 25.0% / 24.6% faster at TP2/4/8.
  • Per layer:
    • Of the 360 per-layer timings, only one is slower than the base: one layer of the 1M context, at TP4 with BF16 K/V, is 2.3% slower.
    • Every other layer is 1.6–37% faster.
  • Accuracy on these layers:
    • FP8 mean error changes by −0.4% to +0.1%.
    • BF16 has no element outside 2e-2.

Testing

  • Unit tests added or updated
  • Performance benchmarks run
  • Tested on MI250X
  • Tested on MI300X

Documentation

  • Docstrings updated
  • User guide updated
  • Performance guide updated
  • No documentation change is needed, with the reason stated below

Dependencies

  • No new third-party dependencies added
  • New dependencies are listed and justified below

Breaking Changes

None.

  • The signature, arguments and documented contract are unchanged.
  • Outputs change only within the accuracy contract: BF16 within 2e-2, FP8 within the error budget.
  • Plan reuse relies on the existing rule that indices follow the documented layout.

PR Self-Review

Reviewed revision

589f338fecc5cff12d6b8e7fb3d97f323fc8dba4

Behavior and intent

  • Outputs change in low-order bits: the BF16 union accumulates with 32x32x8 MFMAs, routes are recalibrated, and FP8 row sums add the probabilities PV multiplies.

  • Plan reuse (new). The entry docstring states: "A layout whose every request spans at most 4T + 3 tokens (each row then selects all of its complete blocks) reuses its plan instead while no other call on the stream, nor a graph capturing its scratch, ran since its last call; such calls do not read indices."

    • For such layouts, the documented indices layout leaves exactly one valid selection.
    • Callers must already follow that layout, which is not validated.
  • batch_invariant=True:

    • turns off the BF16 union's key-range split and the packed direct's key-tile split (the FP8 split was already off);
    • picks wide tiles per request;
    • keeps persistent launches bitwise equal to one-wave launches.

    Short prefills lose the split's gain, for example about 7% on a 2048-token BF16 prefill at TP4.

  • Interface. There are no new arguments, environment variables or platform fields. Only the FP8 route terms are refit.

  • Compilation. New compile-time parameters (K2's unroll; K3's balance, sort-width and mask-partition flags; the BF16 split; the persistent and split direct launches) are part of the compile keys, and each variant compiles on first use. The softmax scale and the long-context weights (SPAN_SLOPE, UNION_STEP) are FP32 kernel arguments.

  • First use in a process.

    • FlyDSL traces a specialization's Python DSL before it looks up its disk cache. Every specialization therefore costs its trace on first use in each process, even when the cache holds the artifact: about 0.1–2.7 s with a warm cache.
    • In the SGLang benchmark sequence below (FP8 K/V, BF16 draft layer, TP2), the base first uses 25 QSA specializations per process and this PR 32. With a warm cache, first use costs 15.6 s per process for the base and 21.6 s for this PR, all of it before the timed items.
    • Before this head, four-round plans had a K3 specialization of their own. Its first use (about 1 s per K/V type) fell inside the timed benchmark, the first time a chunked prefill left a ~3200-row remainder, and lowered that run's throughput by 3.5–4.6%. K3 now decides this per launch, so the 12000-token prefill's specialization serves such plans too (the op test compiles 27 K3 specializations instead of 30).
  • Persistent state.

    • Queues and list counters are workspace tensors, not arena scratch, and each launch leaves them zeroed.
    • Split counters are cleared before every call.
    • Repeated calls and graph replays therefore start clean.
  • Tuning. The constants are calibrated on MI308X. Other gfx942 parts log a warning and reuse them.

Considered and not adopted:

  • Non-temporal output stores. The op gains ~0.6%, but nt lines bypass MALL, so the next GEMM (o_proj) reads them from HBM. The net effect is a 5–16 µs loss on the 12000-token prompt.
  • Clock-paced persistent rounds (s_memrealtime/s_sleep). They gain 2–3% on random full-topk selections, but they depend on the previous launch's timing, have a performance cliff, and help only synthetic loads.
  • A fused task stream in the FP8 union. It hits the register limit and is slower everywhere, even with merged output writes.
  • FP8 key-range splits for fixed-selection plans. They are 3.5% faster in the benchmark but 1.3% slower on SGLang's plan-reuse path, where the split counters need an extra zeroing launch.
  • Others. 3- and 4-way splits, fusing K1 into the pack launch, and exp2 replacements (LDS tables, polynomials, Schraudolph) were slower or outside the FP8 error budget.

Follow-ups:

  • LLVM. Folding DPP moves into v_fmac_f32 in GCNDPPCombine would gain ~2% of the per-token FP8 union.
  • SGLang.
    • Capturing the prefill launches in a hipGraph saves 0.4–0.75 µs per launch.
    • A KV pool in the pack layout would remove the pack launch.
  • API. A block-level indices interface would cut K1 traffic to ~1/4. This is an API change.
  • Empty launch. Full-topk FP8 plans issue one empty ~4.3 µs launch; merging the persistent FP8 direct and union launches would remove it.
  • First use.
    • FlyDSL could key its artifact cache on the source and constexpr arguments, so a cache hit skips the trace. That would remove most of the per-process first-use cost, for the base as well.
    • Alternatively, aiter could offer a warm-up entry per layout family, for SGLang to call at startup.
  • Calibration.
    • Calibrate the >128k weighting per layer.
    • Let a reused plan's union zero its own split counters, then revisit FP8 splits for fixed selections.

Validation and unverified scope

Results are on MI308X (gfx942, 80 CUs) with FlyDSL 0.5.0. The op test, guards and lint below ran on this head.

Tested trees. This head is a single commit on the base's head 43ce3d6bf. The benchmarks and the accuracy audits below were measured on an earlier tree, which differs from this head in two ways:

  • Three docstrings. Once docstrings are removed, the Python ASTs of the two trees are equal.

  • One K3 change (sparse_prefill_prepare.py, +41/−25). Four-round plans no longer compile a K3 specialization of their own (see Behavior and intent). It was verified on this head:

    • Outputs are bitwise equal to the earlier tree on the 203 synthetic layouts, on 42 captured configurations × 12 layers, on the 12000-token, 64k and 1M captured layouts at TP2/4/8 for all three K/V types, and on the benchmark configurations.
    • The whole call moves by −0.25% to +0.19% over 30 configurations (8 ABBA pairs; two short calls re-measured, +0.11% and +0.01% over 16 pairs).
    • op test, guards, black and ruff pass (below).
  • python op_tests/test_flydsl_qsa_sparse_prefill.py: 134 rows (base: 132), all pass.

    • 330 checkAllclose comparisons, none skipped.
    • All 33 FP8 route expectations hold.
    • Cold cache: 392.7 s on this head. Earlier trees took 394–399 s, against 282–283 s for main's 99 rows in the same session: 1.40× main (base: 1.43×).
  • python op_tests/test_flydsl_qsa_sparse_prefill_guards.py: 21 checks pass (base: 16).

    • Cold cache: 146.6 s and 145.7 s, against the base's 102.2 s in the same session (1.43×).
    • When added, each new check was shown to fail on a matching mutation.
  • Lint: black --check (26.10.1) and ruff check (0.16.0) on the 11 changed files are clean.

  • FP8 error vs the base.

    • Method: the same CPU-generated inputs, checked by hash; exact and FP8-Q references; mean and worst-row error.
    • Synthetic coverage (203 layouts):
      • the op test's FP8 rows;
      • per-tensor, per-token, kv_slots, HK2, sink, large-logit and Q-past-2-GiB layouts;
      • prefix chunks and batches;
      • the 24 FP8 benchmark configurations.
    • Captured layers: 42 configurations × 12 layers.
    • Mean error: −0.5% to +0.2% on the captured layers (both references), and at most +2.6% on the synthetic layouts.
    • Worst-row error: recomputed on all rows wherever the sampled rows moved by ≥ 5%. It rises by at most 3.2%, with one exception: a single (row, head) vector, out of 12288, rises 12.5% against the exact reference.
      • The vector is in one synthetic layout: a 4096-token chunk after a 16384-token prefix at H3, per-tensor scales, read through kv_slots. The 12.5% is that layout's all-rows maximum, not a sampled statistic.
      • It comes from the wide tiles that prefix chunks now use.
      • The layout's other statistics barely move or improve: mean −0.1%, p99 −1.8%, p99.9 +0.2%, and every FP8-Q statistic improves.
    • Row sums: rows that add the FP8-decoded probabilities run 2–3.4% closer to the FP8-Q reference.
  • BF16.

    • No element is outside 2e-2 of the FP32 reference, in the op test or on the captured layers (12000-token prompt, 64k- and 1M-token contexts; TP2/4/8).
    • Inputs with logits shifted by 2^21 exceed it in ~0.7% of elements, as the base's union kernel does.
  • batch_invariant=True: 88 FP8 and 38 BF16 scenarios. Each request alone equals its rows in the batch bit for bit, and graph replay equals eager.

  • Plan reuse (checked when added): reused plans give the same bits as a full preparation across layout changes, two streams, alternating layers, graph capture and alternating batch_invariant modes. Core/residual plans never reuse.

  • Output-preserving changes: compared bit for bit, with indices shared across processes.

  • SGLang system test (the base vs this PR):

    • Setup.
      • SGLang 464f5d11d serving Qwen3.8-Flash-Next (PTPC FP8 GEMMs, EAGLE speculative decoding with a BF16 draft layer) on MI308X, TP2 unless noted.
      • The two arms differ only in aiter.
      • Each round starts its own server. One untimed warm round per arm is followed by timed rounds in A B B A order.
      • Every round passes a GPU idle gate before and after. A monitor voids rounds that see locked clocks or a non-auto performance level.
    Group Workload This PR vs base Limit
    Throughput, FP8 K/V Customer workload (256 × 12000/350 tokens, concurrency 8); random 12000/350 tokens at concurrency 1/2/4/8 +0.50%; +0.45 / +1.82 / +0.26 / −0.37% ±2%
    Throughput, BF16 K/V Same +0.61%; +0.41 / −1.58 / −1.13 / −0.12% ±2%
    GSM8K, TP2 1319 questions, 5-shot, FP8 / BF16 K/V −0.0008 / +0.0023; acceptance length −0.0006 / +0.0065 ±0.006; ±0.02
    GSM8K, TP8 FP8 K/V +0.0004; acceptance length +0.0070 ±0.006; ±0.02
    Prefill latency, FP8 K/V 2048 tokens at C1/C8, 12000 at C1/C4, 131072 in 8 chunks, 999648 +0.33 / −0.19 / −0.46 / +0.61 / −0.63 / −0.54% > +1% investigate, > +2% fail
    Prefill latency, BF16 K/V Same −0.41 / −0.35 / −0.38 / +0.28 / −0.20 / −0.29% Same
    1M-token correctness (TEST=1) FP8 / BF16 K/V Pass: 7878 checks per rank, no errors —
    Deterministic smoke --enable-deterministic-inference (all 13 QSA layers run batch_invariant=True), GSM8K first 200 questions 0.995 (base: 0.990) ≥ 0.9
    • Which tree. The throughput rows ran on this head's tree. The other rows ran on the earlier tree, whose outputs are bitwise equal (see Reviewed revision).
    • Earlier tree at concurrency 4. It lost 4.6% (FP8 K/V) and 3.5% (BF16 K/V). The cause was the first use of the four-round K3 specialization, about 1.2 s per K/V type, by a chunked prefill's ~3200-row remainder. On this head:
      • a first-use hook in every round records no FlyDSL trace or compile during the timed items;
      • the concurrency-4 p99 TTFT matches the base's (2.54–2.59 s).
    • Throughput differences at concurrency 2–8. Per-step decode time and TTFT are equal in the two arms. The differences come from speculative-decoding acceptance on the 32 random prompts, whose generated tokens differ between the arms. On real text (GSM8K, the customer workload) the acceptance length is equal or higher with this PR.
    • 2048-token prefill at concurrency 1. Re-measured on an idle machine: +0.33% over the three undisturbed pairs. The QSA calls of one request take 1.52 ms on the GPU, against 1.94 ms with the base.
    • Why system-level gains are small. At TP2 the 12 QSA layers of a 12000-token prompt take 14.3 ms (base: 19.4 ms; captured layers, per-tensor FP8 K/V). That is under 2.5% of the ~790 ms prefill, so a 26% faster op gives about 0.5%. The rest of the prefill is the 36 linear-attention layers, the MoE and the indexer.
  • Not verified:

    • MI300X and other non-MI308X gfx942 parts: no hardware was available. Tuning targets MI308X; other gfx942 parts log a warning and reuse the 80-CU settings. Results are unaffected; performance may not be optimal.
    • MI250X: the kernels are gfx942-only, as in the base.
    • batch_invariant=True and the 4 GiB-limit layouts: not timed against the base. The op test covers their correctness.

PR-specific question

What behavior would each changed op test fail on before this PR, and why is that coverage not already provided by an existing canonical test? (the question the self-review rules select for changes under op_tests/)

  • op_tests/test_flydsl_qsa_sparse_prefill.py:

    • The extra call into a NaN-filled out fails when a launch leaves rows unwritten. The existing check only saw run_perftest's last output, which lands in memory that earlier calls' outputs filled, so unwritten rows passed. A mutation that skips the persistent queue's reset passed the whole op test before this PR (it only ran 47× faster) and fails now.
    • fp8_hk8_chunk_300_ctx_30300_token fails if the plain FP8 direct's persistent launch mishandles its queue (items 640–1919). No other FP8 row runs that launch.
    • fp8_uniform_ragged_4_token fails if the key-range parts of an all-union per-token plan merge inexactly. The existing uniform row mixes union and direct tiles.
  • op_tests/test_flydsl_qsa_sparse_prefill_guards.py: each new check fails on one regression that leaves the outputs correct, which no value-comparing test can see:

    • an LLVM option named outside the primitives;
    • a launch without prim.kernel_attrs(), which brings packed FP32 back (10–61% slower);
    • SGPR spill code inside the FP8 direct's MFMA loops;
    • exec narrowing or an extra atomic in the persistent BF16 packed direct;
    • the uniform-branch option silently ignored (56 instead of 42 s_cbranch);
    • an LDS read in flight across a barrier without its counted wait.

    Each check was shown to fail on a matching mutation when it was added.

Changed op tests

  • op_tests/test_flydsl_qsa_sparse_prefill.py (132 → 134 rows):
    • NaN-filled output.
      • After run_perftest, each BF16 functional row calls FlyDSL once more into a NaN-filled out, and must pass the same check.
      • The FP8 rows' repeated call also writes into a NaN-filled out, and must match bit for bit.
      • run_perftest returns its last call's output in memory that earlier calls filled, so rows a later call skipped stayed hidden. A mutation that skips the persistent queue's reset passed the whole op test (it only ran 47× faster), and now fails.
    • fp8_hk8_chunk_300_ctx_30300_token (300 rows at HK8 after a 30000-token prefix). The plain FP8 direct runs persistent, and its queue hands out items 640–1919. No other FP8 row uses that queue.
    • fp8_uniform_ragged_4_token. It runs the uniform-weight exactness check on an all-union plan whose tasks split into two key ranges. The existing uniform row mixes union and direct.
    • Routes. The 12000- and 24576-token FP8 prefills now also pack K, because some tiles route direct.
    • Run time. These changes keep the cold run within 1.5× of main:
      • functional rows make 1 warm-up and 3 timed calls, while performance rows keep the full timing;
      • SGLang's Triton baseline runs only on the performance rows and at group 12.
  • op_tests/test_flydsl_qsa_sparse_prefill_guards.py (16 → 21 checks; 6 → 11 compiled layouts):
    • New layouts: a BF16 split union; the BF16 union and raw direct over a pool past 4 GiB; the persistent packed direct; the FP8 union variant with the most SGPR spills (HK2, T = 768, slot table); per-tensor FP8 without a slot table; and one FP8 union compiled again with the uniform-branch option off.
    • New checks:
      • LLVM options are named only in the primitives.
      • Every launch passes prim.kernel_attrs(), except four listed exemptions: K2, which is slower with it, and the two BF16 K/V packs, which have no FP32 math.
      • The FP8 direct's MFMA loops have no v_readlane/v_writelane.
      • The persistent BF16 packed direct has no exec narrowing, exactly its buffer atomics, and its s_setprio. The FP8 persistent launch is covered by the chunk-loop check.
      • The uniform-branch option removes branches (42 vs 56 s_cbranch) without adding SGPR spills.
    • Updated checks: occupancy (one wave per SIMD for the BF16 union, two for the FP8 union); the lgkmcnt ledger now also covers the FP8 union's reads across barriers; the FP8 MFMA and conversion counts include the row-sum v_smfmac and the FP8 decode.
    • Why these are not redundant: each of these properties only costs speed when it breaks, or breaks silently after a FlyDSL or LLVM upgrade, so the op test cannot see it.

Speed up the QSA sparse prefill on MI308X (gfx942) for BF16 and FP8 K/V.
The entry's signature is unchanged. BF16 outputs stay within the 2e-2
tolerance, FP8 errors stay close to PR2's (on captured Qwen3.8
selections the mean error changes by -0.5% to +0.2%), and
batch_invariant=True keeps every request's bits independent of the
batch (the new splits are off in that mode).

Prepare chain (K1-K3) and planner:
- K3 builds its task list with a counting sort; its order CTA reads
  before it writes and maps entries with a stride.
- Chunks with a prefix and multi-request batches also use the wide
  union tiles.
- A layout whose every request spans at most 4T + 3 tokens (each row
  selects all of its complete blocks) keeps its plan in the stream's
  scratch: while no other call or graph capture intervenes, repeated
  calls skip K1 and K2 (with kv_slots K3 runs again to map the pool
  blocks, otherwise only the split counters are reset). Such calls do
  not read indices.
- K2 keeps each wave's rows' block ids in registers across up to eight
  dense windows; tiles whose rows look random (a row-overlap probe) are
  found over budget from a lower bound in one short pass. Long ranges
  run on 8-wave CTAs, and the unroll is chosen per plan.
- Routing is recalibrated: per-task overheads come from the union
  kernel, each routing step must save route_margin (FP8 error headroom),
  and past a 128k context the direct (K2) or union (K3) cost is
  weighted by context length through runtime parameters.
- Plans with exactly four rounds re-deal their tasks round by round,
  which balances the CTAs at TP8. K3 decides this per launch, so such
  plans need no specialization of their own (a new specialization costs
  about a second on first use in a process, since FlyDSL traces it
  before looking up its disk cache).

FP8 union:
- Two waves per SIMD share 32 rows. CTA barriers delimit slots, and one
  wave's MFMA stage (QK and PV) runs while the other's VALU stage
  (softmax, K/V copies) does, the MFMA stage holding issue priority.
  The next task's loads issue while the current task drains.
- Conversions, exp and row sums are spread over the two MFMA halves;
  the per-tensor row sum runs on v_smfmac and the per-token row sum
  follows exp.
- LDS reads stay in flight across the slot barriers with partial
  lgkmcnt waits counted by prim.LgkmLedger; the guards match each such
  wait in the final ISA.
- Wave-uniform branches are not structurized in any FP8 union variant
  (one LLVM option, set in the primitives).

FP8 direct and K/V pack:
- K is packed token-minor within each MFMA operand group, so each
  4-lane quad of a direct K load reads 64 contiguous bytes; per-token
  scale records hold one 16-byte entry per token.
- The next chunk's loads issue in three groups between the MFMAs.
- Plans that select the full topk for every row run one persistent
  8-wave workgroup per CU whose waves scan the sorted blocks in step,
  keeping K/V in L2.
- Persistent and listed rows stage their output in LDS and write it as
  contiguous 1 KB stores; the residual merge moves the upper 32 dims to
  idle lanes through DPP.
- The row sum adds the FP8-decoded probabilities that PV multiplies,
  which lowers the error against the FP8-Q reference.
- Launch queues are branch-free: lane 0 issues the buffer atomic and the
  other lanes are out of bounds.
- The pack launch dispatches its latency-bound list CTAs first, so the
  K/V and Q CTAs overlap them.

BF16 union and packed direct:
- The union steady state uses 32x32x8 MFMAs with 32 rows per wave,
  halving its LDS reads.
- Short BF16 prefills split their longest union tasks into two key
  ranges; each wave counts its parts with one buffer atomic and the last
  one merges them into the output.
- The packed direct runs one CTA per work item and splits the last ones
  into key-tile parts; full-topk plans use the persistent synchronized
  scan.

Shared:
- prim.kernel_attrs() (packed FP32 ops off) is now also passed by K1,
  K3 and the new persistent and listed launches; K2's compact kernels
  and the BF16 packs are exempt. The launch queue helpers of the
  persistent BF16 and FP8 direct kernels are shared in common.

Tests:
- op_test adds a uniform ragged per-token FP8 batch and a persistent
  FP8 direct case at HK8, checks that one more call writes every row of
  a NaN-filled output, makes fewer timing calls for functional cases and
  runs their Triton baseline at group 12 only.
- New guards check the LDS operations in flight across each barrier
  against the lgkmcnt ledger, the persistent BF16 packed direct launch,
  the FP8 direct inner loops, that only the primitives name LLVM
  options, that the uniform-branch option removes branches without
  adding spills, and that every kernel launch passes prim.kernel_attrs()
  (with listed exemptions).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant