Repository navigation
[FlyDSL] [QSA] Optimize the sparse prefill kernels on gfx942 - #3
Open
luocheng25 wants to merge 1 commit into
Open
luocheng25 wants to merge 1 commit into
luocheng25 wants to merge 1 commit into
Conversation
Speed up the QSA sparse prefill on MI308X (gfx942) for BF16 and FP8 K/V. The entry's signature is unchanged. BF16 outputs stay within the 2e-2 tolerance, FP8 errors stay close to PR2's (on captured Qwen3.8 selections the mean error changes by -0.5% to +0.2%), and batch_invariant=True keeps every request's bits independent of the batch (the new splits are off in that mode). Prepare chain (K1-K3) and planner: - K3 builds its task list with a counting sort; its order CTA reads before it writes and maps entries with a stride. - Chunks with a prefix and multi-request batches also use the wide union tiles. - A layout whose every request spans at most 4T + 3 tokens (each row selects all of its complete blocks) keeps its plan in the stream's scratch: while no other call or graph capture intervenes, repeated calls skip K1 and K2 (with kv_slots K3 runs again to map the pool blocks, otherwise only the split counters are reset). Such calls do not read indices. - K2 keeps each wave's rows' block ids in registers across up to eight dense windows; tiles whose rows look random (a row-overlap probe) are found over budget from a lower bound in one short pass. Long ranges run on 8-wave CTAs, and the unroll is chosen per plan. - Routing is recalibrated: per-task overheads come from the union kernel, each routing step must save route_margin (FP8 error headroom), and past a 128k context the direct (K2) or union (K3) cost is weighted by context length through runtime parameters. - Plans with exactly four rounds re-deal their tasks round by round, which balances the CTAs at TP8. K3 decides this per launch, so such plans need no specialization of their own (a new specialization costs about a second on first use in a process, since FlyDSL traces it before looking up its disk cache). FP8 union: - Two waves per SIMD share 32 rows. CTA barriers delimit slots, and one wave's MFMA stage (QK and PV) runs while the other's VALU stage (softmax, K/V copies) does, the MFMA stage holding issue priority. The next task's loads issue while the current task drains. - Conversions, exp and row sums are spread over the two MFMA halves; the per-tensor row sum runs on v_smfmac and the per-token row sum follows exp. - LDS reads stay in flight across the slot barriers with partial lgkmcnt waits counted by prim.LgkmLedger; the guards match each such wait in the final ISA. - Wave-uniform branches are not structurized in any FP8 union variant (one LLVM option, set in the primitives). FP8 direct and K/V pack: - K is packed token-minor within each MFMA operand group, so each 4-lane quad of a direct K load reads 64 contiguous bytes; per-token scale records hold one 16-byte entry per token. - The next chunk's loads issue in three groups between the MFMAs. - Plans that select the full topk for every row run one persistent 8-wave workgroup per CU whose waves scan the sorted blocks in step, keeping K/V in L2. - Persistent and listed rows stage their output in LDS and write it as contiguous 1 KB stores; the residual merge moves the upper 32 dims to idle lanes through DPP. - The row sum adds the FP8-decoded probabilities that PV multiplies, which lowers the error against the FP8-Q reference. - Launch queues are branch-free: lane 0 issues the buffer atomic and the other lanes are out of bounds. - The pack launch dispatches its latency-bound list CTAs first, so the K/V and Q CTAs overlap them. BF16 union and packed direct: - The union steady state uses 32x32x8 MFMAs with 32 rows per wave, halving its LDS reads. - Short BF16 prefills split their longest union tasks into two key ranges; each wave counts its parts with one buffer atomic and the last one merges them into the output. - The packed direct runs one CTA per work item and splits the last ones into key-tile parts; full-topk plans use the persistent synchronized scan. Shared: - prim.kernel_attrs() (packed FP32 ops off) is now also passed by K1, K3 and the new persistent and listed launches; K2's compact kernels and the BF16 packs are exempt. The launch queue helpers of the persistent BF16 and FP8 direct kernels are shared in common. Tests: - op_test adds a uniform ragged per-token FP8 batch and a persistent FP8 direct case at HK8, checks that one more call writes every row of a NaN-filled output, makes fewer timing calls for functional cases and runs their Triton baseline at group 12 only. - New guards check the LDS operations in flight across each barrier against the lgkmcnt ledger, the persistent BF16 packed direct launch, the FP8 direct inner loops, that only the primitives name LLVM options, that the uniform-branch option removes branches without adding spills, and that every kernel launch passes prim.kernel_attrs() (with listed exemptions). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Speeds up the FlyDSL QSA sparse prefill (
aiter.ops.qsa.qsa_sparse_prefill) on MI308X (gfx942), for BF16 and FP8 K/V. The signature and the documented contract are unchanged.kv_slots, TP2, per-tensor FP8) drops from 1709.7 to 1251.5 µs (−26.8%).batch_invariant=Trueturns every new split off.indices.This PR targets
luocheng/qsa-fp8(#5, head43ce3d6bf), the QSA FP8 port that follows ROCm#6576 and #4, so its diff holds only these changes; "the base" below means that branch. The changes keep to the rules of ROCm#6576:flydsl._mliroutsidesparse_prefill_primitives.py;batch_invariant=Truecovers every new path;Motivation
SGLang serves Qwen3.8-Flash-Next prefill through this op, one call per QSA layer. With the base, the kernels ran well below what MI308X sustains (measured with the same tooling as the tables below):
Only 2 of the 48 must-reach kernel items reached 80%.
Changes
sparse_prefill_prepare.py, entry):indices. The plan is then reused while the stream's scratch still holds it:kv_slots; split plans only re-zero their counters.route_margin(5%), because direct rows run further from the FP8-Q reference.prim.kernel_attrs(), like the other kernels did in the base; K2 and the two BF16 K/V packs stay exempt.sparse_prefill_union_fp8.py):v_smfmac_f32_16x16x64_fp8_fp8, over the FP8 probabilities PV multiplies.prim.LgkmLedgercounts LDS reads in flight across barriers. One LLVM option, set in the primitives, keeps uniform branches unstructurized in every FP8 union variant.sparse_prefill_direct_fp8.py):(g·4 + token)·16; the base:(token·4 + g)·16), so a wave's 1 KiB load covers whole 64-byte lines. Per-token scale records hold one 16-byte entry per token (base: one 16-byte array per term).sparse_prefill_union.py,sparse_prefill_direct_packed.py,sparse_prefill_direct.py):sparse_prefill_common.py,sparse_prefill_primitives.py):Persistent helpers. The persistent pacing and queue helpers live here. Each queue is a workspace tensor that its launch's last wave zeroes.
New primitives, each with its reason in the module docstring:
smfmac_fp8,LgkmLedger(replacesLdsQueue),uniform_branch_options,buffer_atomic,dpp_banks,fp8_pair,store_lds,buffer_load_x3,lane_id.One QK/PV helper per MFMA shape, on purpose, since each shape has its own fragment layout:
common._qk/_pv: BF16 16x16x16, used by both BF16 direct kernels;union._qk32/_pv32: BF16 32x32x8;direct_fp8._qk_half_fp8/_pv16: FP8 16x16x32;The FP8 MFMA call,
common._mfma_fp8, is shared.Performance (if applicable)
Whole
qsa_sparse_prefillcall (K1–K3, pack and attention kernels).kv_slots(285312-row pool, random 64-token pages; per-tensor FP8 with unit scales, as SGLang stores it), TP2Kernel utilization is max(compute, memory), against peaks measured on this machine:
v_smfmacis excluded.The 21 items below 80%:
v_fmac_f32.kv_slots. 1 item at 79.1%, 0.9 points short.Captured SGLang layers, the base vs this PR.
kv_slots: with unit per-tensor scales (how SGLang stores FP8 K/V), the 12000-token prompt is 25.6% / 25.0% / 24.6% faster at TP2/4/8.Testing
Documentation
Dependencies
Breaking Changes
None.
indicesfollow the documented layout.PR Self-Review
Reviewed revision
589f338fecc5cff12d6b8e7fb3d97f323fc8dba4Behavior and intent
Outputs change in low-order bits: the BF16 union accumulates with 32x32x8 MFMAs, routes are recalibrated, and FP8 row sums add the probabilities PV multiplies.
Plan reuse (new). The entry docstring states: "A layout whose every request spans at most 4T + 3 tokens (each row then selects all of its complete blocks) reuses its plan instead while no other call on the stream, nor a graph capturing its scratch, ran since its last call; such calls do not read indices."
indiceslayout leaves exactly one valid selection.batch_invariant=True:Short prefills lose the split's gain, for example about 7% on a 2048-token BF16 prefill at TP4.
Interface. There are no new arguments, environment variables or platform fields. Only the FP8 route terms are refit.
Compilation. New compile-time parameters (K2's unroll; K3's balance, sort-width and mask-partition flags; the BF16 split; the persistent and split direct launches) are part of the compile keys, and each variant compiles on first use. The softmax scale and the long-context weights (
SPAN_SLOPE,UNION_STEP) are FP32 kernel arguments.First use in a process.
Persistent state.
Tuning. The constants are calibrated on MI308X. Other gfx942 parts log a warning and reuse them.
Considered and not adopted:
s_memrealtime/s_sleep). They gain 2–3% on random full-topk selections, but they depend on the previous launch's timing, have a performance cliff, and help only synthetic loads.Follow-ups:
v_fmac_f32inGCNDPPCombinewould gain ~2% of the per-token FP8 union.indicesinterface would cut K1 traffic to ~1/4. This is an API change.Validation and unverified scope
Results are on MI308X (gfx942, 80 CUs) with FlyDSL 0.5.0. The op test, guards and lint below ran on this head.
Tested trees. This head is a single commit on the base's head
43ce3d6bf. The benchmarks and the accuracy audits below were measured on an earlier tree, which differs from this head in two ways:Three docstrings. Once docstrings are removed, the Python ASTs of the two trees are equal.
One K3 change (
sparse_prefill_prepare.py, +41/−25). Four-round plans no longer compile a K3 specialization of their own (see Behavior and intent). It was verified on this head:python op_tests/test_flydsl_qsa_sparse_prefill.py: 134 rows (base: 132), all pass.checkAllclosecomparisons, none skipped.python op_tests/test_flydsl_qsa_sparse_prefill_guards.py: 21 checks pass (base: 16).Lint:
black --check(26.10.1) andruff check(0.16.0) on the 11 changed files are clean.FP8 error vs the base.
kv_slots, HK2, sink, large-logit and Q-past-2-GiB layouts;kv_slots. The 12.5% is that layout's all-rows maximum, not a sampled statistic.BF16.
batch_invariant=True: 88 FP8 and 38 BF16 scenarios. Each request alone equals its rows in the batch bit for bit, and graph replay equals eager.Plan reuse (checked when added): reused plans give the same bits as a full preparation across layout changes, two streams, alternating layers, graph capture and alternating
batch_invariantmodes. Core/residual plans never reuse.Output-preserving changes: compared bit for bit, with indices shared across processes.
SGLang system test (the base vs this PR):
464f5d11dserving Qwen3.8-Flash-Next (PTPC FP8 GEMMs, EAGLE speculative decoding with a BF16 draft layer) on MI308X, TP2 unless noted.--enable-deterministic-inference(all 13 QSA layers runbatch_invariant=True), GSM8K first 200 questionsNot verified:
batch_invariant=Trueand the 4 GiB-limit layouts: not timed against the base. The op test covers their correctness.PR-specific question
What behavior would each changed op test fail on before this PR, and why is that coverage not already provided by an existing canonical test? (the question the self-review rules select for changes under
op_tests/)op_tests/test_flydsl_qsa_sparse_prefill.py:outfails when a launch leaves rows unwritten. The existing check only sawrun_perftest's last output, which lands in memory that earlier calls' outputs filled, so unwritten rows passed. A mutation that skips the persistent queue's reset passed the whole op test before this PR (it only ran 47× faster) and fails now.fp8_hk8_chunk_300_ctx_30300_tokenfails if the plain FP8 direct's persistent launch mishandles its queue (items 640–1919). No other FP8 row runs that launch.fp8_uniform_ragged_4_tokenfails if the key-range parts of an all-union per-token plan merge inexactly. The existing uniform row mixes union and direct tiles.op_tests/test_flydsl_qsa_sparse_prefill_guards.py: each new check fails on one regression that leaves the outputs correct, which no value-comparing test can see:prim.kernel_attrs(), which brings packed FP32 back (10–61% slower);s_cbranch);Each check was shown to fail on a matching mutation when it was added.
Changed op tests
op_tests/test_flydsl_qsa_sparse_prefill.py(132 → 134 rows):run_perftest, each BF16 functional row calls FlyDSL once more into a NaN-filledout, and must pass the same check.out, and must match bit for bit.run_perftestreturns its last call's output in memory that earlier calls filled, so rows a later call skipped stayed hidden. A mutation that skips the persistent queue's reset passed the whole op test (it only ran 47× faster), and now fails.fp8_hk8_chunk_300_ctx_30300_token(300 rows at HK8 after a 30000-token prefix). The plain FP8 direct runs persistent, and its queue hands out items 640–1919. No other FP8 row uses that queue.fp8_uniform_ragged_4_token. It runs the uniform-weight exactness check on an all-union plan whose tasks split into two key ranges. The existing uniform row mixes union and direct.op_tests/test_flydsl_qsa_sparse_prefill_guards.py(16 → 21 checks; 6 → 11 compiled layouts):prim.kernel_attrs(), except four listed exemptions: K2, which is slower with it, and the two BF16 K/V packs, which have no FP32 math.v_readlane/v_writelane.s_setprio. The FP8 persistent launch is covered by the chunk-loop check.s_cbranch) without adding SGPR spills.v_smfmacand the FP8 decode.