Repository navigation
[FlyDSL] [QSA] Sparse prefill follow-ups: runtime softmax scale, stable FlyDSL API, batch_invariant - #4
Open
luocheng25 wants to merge 7 commits into
Open
[FlyDSL] [QSA] Sparse prefill follow-ups: runtime softmax scale, stable FlyDSL API, batch_invariant#4luocheng25 wants to merge 7 commits into
luocheng25 wants to merge 7 commits into
Conversation
…ment The union, raw direct and packed direct kernels took the softmax scale as a compile-time constant (SCALE: fx.Constexpr[float]), so every new scale compiled all of them again, and a scale first seen under graph capture raised "Warm ... before graph capture" (ROCm#6576 review). The kernels now take SCALE_LOG2: fx.Float32, the host's FP32 rounding of softmax_scale * log2(e). That is the value the constant had, so outputs are unchanged bit for bit. The scale leaves the FlyDSL compile keys and the workspace key, and every call passes it to the cached launchers: a new scale compiles nothing and reuses the layout's workspace, also under graph capture. The entry still rejects scales whose log2 multiple is not a positive finite FP32 value. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Passing the softmax scale as a kernel argument made the union kernel about 4-5% slower (prefill_4096 H12: 536 -> 565 us per launch). The union phase loop already uses every SGPR; with the scale read from one there, LLVM spilled 12 SGPRs at kernel entry and reordered the tuned MFMA/VALU interleave of the phase loop. Each union task now copies the scale into a VGPR (an empty tied asm, as for the other pinned values). The SGPR spills are gone and the phase loops match the compile-time-constant version instruction for instruction, apart from one s_nop placement; outputs are unchanged bit for bit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…y loop _qk and _pv had three copies (union, raw direct, packed direct) and _rescale two, and the raw and packed direct kernels repeated most of their key loop (ROCm#6576 review). Each is now defined once in sparse_prefill_common: - _qk: the QK MFMAs of two 16-key tiles over D = 256 for all three kernels, in the tiled-MMA form the union pipeline was tuned with. The raw direct kernel transposes its K operands first. - _pv: one 16-key PV step; operand(n) supplies each O tile's V operand, so the raw kernel keeps its v_perm transposes next to their MFMAs. The union's softmax-folding wrapper is renamed _pv_half and calls it per step. - _rescale: the lazy rescale of any accumulators by a factor computed only when a lane's maximum moved (union and packed direct). - _read_q_g2r, _v_pair: the Q load and V transpose the kernels had copied. Both direct kernels run _DirectLoop.attend (sparse_prefill_direct_packed): the pipeline of loads, waits, online softmax and PV, with the K/V layout supplied by _Packed or _Raw (V rows, block ids, score masking, reductions, rescale, PV). Outputs are unchanged bit for bit. The union and pack kernels compile to the same ISA; the packed direct loop moves one s_nop; the raw direct QK no longer interleaves its K transposes with the MFMAs, which made the raw direct kernel 4% faster on a 1024-row chunk over a 131072-token context. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The sparse prefill kernel files no longer import flydsl._mlir or ROCDL operations that flydsl.expr.rocdl does not export. What has no stable equivalent moves to sparse_prefill_primitives, each with the reason it stays: the empty inline asm that keeps values in VGPRs, the target-features passthrough, and the ROCDL operations FlyDSL wraps without exporting (lane broadcasts, ballot, ds_bpermute, v_perm_b32, scheduling barriers, raw buffer loads, DPP, mbcnt, s_setprio, s_barrier). - Numeric.shuffle_xor (deprecated) -> fx.gpu.shuffle_xor. - Target features: ir.Attribute.parse -> the llvm.passthrough value attribute. - Union waves per EU: the waves_per_eu compile hint, not a raw kernel attribute. - exp2: fx.math.exp2 with afn, which lowers to the same llvm.amdgcn.exp2. - The preparation's CTA sync: fx.gpu.barrier (release/acquire rather than seq_cst workgroup fences, with the same ISA). - The union pipeline's partial lgkmcnt waits are derived: LdsQueue counts each LDS read and ds_bpermute it issues, checks every wait, and can record the waits (record_lds_waits) for tests that compare them with the ISA. The final ISA of every kernel is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…MI308X warning Sparse prefill picks union tiles or the direct kernel per call: union tiles are planned by the call's row count, BQ21/BQ42 tiles serve only a single full H6/H3 prefill, and K3 routes tiles with a cost model over the whole batch. Union and direct round differently, so the low bits of a request's rows can change with the other requests in the call (ROCm#6576 review). Keep that as the default, now documented, and add a keyword-only batch_invariant=False after kv_slots. With True, every choice that changes the bits is made per request: - only requests of at least the union planning rows get union tiles; the rows of the others get direct-only tiles after them, which K2 does not compact and K3 publishes as direct; - each request picks its own union tile rows (BQ21/BQ42 for its own full prefill); K2, K3 and the union kernel compile for the widest, which gives a narrower tile the same bits; - K3 does not route tiles across the batch (the plan's existing routing switch). A request then gives the bits it gives alone. The modes keep separate workspaces. In both, a layout whose union scratch would exceed 4 GiB still runs direct only. No kernel code changes: the default mode builds the same plans and launches the same kernels with the same arguments as before. Also reword the warning for gfx942 parts without 80 CUs: the tuning targets MI308X only, the 80-CU settings are reused, results are unaffected and performance may not be optimal. Validated on MI308X (gfx942, 80 CUs), FlyDSL 0.5.0: - default mode: 154 outputs of the op_test layouts and 37 multi-request batches equal byte for byte to the previous code; final ISA of all six kernel modules unchanged; device time within +-1% (8 interleaved runs per arm). - batch_invariant=True: every request alone equals its rows of the batch bit for bit in 37 batches (union with packed and raw direct, mixed BQ21/16 and BQ42/32, kv_slots, HK2, group 16, T 256/1024, graph replay); batches match the FP32 reference. - batch_invariant=True costs: single full prefills -4.7% to +0.3%; batches with a long full prefill up to -19% (per-request BQ21/BQ42); H12 1024-row chunks over long contexts +16% to +34% (no global routing); batches of short requests below the union planning rows +4% to +303% (they run direct). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…6 and batch invariance Add six cases to op_tests/test_flydsl_qsa_sparse_prefill.py for the ROCm#6576 follow-ups. Each runs once, at its own group (CASE_GROUPS), whatever -g selects: - Group 16, the most Q heads per KV head the interface accepts: a union request batched with raw-direct rows (H16), and packed direct through a slot table over two KV heads (H32, HK2). - Runtime softmax scale: after one eager call warms a layout, graphs captured at scales no other case uses must replay correctly after the indices, and through a slot table the slots, are rewritten in place. An eager call at another new scale must add no FlyDSL specialization, and scales that round to zero are rejected. - Batch invariance with batch_invariant=True: every request alone must give the bits it gives in one call with the batch's other requests (union planning rows, union tile rows, the BQ21 tiles of a single full prefill). batch_invariance_h12 also runs its batch in the default mode: the output must be within the reference tolerance, and the call must not run on the batch_invariant=True workspace (the workspace cache key holds the mode). latest_workspace() reads the prefill's most recently used workspace; routes() uses it to assert the paths each new case took, and they stay the only helpers reading prefill internals. make_indices accepts one selection mode per request. The Triton QSA code the test copies from SGLang now carries SGLang's Apache-2.0 notice. The copy matches lines 9-309 of sparse_attn.py at SGLang 52fecfdf0908, not lines 9-267 at f2b73d4f as the old comment said; that file has no license header, so the notice is the one SGLang's LICENSE gives. On MI308X (gfx942) all 105 rows pass. Before this series the scale was a compile-time constant in the workspace key, so capturing at a new scale raised "Warm attention on this stream and layout before graph capture", and every decision was taken per batch: a 300-row request batched with a union request (394939 of 921600 elements) or a full H6 prefill batched with an extend (16338 of 6291456 elements) differed from the same request run alone. With the mode dropped from the workspace cache key at run time, batch_invariance_h12 fails ("both modes ran on one workspace"). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…vel code Add op_tests/test_flydsl_qsa_sparse_prefill_guards.py: compile-only, untimed checks that fail when FlyDSL, MLIR or LLVM silently stop doing what the sparse prefill kernels' remaining low-level code (sparse_prefill_primitives.py) relies on. Such failures are otherwise silent: LLVM only prints an asm constraint it cannot meet and the kernel computes garbage, and a dropped attribute, a split load or a merged LDS read only costs speed or leaves a counted wait wrong. One process compiles prefill_12000, kv_slots_ragged_4 and kv_slots_pool_4g_extend_64_ctx_70065 (group 12) with FlyDSL's IR dumps and reads the union, raw direct (also its WIDE variant) and packed direct kernels: - compile output: no "error:" line, finite outputs; - resources: union without scratch or VGPR spill, SGPR spills at most the baseline, two waves per SIMD; the direct kernels without scratch or spill; - "-packed-fp32-ops" and union's waves_per_eu 2 in LLVM IR, no v_pk_*_f32; - the baseline number of empty inline asm blocks and asm constraints; - union's partial lgkmcnt waits as LdsQueue recorded them while tracing equal those an ISA scan finds, with no SMEM load in flight; - the direct loop's hand-counted vmcnt(8) and vmcnt(16): the loads issued on every path into each wait; - every exp2 lowered to llvm.amdgcn.exp2.f32. Without compiling a kernel it checks LdsQueue's assertions, prim.pin rejecting one-word chunks, and that the kernel modules use FlyDSL's stable API only (docs/api_stability.md), allowing the module functions fx.gpu.shuffle_xor and fx.math.exp2. On MI308X (gfx942) all guards pass in about 50 s. Patching the kernels' code at run time makes them fail as intended: dropping union's softmax-scale pin (SGPR spills 0 -> 12, one asm block fewer), dropping the target feature, one-word pins (LLVM: "could not allocate output register for constraint 'v'"), an uncounted ds_bpermute, exp2 without afn (__ocml_exp2_f32), a stricter vmcnt(16), and edited copies of union.py with flydsl._mlir, bare rocdl, unexported or experimental ROCDL paths, deprecated paths and methods, inline asm or llvm.passthrough. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-ups to ROCm#6576 (QSA sparse prefill attention for MI308X), addressing the six non-blocking review notes left there by zhiding512:
qsa_sparse_prefillgains an opt-inbatch_invariant=Truemode that makes every request's output independent of the other requests in the batch. The default behavior is unchanged.In the default mode, outputs are bitwise identical to main and performance is unchanged within ±1% on MI308X.
Motivation
From the review of ROCm#6576:
SCALEwasfx.Constexpr[float], so every newsoftmax_scalerecompiled all kernels (several seconds each). During graph capture, an unseen scale raised "Warm attention on this stream and layout before graph capture"._qk/_pvexisted three times and_rescaletwice, and the direct and direct-packed main loops were about 80% the same.flydsl._mlirir/llvm APIs, empty inline asm to pin registers, and hand-countedlgkmcntwaits. These can silently miscompute after a FlyDSL or LLVM upgrade. We reproduced one such silent failure during this work: a pin LLVM cannot allocate only prints an error, and the kernel then returns non-finite values.Changes
SCALE_LOG2: fx.Float32, computed on the host in double precision and rounded once to FP32, which is the value the constant used to have. The scale is no longer part of the compile key or the workspace key. A new scale needs no compilation and can be captured in a graph directly. The union kernel keeps the scale in a VGPR pinned with an empty asm; passing it in an SGPR spilled 12 SGPRs at entry and made union up to 4.3% slower. Scales that round to zero are still rejected._rescale,_read_q_g2r,_qkand_pvare defined once insparse_prefill_common.py, and the direct and direct-packed kernels share their key loop.flydsl._mlir.fx.gpu.shuffle_xor,fx.math.exp2with afn,fx.gpu.barrier(), and the publicwaves_per_eucompile hint.sparse_prefill_primitives.py, each item with the reason it stays: empty-asm register pins, the-packed-fp32-opstarget feature (viallvm.passthrough), and unexported ROCDL operations.prim.pinnow raises on chunks LLVM cannot allocate.LdsQueueledger that counts the outstanding LDS operations and asserts on misuse, instead of hand-countedlgkmcntvalues.batch_invariant=False, a new trailing keyword-only argument ofqsa_sparse_prefill.batch_invariant=True, these decisions are taken per request: the union planning rows threshold, the union tile rows (BQ21/BQ42 per request; union is compiled for the widest BQ in the batch), and no global K3 routing. Each request then gives the same bits alone as in any batch.op_tests/test_flydsl_qsa_sparse_prefill.py.op_tests/test_flydsl_qsa_sparse_prefill_guards.py.aiter.ops.qsa.qsa_sparse_prefill, get the same outputs and performance.Performance (if applicable)
Default mode, before (main 66be4f6) vs after this PR. MI308X (gfx942, 80 CUs), FlyDSL 0.5.0. op test timing (
run_perftest, 101 iterations, rotating inputs), median of 8 interleaved ABBA pairs in separate processes.Every step of the series was also checked against the step before on all 21 rows of the op test's performance suite, with medians of at least 8 ABBA pairs. All rows are within ±1%, except one row of the shared-code step that is 1.15% faster; per-kernel timing showed that row to be noise.
batch_invariant=Trueis opt-in. Its cost relative to the default mode on the same tree, as medians of 8 ABBA pairs:The last two rows are why per-request decisions are not the default.
Testing
Documentation
Dependencies
Breaking Changes
None.
qsa_sparse_prefillonly gains the trailing keyword-onlybatch_invariant=False, and the default behavior is bitwise unchanged.PR Self-Review
Reviewed revision
a63545c9550b8b67a8d38faa9b13439710c808b5Behavior and intent
batch_invariant=Truegives batch-invariant outputs for callers that need determinism, such as SGLang's--enable-deterministic-inference. It is not the default because per-request decisions make some batches much slower (see Performance).Validation and unverified scope
All results below are on MI308X (gfx942, 80 CUs) with FlyDSL 0.5.0 and torch 2.12.0+rocm7.14.
python op_tests/test_flydsl_qsa_sparse_prefill.py: 105 rows, all pass, none skipped for memory; 237checkAllclosecomparisons pass. Cold-cache runtime is 288.4 s, against 276.0 s for the 99 rows on main in the same session on the same GPU (+4.5%).python op_tests/test_flydsl_qsa_sparse_prefill_guards.py: 11 checks pass, 51.4 s. Patching the kernels at run time makes the guards fail as intended:flydsl._mlir, inline asm or deprecated paths.batch_invariant=True). 37 batches plus one graph case: every request run alone is bitwise equal to its rows in the batch.qsa_sparse_prefill:vmcntwaits. These only affect wait placement; the guard test checks them against the ISA.PR-specific question
What behavior would each changed op test fail on before this PR, and why is that coverage not already provided by an existing canonical test? (the question the self-review rules select for changes under
op_tests/)op_tests/test_flydsl_qsa_sparse_prefill.py:batch_invariant=True, a mode this PR adds. No existing row compared a request alone with the same request in a batch.routes()asserts each new row's path, so a routing change cannot silently drop this coverage.op_tests/test_flydsl_qsa_sparse_prefill_guards.py(new) fails on regressions that leave the op test's outputs correct, or break them only after a FlyDSL or LLVM upgrade. Each was shown to fail on a matching run-time patch:v_pk_*_f32back);LdsQueueledger does not count;vmcnt;flydsl._mlir, inline asm or deprecated paths.Changed op tests
op_tests/test_flydsl_qsa_sparse_prefill.py: six new rows, each run once at its own group.batch_invariant=True, covering the union planning rows and the BQ21 tiles of a single full prefill. The H12 case also checks the default mode's tolerance and that the two modes use separate workspaces. No existing row compared a request alone with the same request in a batch.routes()helper asserts the path each new row takes, so a routing change cannot silently drop coverage.sparse_attn.pyat SGLang 52fecfdf0908, which also corrects the old comment.op_tests/test_flydsl_qsa_sparse_prefill_guards.py(new): compile-only and untimed. It checks what the remaining low-level code relies on:v_pk_*_f32;LdsQueueledger against thelgkmcntwaits in the ISA;vmcntcounts against the loads on every path;These failures would not show up as wrong outputs in the op test, or only as garbage after a FlyDSL or LLVM upgrade, which is the concern of review note 3.