Skip to content

[FlyDSL] [QSA] Sparse prefill follow-ups: runtime softmax scale, stable FlyDSL API, batch_invariant - #4

Open
luocheng25 wants to merge 7 commits into
mainfrom
luocheng/qsa-prefill-followups
Open

luocheng25 wants to merge 7 commits into
mainfrom
luocheng/qsa-prefill-followups

Conversation

@luocheng25

Copy link
Copy Markdown
Owner

Summary

Follow-ups to ROCm#6576 (QSA sparse prefill attention for MI308X), addressing the six non-blocking review notes left there by zhiding512:

  1. The softmax scale is a runtime kernel argument instead of a compile-time constant.
  2. The QK, PV and rescale code and the direct key loop exist once and are shared.
  3. The kernels use FlyDSL's stable API. The remaining low-level code is gathered in one module, and new compile-only guard tests fail when it stops working.
  4. The tuning-only-for-MI308X warning is explicit, and group 16 (the largest group the interface accepts) is tested.
  5. qsa_sparse_prefill gains an opt-in batch_invariant=True mode that makes every request's output independent of the other requests in the batch. The default behavior is unchanged.
  6. The Triton QSA code copied from SGLang into the op test keeps SGLang's Apache-2.0 notice.

In the default mode, outputs are bitwise identical to main and performance is unchanged within ±1% on MI308X.

Motivation

From the review of ROCm#6576:

  • Scale as a compile-time constant. SCALE was fx.Constexpr[float], so every new softmax_scale recompiled all kernels (several seconds each). During graph capture, an unseen scale raised "Warm attention on this stream and layout before graph capture".
  • Duplicated code. _qk/_pv existed three times and _rescale twice, and the direct and direct-packed main loops were about 80% the same.
  • Low-level dependencies. The kernels used flydsl._mlir ir/llvm APIs, empty inline asm to pin registers, and hand-counted lgkmcnt waits. These can silently miscompute after a FlyDSL or LLVM upgrade. We reproduced one such silent failure during this work: a pin LLVM cannot allocate only prints an error, and the kernel then returns non-finite values.
  • Tuning tied to MI308X. Other gfx942 parts only got a terse warning. Group 16 was accepted by the interface but untested.
  • Batch-dependent rounding. Whether a tile takes the union or the direct path depends on the whole batch, so the low bits of a request's output could change with the other requests in the batch.
  • License notice. The copied SGLang code carried only AMD's MIT header.

Changes

  • Runtime softmax scale. The kernels take SCALE_LOG2: fx.Float32, computed on the host in double precision and rounded once to FP32, which is the value the constant used to have. The scale is no longer part of the compile key or the workspace key. A new scale needs no compilation and can be captured in a graph directly. The union kernel keeps the scale in a VGPR pinned with an empty asm; passing it in an SGPR spilled 12 SGPRs at entry and made union up to 4.3% slower. Scales that round to zero are still rejected.
  • Shared kernel code. _rescale, _read_q_g2r, _qk and _pv are defined once in sparse_prefill_common.py, and the direct and direct-packed kernels share their key loop.
  • Stable FlyDSL API.
    • Kernel files no longer import flydsl._mlir.
    • Replacements: fx.gpu.shuffle_xor, fx.math.exp2 with afn, fx.gpu.barrier(), and the public waves_per_eu compile hint.
    • Everything without a stable equivalent is gathered in the new sparse_prefill_primitives.py, each item with the reason it stays: empty-asm register pins, the -packed-fp32-ops target feature (via llvm.passthrough), and unexported ROCDL operations.
    • prim.pin now raises on chunks LLVM cannot allocate.
    • The union pipeline's cross-wave LDS waits come from an LdsQueue ledger that counts the outstanding LDS operations and asserts on misuse, instead of hand-counted lgkmcnt values.
  • batch_invariant=False, a new trailing keyword-only argument of qsa_sparse_prefill.
    • The default keeps the whole-batch decisions, and the docstring now states that the low bits may then depend on the other requests in the batch.
    • With batch_invariant=True, these decisions are taken per request: the union planning rows threshold, the union tile rows (BQ21/BQ42 per request; union is compiled for the widest BQ in the batch), and no global K3 routing. Each request then gives the same bits alone as in any batch.
    • In both modes, a batch whose union scratch would exceed the budget runs direct for the whole batch. This is a memory-safety fallback for batches of about a million rows, and the docstring states it.
    • The workspace cache key includes the mode.
  • Non-MI308X warning. The warning now reads: "QSA sparse attention is tuned only for MI308X (gfx942, 80 CUs); this gfx942 has %d CUs and reuses the 80-CU MI308X settings: results are unaffected, performance may not be optimal". The entry docstring says the same.
  • Tests.
    • Six new rows in op_tests/test_flydsl_qsa_sparse_prefill.py.
    • A new compile-only op_tests/test_flydsl_qsa_sparse_prefill_guards.py.
    • The SGLang notice in the op test (see "Changed op tests").
  • Impact on users. No breaking change. Existing callers, including ATOM's call to aiter.ops.qsa.qsa_sparse_prefill, get the same outputs and performance.

Performance (if applicable)

Default mode, before (main 66be4f6) vs after this PR. MI308X (gfx942, 80 CUs), FlyDSL 0.5.0. op test timing (run_perftest, 101 iterations, rotating inputs), median of 8 interleaved ABBA pairs in separate processes.

Hardware Workload/configuration Before After Change
MI308X prefill_4096, H12 676.39 µs 676.93 µs +0.08%
MI308X prefill_12000, H12 2463.43 µs 2463.93 µs +0.02%
MI308X prefill_12000, H3 1068.33 µs 1065.43 µs −0.27%
MI308X prefill_32768, H6 8431.46 µs 8435.18 µs +0.04%
MI308X 5-request mixed batch (300/1024/0/64/1001 rows), H12 679.41 µs 678.56 µs −0.12%
MI308X chunk_768_ctx_8386560, H12 (K/V each at 4 GiB − 1 MiB) 474.44 µs 474.67 µs +0.05%

Every step of the series was also checked against the step before on all 21 rows of the op test's performance suite, with medians of at least 8 ABBA pairs. All rows are within ±1%, except one row of the shared-code step that is 1.15% faster; per-kernel timing showed that row to be noise.

batch_invariant=True is opt-in. Its cost relative to the default mode on the same tree, as medians of 8 ABBA pairs:

Hardware Workload/configuration Default batch_invariant=True Change
MI308X Single full prefills (performance suite, 21 rows) — — −4.7% to +0.3%
MI308X Full prefill 12000 + short requests, H3 (BQ42 per request) 1586.97 µs 1279.52 µs −19.4%
MI308X Two full prefills, H6 500.24 µs 426.67 µs −14.7%
MI308X chunk_1024_ctx_132096, H12 (no global routing) 292.13 µs 384.24 µs +31.5%
MI308X 8 × 256-row extends, 2k–16k prefix, H3 (below the union planning rows: direct only) 150.65 µs 606.73 µs +302.8%

The last two rows are why per-request decisions are not the default.

Testing

  • Unit tests added or updated
  • Performance benchmarks run
  • Tested on MI250X
  • Tested on MI300X

Documentation

  • Docstrings updated
  • User guide updated
  • Performance guide updated
  • No documentation change is needed, with the reason stated below

Dependencies

  • No new third-party dependencies added
  • New dependencies are listed and justified below

Breaking Changes

None. qsa_sparse_prefill only gains the trailing keyword-only batch_invariant=False, and the default behavior is bitwise unchanged.

PR Self-Review

Reviewed revision

a63545c9550b8b67a8d38faa9b13439710c808b5

Behavior and intent

  • Default mode. Behavior is unchanged: same outputs bit for bit and the same performance. The visible differences are:
    • a new softmax scale no longer compiles kernels and can be graph-captured directly;
    • the warning on non-MI308X gfx942 parts is explicit.
  • batch_invariant=True gives batch-invariant outputs for callers that need determinism, such as SGLang's --enable-deterministic-inference. It is not the default because per-request decisions make some batches much slower (see Performance).

Validation and unverified scope

All results below are on MI308X (gfx942, 80 CUs) with FlyDSL 0.5.0 and torch 2.12.0+rocm7.14.

  • python op_tests/test_flydsl_qsa_sparse_prefill.py: 105 rows, all pass, none skipped for memory; 237 checkAllclose comparisons pass. Cold-cache runtime is 288.4 s, against 276.0 s for the 99 rows on main in the same session on the same GPU (+4.5%).
  • python op_tests/test_flydsl_qsa_sparse_prefill_guards.py: 11 checks pass, 51.4 s. Patching the kernels at run time makes the guards fail as intended:
    • dropping union's scale pin (SGPR spills 0 → 12);
    • dropping the target feature;
    • one-word pins (LLVM "could not allocate output register");
    • an uncounted ds_bpermute;
    • exp2 without afn;
    • a stricter vmcnt;
    • edited sources with flydsl._mlir, inline asm or deprecated paths.
  • Bitwise comparison with main (default mode). Inputs are generated by the op test, and each tree runs in its own process with a cold cache. 195 cases and 238 outputs are byte-identical:
    • the functional rows, and the kv_slots rows including pools over 4 GiB;
    • the performance rows;
    • 37 mixed batches;
    • 9 path-covering layouts × 7 softmax scales.
  • Batch invariance (batch_invariant=True). 37 batches plus one graph case: every request run alone is bitwise equal to its rows in the batch.
  • SGLang end to end. Qwen3.8-Flash-Next, the customer TP2 configuration (1M context, NEXTN speculative decoding, BF16 KV), GSM8K 1319 questions 5-shot, comparing the PyHIP QSA prefill with aiter's qsa_sparse_prefill:
    • GSM8K 0.9775 vs 0.9783, accept length 3.1274 vs 3.1307;
    • customer workload and random 12000/350 C1–C8 throughput within −0.98% to +0.26%.
    • This comparison ran on main. This PR's default mode is bitwise identical to main and within ±1% in performance, so the result carries over.
  • Not verified:
    • Other gfx942 parts such as MI300X (no hardware). They get the warning above; results are unaffected.
    • MI250X: the kernels are gfx942-only.
    • The union-scratch-budget fallback (it needs about a million rows).
    • The decode kernels from [FlyDSL] Add fused QSA sparse decode for gfx950 ROCm/aiter#6670 are not touched.
    • The direct loop keeps two hand-counted vmcnt waits. These only affect wait placement; the guard test checks them against the ISA.

PR-specific question

What behavior would each changed op test fail on before this PR, and why is that coverage not already provided by an existing canonical test? (the question the self-review rules select for changes under op_tests/)

  • op_tests/test_flydsl_qsa_sparse_prefill.py:
    • Runtime scale rows fail on main. A graph captured at a scale no earlier call used raised "Warm attention on this stream and layout before graph capture", and an eager call at a new scale compiled new specializations. No existing row changed the scale.
    • Batch invariance rows fail if a request's bits depend on the rest of the batch with batch_invariant=True, a mode this PR adds. No existing row compared a request alone with the same request in a batch.
    • Group 16 rows fail if the largest group the interface accepts breaks (a 16-head direct wave, BQ8 union tiles). The interface accepted group 16, but no row ran it.
    • routes() asserts each new row's path, so a routing change cannot silently drop this coverage.
  • op_tests/test_flydsl_qsa_sparse_prefill_guards.py (new) fails on regressions that leave the op test's outputs correct, or break them only after a FlyDSL or LLVM upgrade. Each was shown to fail on a matching run-time patch:
    • a register pin LLVM cannot allocate (LLVM only prints an error, and the kernel returns non-finite values);
    • union's scale pin dropped (SGPR spills 0 → 12, up to 4.3% slower);
    • the target feature dropped (v_pk_*_f32 back);
    • an LDS operation the LdsQueue ledger does not count;
    • exp2 without afn, or a stricter vmcnt;
    • kernel sources using flydsl._mlir, inline asm or deprecated paths.

Changed op tests

  • op_tests/test_flydsl_qsa_sparse_prefill.py: six new rows, each run once at its own group.

    • Group 16 (two rows): a union request batched with raw-direct rows (H16), and packed direct through a slot table over two KV heads (H32, HK2). No existing row used group 16.
    • Runtime scale (two rows): graphs captured at scales no other case uses replay correctly after the indices or slots are rewritten in place. An eager call at a new scale adds no FlyDSL specialization, and scales that round to zero are rejected. No existing row changed the scale.
    • Batch invariance (two rows): each request alone matches its rows in a batch with batch_invariant=True, covering the union planning rows and the BQ21 tiles of a single full prefill. The H12 case also checks the default mode's tolerance and that the two modes use separate workspaces. No existing row compared a request alone with the same request in a batch.
    • A routes() helper asserts the path each new row takes, so a routing change cannot silently drop coverage.
    • The SGLang Apache-2.0 notice is added to the copied Triton code. The copy matches lines 9–309 of sparse_attn.py at SGLang 52fecfdf0908, which also corrects the old comment.
  • op_tests/test_flydsl_qsa_sparse_prefill_guards.py (new): compile-only and untimed. It checks what the remaining low-level code relies on:

    • no compiler errors;
    • scratch, spills and occupancy;
    • the target feature and the absence of v_pk_*_f32;
    • the number of empty asm blocks;
    • the LdsQueue ledger against the lgkmcnt waits in the ISA;
    • the direct loop's vmcnt counts against the loads on every path;
    • the lowering of exp2;
    • that kernel sources use only FlyDSL's stable API.

    These failures would not show up as wrong outputs in the op test, or only as garbage after a FlyDSL or LLVM upgrade, which is the concern of review note 3.

luocheng25 and others added 7 commits November 12, 2026 14:48
…ment

The union, raw direct and packed direct kernels took the softmax scale as a
compile-time constant (SCALE: fx.Constexpr[float]), so every new scale
compiled all of them again, and a scale first seen under graph capture raised
"Warm ... before graph capture" (ROCm#6576 review).

The kernels now take SCALE_LOG2: fx.Float32, the host's FP32 rounding of
softmax_scale * log2(e). That is the value the constant had, so outputs are
unchanged bit for bit. The scale leaves the FlyDSL compile keys and the
workspace key, and every call passes it to the cached launchers: a new scale
compiles nothing and reuses the layout's workspace, also under graph capture.
The entry still rejects scales whose log2 multiple is not a positive finite
FP32 value.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Passing the softmax scale as a kernel argument made the union kernel about
4-5% slower (prefill_4096 H12: 536 -> 565 us per launch). The union phase loop
already uses every SGPR; with the scale read from one there, LLVM spilled 12
SGPRs at kernel entry and reordered the tuned MFMA/VALU interleave of the
phase loop.

Each union task now copies the scale into a VGPR (an empty tied asm, as for
the other pinned values). The SGPR spills are gone and the phase loops match
the compile-time-constant version instruction for instruction, apart from one
s_nop placement; outputs are unchanged bit for bit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…y loop

_qk and _pv had three copies (union, raw direct, packed direct) and _rescale
two, and the raw and packed direct kernels repeated most of their key loop
(ROCm#6576 review).

Each is now defined once in sparse_prefill_common:
- _qk: the QK MFMAs of two 16-key tiles over D = 256 for all three kernels,
  in the tiled-MMA form the union pipeline was tuned with. The raw direct
  kernel transposes its K operands first.
- _pv: one 16-key PV step; operand(n) supplies each O tile's V operand, so
  the raw kernel keeps its v_perm transposes next to their MFMAs. The union's
  softmax-folding wrapper is renamed _pv_half and calls it per step.
- _rescale: the lazy rescale of any accumulators by a factor computed only
  when a lane's maximum moved (union and packed direct).
- _read_q_g2r, _v_pair: the Q load and V transpose the kernels had copied.

Both direct kernels run _DirectLoop.attend (sparse_prefill_direct_packed): the
pipeline of loads, waits, online softmax and PV, with the K/V layout supplied by
_Packed or _Raw (V rows, block ids, score masking, reductions, rescale, PV).

Outputs are unchanged bit for bit. The union and pack kernels compile to the
same ISA; the packed direct loop moves one s_nop; the raw direct QK no longer
interleaves its K transposes with the MFMAs, which made the raw direct kernel
4% faster on a 1024-row chunk over a 131072-token context.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The sparse prefill kernel files no longer import flydsl._mlir or ROCDL operations
that flydsl.expr.rocdl does not export. What has no stable equivalent moves to
sparse_prefill_primitives, each with the reason it stays: the empty inline asm that
keeps values in VGPRs, the target-features passthrough, and the ROCDL operations
FlyDSL wraps without exporting (lane broadcasts, ballot, ds_bpermute, v_perm_b32,
scheduling barriers, raw buffer loads, DPP, mbcnt, s_setprio, s_barrier).

- Numeric.shuffle_xor (deprecated) -> fx.gpu.shuffle_xor.
- Target features: ir.Attribute.parse -> the llvm.passthrough value attribute.
- Union waves per EU: the waves_per_eu compile hint, not a raw kernel attribute.
- exp2: fx.math.exp2 with afn, which lowers to the same llvm.amdgcn.exp2.
- The preparation's CTA sync: fx.gpu.barrier (release/acquire rather than seq_cst
  workgroup fences, with the same ISA).
- The union pipeline's partial lgkmcnt waits are derived: LdsQueue counts each LDS
  read and ds_bpermute it issues, checks every wait, and can record the waits
  (record_lds_waits) for tests that compare them with the ISA.

The final ISA of every kernel is unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…MI308X warning

Sparse prefill picks union tiles or the direct kernel per call: union tiles are
planned by the call's row count, BQ21/BQ42 tiles serve only a single full H6/H3
prefill, and K3 routes tiles with a cost model over the whole batch. Union and
direct round differently, so the low bits of a request's rows can change with
the other requests in the call (ROCm#6576 review). Keep that as the default, now
documented, and add a keyword-only batch_invariant=False after kv_slots. With
True, every choice that changes the bits is made per request:

- only requests of at least the union planning rows get union tiles; the rows
  of the others get direct-only tiles after them, which K2 does not compact and
  K3 publishes as direct;
- each request picks its own union tile rows (BQ21/BQ42 for its own full
  prefill); K2, K3 and the union kernel compile for the widest, which gives a
  narrower tile the same bits;
- K3 does not route tiles across the batch (the plan's existing routing switch).

A request then gives the bits it gives alone. The modes keep separate workspaces.
In both, a layout whose union scratch would exceed 4 GiB still runs direct only.
No kernel code changes: the default mode builds the same plans and launches the
same kernels with the same arguments as before.

Also reword the warning for gfx942 parts without 80 CUs: the tuning targets
MI308X only, the 80-CU settings are reused, results are unaffected and
performance may not be optimal.

Validated on MI308X (gfx942, 80 CUs), FlyDSL 0.5.0:
- default mode: 154 outputs of the op_test layouts and 37 multi-request batches
  equal byte for byte to the previous code; final ISA of all six kernel modules
  unchanged; device time within +-1% (8 interleaved runs per arm).
- batch_invariant=True: every request alone equals its rows of the batch bit for
  bit in 37 batches (union with packed and raw direct, mixed BQ21/16 and
  BQ42/32, kv_slots, HK2, group 16, T 256/1024, graph replay); batches match
  the FP32 reference.
- batch_invariant=True costs: single full prefills -4.7% to +0.3%; batches with
  a long full prefill up to -19% (per-request BQ21/BQ42); H12 1024-row chunks
  over long contexts +16% to +34% (no global routing); batches of short requests
  below the union planning rows +4% to +303% (they run direct).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…6 and batch invariance

Add six cases to op_tests/test_flydsl_qsa_sparse_prefill.py for the ROCm#6576
follow-ups. Each runs once, at its own group (CASE_GROUPS), whatever -g
selects:
- Group 16, the most Q heads per KV head the interface accepts: a union
  request batched with raw-direct rows (H16), and packed direct through a
  slot table over two KV heads (H32, HK2).
- Runtime softmax scale: after one eager call warms a layout, graphs captured
  at scales no other case uses must replay correctly after the indices, and
  through a slot table the slots, are rewritten in place. An eager call at
  another new scale must add no FlyDSL specialization, and scales that round
  to zero are rejected.
- Batch invariance with batch_invariant=True: every request alone must give
  the bits it gives in one call with the batch's other requests (union
  planning rows, union tile rows, the BQ21 tiles of a single full prefill).
  batch_invariance_h12 also runs its batch in the default mode: the output
  must be within the reference tolerance, and the call must not run on the
  batch_invariant=True workspace (the workspace cache key holds the mode).
latest_workspace() reads the prefill's most recently used workspace; routes()
uses it to assert the paths each new case took, and they stay the only
helpers reading prefill internals. make_indices accepts one selection mode
per request.

The Triton QSA code the test copies from SGLang now carries SGLang's
Apache-2.0 notice. The copy matches lines 9-309 of sparse_attn.py at SGLang
52fecfdf0908, not lines 9-267 at f2b73d4f as the old comment said; that file
has no license header, so the notice is the one SGLang's LICENSE gives.

On MI308X (gfx942) all 105 rows pass. Before this series the scale was a
compile-time constant in the workspace key, so capturing at a new scale raised
"Warm attention on this stream and layout before graph capture", and every
decision was taken per batch: a 300-row request batched with a union request
(394939 of 921600 elements) or a full H6 prefill batched with an extend (16338
of 6291456 elements) differed from the same request run alone. With the mode
dropped from the workspace cache key at run time, batch_invariance_h12 fails
("both modes ran on one workspace").

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…vel code

Add op_tests/test_flydsl_qsa_sparse_prefill_guards.py: compile-only, untimed
checks that fail when FlyDSL, MLIR or LLVM silently stop doing what the
sparse prefill kernels' remaining low-level code (sparse_prefill_primitives.py)
relies on. Such failures are otherwise silent: LLVM only prints an asm
constraint it cannot meet and the kernel computes garbage, and a dropped
attribute, a split load or a merged LDS read only costs speed or leaves a
counted wait wrong.

One process compiles prefill_12000, kv_slots_ragged_4 and
kv_slots_pool_4g_extend_64_ctx_70065 (group 12) with FlyDSL's IR dumps and
reads the union, raw direct (also its WIDE variant) and packed direct kernels:

- compile output: no "error:" line, finite outputs;
- resources: union without scratch or VGPR spill, SGPR spills at most the
  baseline, two waves per SIMD; the direct kernels without scratch or spill;
- "-packed-fp32-ops" and union's waves_per_eu 2 in LLVM IR, no v_pk_*_f32;
- the baseline number of empty inline asm blocks and asm constraints;
- union's partial lgkmcnt waits as LdsQueue recorded them while tracing equal
  those an ISA scan finds, with no SMEM load in flight;
- the direct loop's hand-counted vmcnt(8) and vmcnt(16): the loads issued on
  every path into each wait;
- every exp2 lowered to llvm.amdgcn.exp2.f32.

Without compiling a kernel it checks LdsQueue's assertions, prim.pin rejecting
one-word chunks, and that the kernel modules use FlyDSL's stable API only
(docs/api_stability.md), allowing the module functions fx.gpu.shuffle_xor and
fx.math.exp2.

On MI308X (gfx942) all guards pass in about 50 s. Patching the kernels' code
at run time makes them fail as intended: dropping union's softmax-scale pin
(SGPR spills 0 -> 12, one asm block fewer), dropping the target feature,
one-word pins (LLVM: "could not allocate output register for constraint 'v'"),
an uncounted ds_bpermute, exp2 without afn (__ocml_exp2_f32), a stricter
vmcnt(16), and edited copies of union.py with flydsl._mlir, bare rocdl,
unexported or experimental ROCDL paths, deprecated paths and methods, inline
asm or llvm.passthrough.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant