Skip to content

[FlyDSL] topk_select: half formats for k=1, routing re-fit, and a barrier the streaming selector was missing - #5526

Merged
gyohuangxin merged 4 commits into
mainfrom
perf/topk-select-half-formats-and-routing
Sep 15, 2026
Merged

gyohuangxin merged 4 commits into
mainfrom
perf/topk-select-half-formats-and-routing

Conversation

@valarLip

Copy link
Copy Markdown
Collaborator

What this does

Three commits against aiter.topk_select and the selectors behind it: a half-format path for k=1, a routing re-fit, and a silent-wrong-answer fix in the streaming selector that made the routing stale twice over.

1. feat(topk): let the k=1 reduction take bf16 and fp16

topk_select refused anything but fp32. The k=1 reduction folds int32 ordering keys, so it can widen each element as it reads it; the selectors compare and re-read the scores themselves, so for them a half format would be a second set of kernels rather than a load. Only the reduction gained the formats, and the entry now says so instead of rejecting the call.

2. perf(topk): route topk_select on what the backends actually cost

stable follows the caller rather than being hardwired on for decode (0-20%, and only tie='low' asks for it), and the routing is re-fitted over a 565-cell sweep that includes the non-power-of-two widths a sparse indexer produces -- max_seq_len // compress_ratio, which fell between the samples the old rules were fitted on.

3. fix(topk): barrier the streaming loop's back edge, and retune what that uncovered

The streaming selector answered 8 to 22 rows in 338484 wrongly, only above a row count, and any added instruction hid it.

absorb publishes its arrivals with a barrier at its end, and the loop then reads that count to decide whether to re-select. Nothing separated the READ from the next group's writes: a wave that cleared the read ran into the next absorb and added to a counter another wave had not looked at yet. That count gates compact, which is barriers, so one wave over the trigger and another under it left the workgroup executing different numbers of them.

Witnessed rather than inferred: each wave accumulating the counts it read gives 161 rows in 338484 whose waves disagreed, holding all 9 that answered wrongly. Both go to zero with the barrier, which costs 0.994x..1.010x over six shapes against a stable amax control.

Fixing it made two tables stale, so both were re-swept:

  • topk_per_row_radix_stream_lds_plan supplies the row-count term _resolve_lds lacks. Scoped to where it pays: 1.118x as an A/B with one cell of 100 below 0.95x, against 1.126x with eleven if it fires everywhere. A preference -- _resolve_lds declines a pair that does not fit.
  • The LDS budget comes from the device. The literals were gfx950's 160 KiB CU while this entry advertises gfx942 at 64 and routes without an arch term, so above k=1024 the build claimed more LDS than the card has. On gfx950 the derivation reproduces 76 KiB and 148 KiB exactly.
  • The partial split was unreachable -- of 565 cells 36 would have split and the router picked stream on none -- and its constants were therefore fitted on cells that never ran. Re-swept over the 108 cells the router does send there.
  • Routing re-fitted twice more, because each of those changed what _dispatch hands a backend.

Against the per-cell oracle over the 565-cell sweep: mean 1.050x -> 1.014x, p90 1.204x -> 1.020x, worst 2.080x -> 1.564x, more than 1.2x off on 12 cells against 58.

Method note

Every step is scored as an A/B against the rule it replaces as well as against the per-cell oracle. The two disagreed five times, and each time the oracle preferred something that made a shape markedly slower -- 0.578x, 0.632x, 0.699x, 0.704x, 0.886x. Those rejected neighbours are recorded beside the constants they would have moved, so the next person does not re-propose them.

Test plan

  • topk_stream_soak.py, the streaming selector's own gate: clean over 338484 rows at k=16..2048, on both block widths. Dirty before the barrier on every run.
  • The same soak on each knob pair the plan can emit, and on the two the sweep found wrong before the fix: clean.
  • op_tests/test_topk_select.py -- invariants, half-format invariants, and the new test_lds_sizing, which is arithmetic so it can check a card this box does not have. Armed: restoring the literals makes it report 31 failures.
  • topk_select against torch.topk on 18 shapes spanning all five backends, including three that take the split.
  • ruff and black clean on all changed files.
  • CI on a gfx942 runner -- the LDS derivation is covered by arithmetic here, but has not been run on that card.

The per-row argmax read fp32 only, so a caller holding bf16 logits -- which
is every LLM sampler and every draft head -- had to materialise an fp32 copy
of the row first. That cast is its own pass over the row and it dominates:
at 1024 rows of 129280, cast+select measured 252.8us against 96.5 for the
selection alone.

One ordering rule still serves all three dtypes. fp32 covers the range and
the precision of both half formats, subnormals included, so widening either
one is exact -- injective and order-preserving -- and the existing
`ord_signed_f32` lands on the same answer after the widen. Ties to the
smallest column and NaN over +inf therefore follow by construction rather
than by a second rule that would have to be kept in step with the first.
Elements are widened as they are read, so the row stays half the bytes.

Measured against `torch.argmax` at the tensor's own dtype, exact on every
cell (rtol=0 atol=0), 18 cells of rows 1..1024 by widths 2048..262144:
1.23x..4.59x for both bf16 and fp16, and never below 1.

Also fixes a latent fault this change would have made live: the module
builder takes `vec`, but the kernel body read the module-level `_VEC` for
its layout and its column arithmetic. The two agreed only because nothing
ever passed a different `vec`; a half format wants eight elements per
16-byte load and would have silently mispaired every column with its score.
…cost

Four changes to the entry, each measured on a 565-cell sweep of rows
1..16384 by widths 2048..1M -- including the non-power-of-two widths a
sparse indexer produces, `max_seq_len // compress_ratio`, which fell
between the samples the previous rules were fitted on.

1. `stable` follows the caller. `_dispatch` hardwired it on for the decode
   backend, so every call bought a promise -- smallest-index ties and an
   ascending emit -- that only `tie='low'` asks for, at 0-20%. Worse, the
   routing had been fitted against a decode that was always paying it, which
   is part of why the old rules sent these shapes elsewhere.

   `deterministic` needs it too, and that was measured rather than assumed:
   without `stable` this backend selects a DIFFERENT set of tied columns from
   one call to the next. `test_invariants` catches it at 16 rows of 262144,
   k=2048. A radix select being a pure function of the row is the obvious
   guess and it is wrong.

2. Routing re-fitted. decode is the fastest backend on 257 of 565 cells and
   the old rules reached it on 65, because its gate read `rows <= 64 AND
   (width >= 1048576 OR (width >= 8192 AND k >= 4096))` against real shapes
   of width <= 262144 and k <= 2048. It is now a row bound and a width floor
   with nothing conjoined after them. Against the per-cell oracle:
   mean 1.295x -> 1.046x, p90 1.795x -> 1.195x, worst 5.916x -> 2.183x,
   and more than 1.2x off on 55 cells against 205.

   The `plain` few-rows band is dropped rather than retuned: with decode's
   gate widened it is decode's on every cell of it, and every value of the
   old bound scored identically, which is what a dead rule looks like.

3. The streaming selector's block width follows the row count and k.
   Fitted over 195 cells of both widths: `k >= 4096 or rows <= 256` costs a
   mean 1.001x against picking per cell, where holding 512 everywhere costs
   1.135x and a worst 2.03x. A width term scores identically and so is not
   in the rule -- the apparent width effect was rows in disguise.

4. `return_value` / `sorted` gather in one launch. The five ops it replaces
   -- `long()`, `clamp_min_`, `gather`, `idx < 0`, `masked_fill_` -- cost
   17.1us over 640 elements and 38.1us over 524288, so what they cost is
   being five rather than what they touch. One kernel over the same data:
   21.0us -> 3.7us at 64x129280 k=10, 22.4 -> 3.8 at 1024x32768 k=16.

Documentation caught up with DeepSelect v1.0.0 / main @ 0f03b68 while
checking the compatibility claim: the first fourteen parameters do match by
name, order and default, and all four documented divergences are still
accurate. The dtype row is the one that is not a pure narrowing -- fp16 is
not a DeepSelect dtype at all -- and that is now said.
…e what that uncovered

The streaming selector answered a few rows in ten thousand wrongly -- 8 to 22
in 338484, only above a row count, and hidden by any added instruction. It was
a missing barrier, and fixing it made two tuning tables stale.

1. The barrier. `absorb` publishes its arrivals with a barrier at its end, and
   the loop then reads that count and decides whether to re-select. Nothing
   separated the READ from the next group's writes: a wave that cleared the
   read ran into the next `absorb` and added to a counter another wave had not
   looked at yet. The count gates `compact`, which is barriers, so one wave
   over the trigger and another under it left the workgroup executing
   different numbers of them -- after which `pick_bucket` selects no bucket,
   the cut keeps a stale value, and the placement pass writes nothing.

   Witnessed rather than inferred, by having each wave accumulate the counts
   it read: 161 rows in 338484 whose waves disagreed, holding all 9 that
   answered wrongly. Both go to zero with the barrier. It costs 0.994x..1.010x
   over six shapes against a stable `amax`, including its own worst case --
   descending rows, where the trigger never fires and it is the only barrier
   the group pays.

   The account in `radix_cut` blaming register pressure is marked unproven.
   The failing build spills nothing at all -- 0 VGPR, 0 SGPR, 0 scratch, 0
   AGPR -- and is LDS-bound, and every instruction added to test that
   hypothesis perturbs the schedule enough to hide a race.

2. `topk_per_row_radix_stream_lds_plan`. `_resolve_lds` takes the deepest
   prefetch the budget allows, which knows k and not the ROW COUNT -- and the
   row count is what decides the depth while the kernel is mostly streaming.
   Fitted over 100 cells by 24 knob pairs and scoped to where it pays: an A/B
   of 1.118x with one cell below 0.95x, where firing everywhere is 1.126x with
   eleven. A preference, not a decision; `_resolve_lds` declines a pair that
   does not fit.

3. The LDS budget is asked of the device. The literals were sized for gfx950's
   160 KiB CU while `topk_select` advertises gfx942 at 64 and routes without
   an arch term, so above k=1024 the build claimed more LDS than the card has
   -- loudly, but after `serves` had told the router the geometry was fine. On
   gfx950 the derivation reproduces 76 KiB and 148 KiB exactly. `test_lds_sizing`
   covers it, and is arithmetic so it can check a card this box does not have.

4. The `partial` split was unreachable, and is now tuned where it runs. Of 565
   routing cells 36 would have split and the router picked `stream` on none of
   them -- its gate sat entirely inside decode's -- which is how its block
   target came to contradict itself, stopping G at 2 for 128 rows where 4 is
   1.2x faster. Re-swept over the 108 cells the router does send there: twelve
   cells change, 1.019x to 1.314x, and none regress.

5. Routing re-fitted, twice, because 2 and 4 each changed what `_dispatch`
   hands a backend. decode's row bound gained a k term; it takes back narrow
   rows at k=4096; it now needs the work to be worth its launch, its time
   being flat at 30..36us across widths 32768..65536 and 1..64 rows where
   `stream` runs 18.6..51.4us; and `plain` gained a k floor, never being
   fastest below k=1024 -- it wins 29 cells at 1024 and 35 at 2048 and none
   under. Against the per-cell oracle: mean 1.050x -> 1.014x, p90 1.204x ->
   1.020x, worst 2.080x -> 1.564x, more than 1.2x off on 12 cells against 58.

Every step is scored as an A/B against the rule it replaces as well as against
the per-cell oracle. The two disagreed five times, and each time the oracle
preferred something that made a shape markedly slower -- 0.578x, 0.632x,
0.699x, 0.704x, 0.886x. Those neighbours are recorded beside the constants
they would have moved.
@valarLip
valarLip requested a review from a team September 15, 2026 08:49
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
multigpu Aiter multi-GPU tests on the 8-GPU runner
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 5526 --add-label <label>

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to opt this PR out.

`_LDS_LIMIT = 160 * 1024` under a comment reading "gfx9 gives a workgroup 160
KiB of LDS". gfx9 includes gfx942, which gives 64 KiB, so `serves` promised the
router a selector the card could not load:

    error: local memory (69696) exceeds limit (65536) in function
      'topk_per_row_small_k_k16_n32768_wave64_blk1024_tlow0_pins0_pf4'

followed by several hundred `hipErrorIllegalState` -- the failure mode that
function's own comment describes, since an over-budget build takes the HIP
context with it rather than raising. aiter already ships the table
(`get_lds_capacity_bytes`) and two kernels already ask it; this one wrote the
number down instead.

Eight (k, width) geometries small_k serves on gfx950 must decline on gfx942 --
k=16 at 32768, k=4 at 131072, k=32 at 16384, k=64 at 8192, and each one width
up. The router has a backend for all of them.

Swept rather than fixed in place, and the sweep found a second site: the
streaming selector took the same table but through `get_lds_capacity_bytes()`,
whose default arch is `get_gfx()` -- which honours `GPU_ARCHS` while FlyDSL
compiles for the live device either way. Measured:

    GPU_ARCHS=gfx942  flydsl_target=gfx950  get_gfx()=gfx942  cap=65536

so the budget sizes for one card and the compiler for another. Both selectors
now ask `get_gfx_runtime()`. The wrapper's predicate reads the same function as
the builder, which is what keeps the two from promising different things.

Verified on gfx950 that nothing changed -- the cap is still 163840 and all eight
geometries still serve. The gfx942 half was checked by forcing the cap to 64 KiB:
the predicate declines exactly those eight, the router sends them to stream or
decode, and all ten (shape, row count) cases come back exact. That covers the
arch-independent half; gfx942 codegen is for CI, which is where this was found.
@gyohuangxin
gyohuangxin merged commit 958c31e into main Sep 15, 2026
56 checks passed
@gyohuangxin
gyohuangxin deleted the perf/topk-select-half-formats-and-routing branch September 15, 2026 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants