Repository navigation
[FlyDSL] topk_select: half formats for k=1, routing re-fit, and a barrier the streaming selector was missing - #5526
Merged
Conversation
The per-row argmax read fp32 only, so a caller holding bf16 logits -- which is every LLM sampler and every draft head -- had to materialise an fp32 copy of the row first. That cast is its own pass over the row and it dominates: at 1024 rows of 129280, cast+select measured 252.8us against 96.5 for the selection alone. One ordering rule still serves all three dtypes. fp32 covers the range and the precision of both half formats, subnormals included, so widening either one is exact -- injective and order-preserving -- and the existing `ord_signed_f32` lands on the same answer after the widen. Ties to the smallest column and NaN over +inf therefore follow by construction rather than by a second rule that would have to be kept in step with the first. Elements are widened as they are read, so the row stays half the bytes. Measured against `torch.argmax` at the tensor's own dtype, exact on every cell (rtol=0 atol=0), 18 cells of rows 1..1024 by widths 2048..262144: 1.23x..4.59x for both bf16 and fp16, and never below 1. Also fixes a latent fault this change would have made live: the module builder takes `vec`, but the kernel body read the module-level `_VEC` for its layout and its column arithmetic. The two agreed only because nothing ever passed a different `vec`; a half format wants eight elements per 16-byte load and would have silently mispaired every column with its score.
…cost Four changes to the entry, each measured on a 565-cell sweep of rows 1..16384 by widths 2048..1M -- including the non-power-of-two widths a sparse indexer produces, `max_seq_len // compress_ratio`, which fell between the samples the previous rules were fitted on. 1. `stable` follows the caller. `_dispatch` hardwired it on for the decode backend, so every call bought a promise -- smallest-index ties and an ascending emit -- that only `tie='low'` asks for, at 0-20%. Worse, the routing had been fitted against a decode that was always paying it, which is part of why the old rules sent these shapes elsewhere. `deterministic` needs it too, and that was measured rather than assumed: without `stable` this backend selects a DIFFERENT set of tied columns from one call to the next. `test_invariants` catches it at 16 rows of 262144, k=2048. A radix select being a pure function of the row is the obvious guess and it is wrong. 2. Routing re-fitted. decode is the fastest backend on 257 of 565 cells and the old rules reached it on 65, because its gate read `rows <= 64 AND (width >= 1048576 OR (width >= 8192 AND k >= 4096))` against real shapes of width <= 262144 and k <= 2048. It is now a row bound and a width floor with nothing conjoined after them. Against the per-cell oracle: mean 1.295x -> 1.046x, p90 1.795x -> 1.195x, worst 5.916x -> 2.183x, and more than 1.2x off on 55 cells against 205. The `plain` few-rows band is dropped rather than retuned: with decode's gate widened it is decode's on every cell of it, and every value of the old bound scored identically, which is what a dead rule looks like. 3. The streaming selector's block width follows the row count and k. Fitted over 195 cells of both widths: `k >= 4096 or rows <= 256` costs a mean 1.001x against picking per cell, where holding 512 everywhere costs 1.135x and a worst 2.03x. A width term scores identically and so is not in the rule -- the apparent width effect was rows in disguise. 4. `return_value` / `sorted` gather in one launch. The five ops it replaces -- `long()`, `clamp_min_`, `gather`, `idx < 0`, `masked_fill_` -- cost 17.1us over 640 elements and 38.1us over 524288, so what they cost is being five rather than what they touch. One kernel over the same data: 21.0us -> 3.7us at 64x129280 k=10, 22.4 -> 3.8 at 1024x32768 k=16. Documentation caught up with DeepSelect v1.0.0 / main @ 0f03b68 while checking the compatibility claim: the first fourteen parameters do match by name, order and default, and all four documented divergences are still accurate. The dtype row is the one that is not a pure narrowing -- fp16 is not a DeepSelect dtype at all -- and that is now said.
…e what that uncovered The streaming selector answered a few rows in ten thousand wrongly -- 8 to 22 in 338484, only above a row count, and hidden by any added instruction. It was a missing barrier, and fixing it made two tuning tables stale. 1. The barrier. `absorb` publishes its arrivals with a barrier at its end, and the loop then reads that count and decides whether to re-select. Nothing separated the READ from the next group's writes: a wave that cleared the read ran into the next `absorb` and added to a counter another wave had not looked at yet. The count gates `compact`, which is barriers, so one wave over the trigger and another under it left the workgroup executing different numbers of them -- after which `pick_bucket` selects no bucket, the cut keeps a stale value, and the placement pass writes nothing. Witnessed rather than inferred, by having each wave accumulate the counts it read: 161 rows in 338484 whose waves disagreed, holding all 9 that answered wrongly. Both go to zero with the barrier. It costs 0.994x..1.010x over six shapes against a stable `amax`, including its own worst case -- descending rows, where the trigger never fires and it is the only barrier the group pays. The account in `radix_cut` blaming register pressure is marked unproven. The failing build spills nothing at all -- 0 VGPR, 0 SGPR, 0 scratch, 0 AGPR -- and is LDS-bound, and every instruction added to test that hypothesis perturbs the schedule enough to hide a race. 2. `topk_per_row_radix_stream_lds_plan`. `_resolve_lds` takes the deepest prefetch the budget allows, which knows k and not the ROW COUNT -- and the row count is what decides the depth while the kernel is mostly streaming. Fitted over 100 cells by 24 knob pairs and scoped to where it pays: an A/B of 1.118x with one cell below 0.95x, where firing everywhere is 1.126x with eleven. A preference, not a decision; `_resolve_lds` declines a pair that does not fit. 3. The LDS budget is asked of the device. The literals were sized for gfx950's 160 KiB CU while `topk_select` advertises gfx942 at 64 and routes without an arch term, so above k=1024 the build claimed more LDS than the card has -- loudly, but after `serves` had told the router the geometry was fine. On gfx950 the derivation reproduces 76 KiB and 148 KiB exactly. `test_lds_sizing` covers it, and is arithmetic so it can check a card this box does not have. 4. The `partial` split was unreachable, and is now tuned where it runs. Of 565 routing cells 36 would have split and the router picked `stream` on none of them -- its gate sat entirely inside decode's -- which is how its block target came to contradict itself, stopping G at 2 for 128 rows where 4 is 1.2x faster. Re-swept over the 108 cells the router does send there: twelve cells change, 1.019x to 1.314x, and none regress. 5. Routing re-fitted, twice, because 2 and 4 each changed what `_dispatch` hands a backend. decode's row bound gained a k term; it takes back narrow rows at k=4096; it now needs the work to be worth its launch, its time being flat at 30..36us across widths 32768..65536 and 1..64 rows where `stream` runs 18.6..51.4us; and `plain` gained a k floor, never being fastest below k=1024 -- it wins 29 cells at 1024 and 35 at 2048 and none under. Against the per-cell oracle: mean 1.050x -> 1.014x, p90 1.204x -> 1.020x, worst 2.080x -> 1.564x, more than 1.2x off on 12 cells against 58. Every step is scored as an A/B against the rule it replaces as well as against the per-cell oracle. The two disagreed five times, and each time the oracle preferred something that made a shape markedly slower -- 0.578x, 0.632x, 0.699x, 0.704x, 0.886x. Those neighbours are recorded beside the constants they would have moved.
Contributor
🏷️ CI GuideRuns automatically on every PR:
Extended tests (opt-in via labels):
PR title tags & labels: |
3 of 4 tasks
`_LDS_LIMIT = 160 * 1024` under a comment reading "gfx9 gives a workgroup 160
KiB of LDS". gfx9 includes gfx942, which gives 64 KiB, so `serves` promised the
router a selector the card could not load:
error: local memory (69696) exceeds limit (65536) in function
'topk_per_row_small_k_k16_n32768_wave64_blk1024_tlow0_pins0_pf4'
followed by several hundred `hipErrorIllegalState` -- the failure mode that
function's own comment describes, since an over-budget build takes the HIP
context with it rather than raising. aiter already ships the table
(`get_lds_capacity_bytes`) and two kernels already ask it; this one wrote the
number down instead.
Eight (k, width) geometries small_k serves on gfx950 must decline on gfx942 --
k=16 at 32768, k=4 at 131072, k=32 at 16384, k=64 at 8192, and each one width
up. The router has a backend for all of them.
Swept rather than fixed in place, and the sweep found a second site: the
streaming selector took the same table but through `get_lds_capacity_bytes()`,
whose default arch is `get_gfx()` -- which honours `GPU_ARCHS` while FlyDSL
compiles for the live device either way. Measured:
GPU_ARCHS=gfx942 flydsl_target=gfx950 get_gfx()=gfx942 cap=65536
so the budget sizes for one card and the compiler for another. Both selectors
now ask `get_gfx_runtime()`. The wrapper's predicate reads the same function as
the builder, which is what keeps the two from promising different things.
Verified on gfx950 that nothing changed -- the cap is still 163840 and all eight
geometries still serve. The gfx942 half was checked by forcing the cap to 64 KiB:
the predicate declines exactly those eight, the router sends them to stream or
decode, and all ten (shape, row count) cases come back exact. That covers the
arch-independent half; gfx942 codegen is for CI, which is where this was found.
gyohuangxin
approved these changes
Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Three commits against
aiter.topk_selectand the selectors behind it: a half-format path for k=1, a routing re-fit, and a silent-wrong-answer fix in the streaming selector that made the routing stale twice over.1.
feat(topk): let the k=1 reduction take bf16 and fp16topk_selectrefused anything but fp32. The k=1 reduction folds int32 ordering keys, so it can widen each element as it reads it; the selectors compare and re-read the scores themselves, so for them a half format would be a second set of kernels rather than a load. Only the reduction gained the formats, and the entry now says so instead of rejecting the call.2.
perf(topk): route topk_select on what the backends actually coststablefollows the caller rather than being hardwired on fordecode(0-20%, and onlytie='low'asks for it), and the routing is re-fitted over a 565-cell sweep that includes the non-power-of-two widths a sparse indexer produces --max_seq_len // compress_ratio, which fell between the samples the old rules were fitted on.3.
fix(topk): barrier the streaming loop's back edge, and retune what that uncoveredThe streaming selector answered 8 to 22 rows in 338484 wrongly, only above a row count, and any added instruction hid it.
absorbpublishes its arrivals with a barrier at its end, and the loop then reads that count to decide whether to re-select. Nothing separated the READ from the next group's writes: a wave that cleared the read ran into the nextabsorband added to a counter another wave had not looked at yet. That count gatescompact, which is barriers, so one wave over the trigger and another under it left the workgroup executing different numbers of them.Witnessed rather than inferred: each wave accumulating the counts it read gives 161 rows in 338484 whose waves disagreed, holding all 9 that answered wrongly. Both go to zero with the barrier, which costs 0.994x..1.010x over six shapes against a stable
amaxcontrol.Fixing it made two tables stale, so both were re-swept:
topk_per_row_radix_stream_lds_plansupplies the row-count term_resolve_ldslacks. Scoped to where it pays: 1.118x as an A/B with one cell of 100 below 0.95x, against 1.126x with eleven if it fires everywhere. A preference --_resolve_ldsdeclines a pair that does not fit.partialsplit was unreachable -- of 565 cells 36 would have split and the router pickedstreamon none -- and its constants were therefore fitted on cells that never ran. Re-swept over the 108 cells the router does send there._dispatchhands a backend.Against the per-cell oracle over the 565-cell sweep: mean 1.050x -> 1.014x, p90 1.204x -> 1.020x, worst 2.080x -> 1.564x, more than 1.2x off on 12 cells against 58.
Method note
Every step is scored as an A/B against the rule it replaces as well as against the per-cell oracle. The two disagreed five times, and each time the oracle preferred something that made a shape markedly slower -- 0.578x, 0.632x, 0.699x, 0.704x, 0.886x. Those rejected neighbours are recorded beside the constants they would have moved, so the next person does not re-propose them.
Test plan
topk_stream_soak.py, the streaming selector's own gate: clean over 338484 rows at k=16..2048, on both block widths. Dirty before the barrier on every run.op_tests/test_topk_select.py-- invariants, half-format invariants, and the newtest_lds_sizing, which is arithmetic so it can check a card this box does not have. Armed: restoring the literals makes it report 31 failures.topk_selectagainsttorch.topkon 18 shapes spanning all five backends, including three that take the split.ruffandblackclean on all changed files.