Skip to content

[Triton/Gluon] [FlyDSL] [Perf] argmax: take the split count as an argument, not a build parameter - #5703

Merged
valarLip merged 2 commits into
mainfrom
flydsl-argmax-runtime-splits
Sep 20, 2026
Merged

valarLip merged 2 commits into
mainfrom
flydsl-argmax-runtime-splits

Conversation

@valarLip

Copy link
Copy Markdown
Collaborator

What

build_topk_per_row_argmax_module is keyed on splits, and splits follows the
row count, so a build per split is a build per batch size. Over a 129280-wide
vocabulary that is 31 distinct builds at bf16 and 63 at fp32, with buckets one
row wide near the bottom — rows 35, 36, 37 and 40 each get their own.

This makes the split count a runtime argument and keys the build on a bool
(single: is the row held by one workgroup?), so two builds per dtype serve
every shape
.

While the sweep was cheap, it also showed the split rule picking 2 or 4 for rows
too short for any split to pay, so this adds _SPLITTABLE_ROW_COLUMNS.

Why it matters

On a serving trace (8.3 s window, one rank):

before after
distinct argmax kernels 28 2
device idle before a first launch 195.6 ms 0
device idle before any later launch 0 0

The 195.6 ms is 79% of all device idle in that region, in stalls of 16–27 ms
each, against a largest non-argmax gap of 0.97 ms. Each stall is one first launch
of a new splits; all 114 later launches have a gap of exactly 0. Inside a stall
the only traced host call is hipModuleLoadData.

How splits stops being compile-time

Only one use ever needed it: the fold's range_constexpr(ceil(splits / block_threads)). A runtime range(tid, splits, block_threads) covers it and
drops the masking the constexpr form needed. Note this must be a runtime
loop — over vocabulary widths the trip count is always 1, but the rule reaches
1024 splits on 4M-wide rows, where it is 4.

grid=(splits, rows, 1) already carried a runtime rows beside the constant
splits, so the launch side needed nothing new.

The rule gate

At 8192 columns a row is four block-iterations at bf16 and eight at fp32; a
split trades two of them for a second kernel launch. The existing
_MIN_SLICE_VECTORS caps how far a row may be split rather than deciding
whether to, which is how 2 and 4 got through.

The threshold is in columns, not vectors: at these widths the two dtypes
agree cell for cell while their vector counts differ by two. (It is deliberately
not the same quantity as _LONG_ROW_VECTORS despite the equal number.)

Measurements

flydsl 0.3.4.1, gfx950, 166 cells — rows 1..16384 × widths 2048..1M, both
dtypes — every split count timed twice in opposite orders:

split rule before after
worst cell vs per-cell oracle 2.362x 1.582x
cells over 1.10x 61/166 33/166
mean 1.1573 1.0565
total time given up 196.0 µs 117.1 µs

Threshold swept at 8192 / 16384 / 32768 columns; 16384 wins on all four figures,
under flydsl 0.3.4.1 and again under 0.3.2.

The runtime split count itself costs nothing measurable. 18 shapes, arms
alternated A/B/A/B: geometric mean 1.0019, worst cell 1.017, and every cell's
difference is inside that cell's own round-to-round spread, which runs to 4.4%.

Scope of behaviour change: the rule value differs only for widths
8185..16384 at bf16 and 4093..16384 at fp32. Every other shape selects
exactly what it selected before.

Correctness

  • Bit-identical to torch.argmax across the sweep's row counts at both dtypes,
    ragged row_lens included.
  • op_tests/test_topk_select.py passes: table clean, test_topk_argmax_half
    over 48 cells, and the tie / NaN invariants at the widths this touches.
  • The parity check is armed: reverting the fold's loop bound by one fails it on
    35 of 170 rows.

Notes

  • Honest limit on the ratios: the cells with the largest ratios are the ones
    where the whole kernel is 4–9 µs of launch overhead, so the absolute stakes in
    the rule are small — the largest absolute loss anywhere in the domain is
    15.6 µs on a 440 µs kernel. The per-shape compiles are the part worth having.
  • The remaining tail (1.42–1.58x) is low-row cells where the slice floor blocks a
    large split. Mechanism understood, absolute loss 3–4 µs, and those cells are
    launch-bound in a microbenchmark — left alone rather than refit four coupled
    constants against that.
  • The sweep and scoring scripts live outside this repo; happy to add them under
    op_tests/ if that is wanted, since the in-tree comments already reference an
    argmax_rule_fit.py that is not present.

…ild parameter

`build_topk_per_row_argmax_module` was keyed on `splits`, and `splits` follows
the ROW COUNT -- so a build per split is a build per batch size. Over a
129280-wide vocabulary that is 31 distinct builds at bf16 and 63 at fp32, and
the buckets are one row wide near the bottom (rows 35, 36, 37 and 40 each get
their own). A server whose batch drains through them compiles a fresh kernel
for almost every step.

Measured on a serving trace: 28 distinct argmax kernels in one 8.3 s window,
and 195.6 ms of device idle sitting in front of them -- 79% of all idle in that
region, in stalls of 16-27 ms each, against a largest non-argmax gap of 0.97 ms.
Every stall is a first launch of a new `splits`; all 114 later launches have a
gap of exactly 0.

Only one use of `splits` was ever compile-time -- the fold's
`range_constexpr(ceil(splits / block_threads))` -- and a runtime `range` covers
it while dropping the masking the constexpr form needed. `grid=(splits, rows, 1)`
already carried a runtime `rows` beside it. What remains build-time is whether
the row is held by ONE workgroup, where the slice kernel writes the answer and
no fold runs: that is a bool, so two builds per dtype serve every shape.

The same sweep, now free of per-split compiles, showed the split rule itself
picking 2 or 4 for rows short enough that no split can pay: at 8192 columns a
row is four block-iterations at bf16, and a split trades two of them for a
second kernel launch. `_SPLITTABLE_ROW_COLUMNS` gates that. It is in columns,
not vectors, because there the two dtypes agree cell for cell.

Measured, flydsl 0.3.4.1, 166 cells (rows 1..16384 x widths 2048..1M, both
dtypes), every split count timed twice in opposite orders:

  kernel names in a serving trace   28 -> 2
  idle before a first launch        195.6 ms -> 0
  split rule, worst cell            2.362x -> 1.582x
  split rule, cells over 1.10x      61/166 -> 33/166
  split rule, mean                  1.1573 -> 1.0565

The runtime split count costs nothing measurable: over 18 shapes read twice in
alternating arms, geometric mean 1.0019, worst cell 1.017, and every cell's
difference is inside that cell's own round-to-round spread, which runs to 4.4%.
The gate threshold was swept at 8192, 16384 and 32768 columns; 16384 wins on
worst cell, cells over 1.10x, mean and total time, under flydsl 0.3.4.1 and
again under 0.3.2.

The rule value changes only for widths 8185..16384 at bf16 and 4093..16384 at
fp32. Every other shape selects exactly what it selected before.

Correctness: bit-identical to `torch.argmax` over the sweep's row counts at
both dtypes, ragged `row_lens` included, and `op_tests/test_topk_select.py`
passes -- table clean, `test_topk_argmax_half` over 48 cells, and the tie and
NaN invariants at the widths this touches. Reverting the fold's loop bound by
one makes the parity check fail on 35 of 170 rows, so it is armed.
@valarLip
valarLip requested a review from a team September 20, 2026 09:20
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
multigpu Aiter multi-GPU tests on the 8-GPU runner
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 5703 --add-label <label>

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to opt this PR out.

…uild parameter

The same axis as the commit before it, one file over. `NUM_PRGMS` is declared
`tl.constexpr` in `_fused_rms_mxfp8_kernel` and in
`_fused_dual_rmsnorm_mxfp8_quant_kernel`, and both wrappers set it to `M` --
the token count. So a build per program count is a build per prefill chunk
length and per batch size: a consumer that reached these kernels for a few
hours left 732 variants of the dual one in its Triton cache.

They are not free at serving time. In a profiled serving window the device
sat idle 350-614 ms in front of `_fused_dual_rmsnorm_mxfp8_quant_kernel`, in
stalls of 113-133 ms, each holding nothing but `hipModuleLoadData`. With the
count runtime, eight token counts the box had never seen compile one kernel
between them, where the constexpr form compiled eight.

`NUM_PRGMS` is only the row loop's stride, which a runtime value carries as
well -- and with `grid = (M,)` and `NUM_PRGMS = M`, as both wrappers have it,
that loop runs exactly once either way.

Measured with both spellings in ONE process, alternating over the same
tensors: across processes a JIT'd kernel drifts by 4-8%, which is more than
the effect. Output is bit-identical; time is +1.0 / -0.5 / +1.3 / +1.3% at
64 / 384 / 2048 / 16384 rows on gfx950, KQ=1280 KK=512.

The persistent form `normalization/rmsnorm.py` uses -- `grid =
min(M, num_sms * 4)`, the count constexpr again but bounded -- was tried and
is not here: it costs 11% at 16384 rows and ties below that, so the grid
stays `M`.

`op_tests/triton_tests/quant/test_quant_mxfp8.py` passes, 85 cases.
@github-actions github-actions Bot changed the title [FlyDSL] [Perf] argmax: take the split count as an argument, not a build parameter [Triton/Gluon] [FlyDSL] [Perf] argmax: take the split count as an argument, not a build parameter Sep 20, 2026
@valarLip
valarLip merged commit 6a9a005 into main Sep 20, 2026
37 checks passed
@valarLip
valarLip deleted the flydsl-argmax-runtime-splits branch September 20, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant