Repository navigation
[Triton/Gluon] [FlyDSL] [Perf] argmax: take the split count as an argument, not a build parameter - #5703
Merged
Conversation
…ild parameter `build_topk_per_row_argmax_module` was keyed on `splits`, and `splits` follows the ROW COUNT -- so a build per split is a build per batch size. Over a 129280-wide vocabulary that is 31 distinct builds at bf16 and 63 at fp32, and the buckets are one row wide near the bottom (rows 35, 36, 37 and 40 each get their own). A server whose batch drains through them compiles a fresh kernel for almost every step. Measured on a serving trace: 28 distinct argmax kernels in one 8.3 s window, and 195.6 ms of device idle sitting in front of them -- 79% of all idle in that region, in stalls of 16-27 ms each, against a largest non-argmax gap of 0.97 ms. Every stall is a first launch of a new `splits`; all 114 later launches have a gap of exactly 0. Only one use of `splits` was ever compile-time -- the fold's `range_constexpr(ceil(splits / block_threads))` -- and a runtime `range` covers it while dropping the masking the constexpr form needed. `grid=(splits, rows, 1)` already carried a runtime `rows` beside it. What remains build-time is whether the row is held by ONE workgroup, where the slice kernel writes the answer and no fold runs: that is a bool, so two builds per dtype serve every shape. The same sweep, now free of per-split compiles, showed the split rule itself picking 2 or 4 for rows short enough that no split can pay: at 8192 columns a row is four block-iterations at bf16, and a split trades two of them for a second kernel launch. `_SPLITTABLE_ROW_COLUMNS` gates that. It is in columns, not vectors, because there the two dtypes agree cell for cell. Measured, flydsl 0.3.4.1, 166 cells (rows 1..16384 x widths 2048..1M, both dtypes), every split count timed twice in opposite orders: kernel names in a serving trace 28 -> 2 idle before a first launch 195.6 ms -> 0 split rule, worst cell 2.362x -> 1.582x split rule, cells over 1.10x 61/166 -> 33/166 split rule, mean 1.1573 -> 1.0565 The runtime split count costs nothing measurable: over 18 shapes read twice in alternating arms, geometric mean 1.0019, worst cell 1.017, and every cell's difference is inside that cell's own round-to-round spread, which runs to 4.4%. The gate threshold was swept at 8192, 16384 and 32768 columns; 16384 wins on worst cell, cells over 1.10x, mean and total time, under flydsl 0.3.4.1 and again under 0.3.2. The rule value changes only for widths 8185..16384 at bf16 and 4093..16384 at fp32. Every other shape selects exactly what it selected before. Correctness: bit-identical to `torch.argmax` over the sweep's row counts at both dtypes, ragged `row_lens` included, and `op_tests/test_topk_select.py` passes -- table clean, `test_topk_argmax_half` over 48 cells, and the tie and NaN invariants at the widths this touches. Reverting the fold's loop bound by one makes the parity check fail on 35 of 170 rows, so it is armed.
Contributor
🏷️ CI GuideRuns automatically on every PR:
Extended tests (opt-in via labels):
PR title tags & labels: |
…uild parameter The same axis as the commit before it, one file over. `NUM_PRGMS` is declared `tl.constexpr` in `_fused_rms_mxfp8_kernel` and in `_fused_dual_rmsnorm_mxfp8_quant_kernel`, and both wrappers set it to `M` -- the token count. So a build per program count is a build per prefill chunk length and per batch size: a consumer that reached these kernels for a few hours left 732 variants of the dual one in its Triton cache. They are not free at serving time. In a profiled serving window the device sat idle 350-614 ms in front of `_fused_dual_rmsnorm_mxfp8_quant_kernel`, in stalls of 113-133 ms, each holding nothing but `hipModuleLoadData`. With the count runtime, eight token counts the box had never seen compile one kernel between them, where the constexpr form compiled eight. `NUM_PRGMS` is only the row loop's stride, which a runtime value carries as well -- and with `grid = (M,)` and `NUM_PRGMS = M`, as both wrappers have it, that loop runs exactly once either way. Measured with both spellings in ONE process, alternating over the same tensors: across processes a JIT'd kernel drifts by 4-8%, which is more than the effect. Output is bit-identical; time is +1.0 / -0.5 / +1.3 / +1.3% at 64 / 384 / 2048 / 16384 rows on gfx950, KQ=1280 KK=512. The persistent form `normalization/rmsnorm.py` uses -- `grid = min(M, num_sms * 4)`, the count constexpr again but bounded -- was tried and is not here: it costs 11% at 16384 rows and ties below that, so the grid stays `M`. `op_tests/triton_tests/quant/test_quant_mxfp8.py` passes, 85 cases.
5 of 6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
build_topk_per_row_argmax_moduleis keyed onsplits, andsplitsfollows therow count, so a build per split is a build per batch size. Over a 129280-wide
vocabulary that is 31 distinct builds at bf16 and 63 at fp32, with buckets one
row wide near the bottom — rows 35, 36, 37 and 40 each get their own.
This makes the split count a runtime argument and keys the build on a bool
(
single: is the row held by one workgroup?), so two builds per dtype serveevery shape.
While the sweep was cheap, it also showed the split rule picking 2 or 4 for rows
too short for any split to pay, so this adds
_SPLITTABLE_ROW_COLUMNS.Why it matters
On a serving trace (8.3 s window, one rank):
The 195.6 ms is 79% of all device idle in that region, in stalls of 16–27 ms
each, against a largest non-argmax gap of 0.97 ms. Each stall is one first launch
of a new
splits; all 114 later launches have a gap of exactly 0. Inside a stallthe only traced host call is
hipModuleLoadData.How
splitsstops being compile-timeOnly one use ever needed it: the fold's
range_constexpr(ceil(splits / block_threads)). A runtimerange(tid, splits, block_threads)covers it anddrops the masking the constexpr form needed. Note this must be a runtime
loop — over vocabulary widths the trip count is always 1, but the rule reaches
1024 splits on 4M-wide rows, where it is 4.
grid=(splits, rows, 1)already carried a runtimerowsbeside the constantsplits, so the launch side needed nothing new.The rule gate
At 8192 columns a row is four block-iterations at bf16 and eight at fp32; a
split trades two of them for a second kernel launch. The existing
_MIN_SLICE_VECTORScaps how far a row may be split rather than decidingwhether to, which is how 2 and 4 got through.
The threshold is in columns, not vectors: at these widths the two dtypes
agree cell for cell while their vector counts differ by two. (It is deliberately
not the same quantity as
_LONG_ROW_VECTORSdespite the equal number.)Measurements
flydsl 0.3.4.1, gfx950, 166 cells — rows 1..16384 × widths 2048..1M, both
dtypes — every split count timed twice in opposite orders:
Threshold swept at 8192 / 16384 / 32768 columns; 16384 wins on all four figures,
under flydsl 0.3.4.1 and again under 0.3.2.
The runtime split count itself costs nothing measurable. 18 shapes, arms
alternated A/B/A/B: geometric mean 1.0019, worst cell 1.017, and every cell's
difference is inside that cell's own round-to-round spread, which runs to 4.4%.
Scope of behaviour change: the rule value differs only for widths
8185..16384 at bf16 and 4093..16384 at fp32. Every other shape selects
exactly what it selected before.
Correctness
torch.argmaxacross the sweep's row counts at both dtypes,ragged
row_lensincluded.op_tests/test_topk_select.pypasses: table clean,test_topk_argmax_halfover 48 cells, and the tie / NaN invariants at the widths this touches.
35 of 170 rows.
Notes
where the whole kernel is 4–9 µs of launch overhead, so the absolute stakes in
the rule are small — the largest absolute loss anywhere in the domain is
15.6 µs on a 440 µs kernel. The per-shape compiles are the part worth having.
large split. Mechanism understood, absolute loss 3–4 µs, and those cells are
launch-bound in a microbenchmark — left alone rather than refit four coupled
constants against that.
op_tests/if that is wanted, since the in-tree comments already reference anargmax_rule_fit.pythat is not present.