Repository navigation
perf(sampler): take the greedy picks from aiter.topk_select - #2244
Merged
Merged
Conversation
Four call sites reduced a [rows, vocab] tensor with `torch.argmax` or `torch.topk`. `aiter.topk_select` dispatches the same reduction over whichever of its backends is fastest for the shape, and takes bf16 without a cast in front of it -- the k=1 reduction folds int32 ordering keys, so it widens each element as it reads rather than materialising a [rows, vocab] fp32 copy. Every call passes `tie="low"`, and that is the whole reason the answer is the pick `torch.argmax` made. The default promises no direction among equal scores, and neither did `torch.topk`, so a near-tie would resolve differently from one TP rank to the next -- and in the rejection sampler the accept/reject pattern has to agree across ranks or `num_bonus_tokens` desyncs. It is free at k=1. 1. `Sampler`'s all-greedy fast path. 2. `_greedy_tokens`, for batches that are only partly greedy. The obvious spelling, `probs[greedy_mask].argmax(-1)`, gathers a [greedy, vocab] copy first, and that copy costs more than reducing the rows it would have dropped: 256 rows of 200064 with three quarters greedy, 134.6us against 71.9. 3. `ParallelLMHead`'s draft greedy argmax at tp<=1, writing through `output_idx` rather than a returned tensor -- a draft pass is recorded, and a capture bakes the address the answer lands at. 4. `rejection_sample`'s target argmax on both greedy paths, and the relaxed path's top-N. The top-N keeps `sorted=True`, which is what puts the top-1 in slot 0 for `top1_probs`. Among EQUAL probabilities the slot order is unspecified, here as it was under `torch.topk`: measured over 64-256 rows of three vocabularies, the id set per row is identical every time and the values are bitwise equal, so `valid_mask` and the kernel's membership test are unaffected; slot 0 moved on 2 rows of 256, and in both the two ids held the same probability. Also drops a now-redundant `.to(torch.int32)` -- the indices come back int32 -- and modernises one `Optional` in the same signature block. Gates ----- GSM8K, DeepSeek-V4-Flash-DSpark tp2 MTP3 fp8 `--level 0`, 1319 questions, three runs: 0.9484 / 0.9522 / 0.9484 flexible, mean 0.9497 against a recorded baseline of 0.9522 +- 0.0059. Every sample inside one sigma, and all three clear the 0.94 threshold. That exercises 1, 2 and 4. Unit tests: `tests/test_lm_head_argmax_out.py` and the new `tests/test_sampler_greedy_rows.py`. The suite's 13 other failures were compared with and without this change -- the sets are identical, so they are not from here.
Contributor
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
valarLip
marked this pull request as ready for review
September 18, 2026 16:59
ganyi1996ppo
added a commit
that referenced
this pull request
Sep 28, 2026
…strides
Two defects in the flydsl index-score integration, both found by
measurement after rebasing onto main.
1. The ubatch work-map buffer was sized with the default cp_world=1
while `_build_ubatch_metadata` builds it with the real CP pair. The
shard's dispatch is denser than the whole context's, so the CP map is
LONGER, not shorter -- 128 rows against 64 at bs=8, max_block=64 --
and the build rejects the buffer:
ValueError: work_map: expected packed [at least 128, 2] int32
So ubatch plus indexer CP could not start. It fails loudly rather
than writing out of bounds, which is why it never showed as
corruption. The non-ubatch buffer next to it already passed the pair;
this just does the same.
2. The three score strides were `tl.constexpr`. `S_HEAD` is
`tokens * local`, so it moves with the batch, and specializing on it
cost a ~430ms Triton recompile for every distinct decode batch size:
8 of 8 sizes recompiled with a cold Triton cache, 0 of 8 after. That
is the same reason `LOCAL_BLOCKS` and `GLOBAL_BLOCKS` are already
runtime arguments in this kernel. Triton still folds the
divisible-by-16 and equal-to-1 cases, which is where the addressing
win is; nothing about the layout handling changes.
tests/test_minimax_m3_index_topk.py, tests/model_ops/test_indexer_cp_parity.py
and tests/test_paged_attention_dispatch.py pass (216 passed, 21 skipped).
Not verified end to end on a server: main now imports `aiter.topk_select`
(#2244, #2314) and the aiter installed here does not have it, so the
engine cannot start from this base at all -- with or without this branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
edwingao28
pushed a commit
to SemiAnalysisAI/InferenceX
that referenced
this pull request
Oct 1, 2026
…V4 ATOM AgentX 镜像更新至 nightly_202609291501 (#3605) * perf(amd): bump DSV4 ATOM AgentX image to nightly_202609291501 Image-only update of dsv4-fp4-mi355x-atom-agentic-mtp on its native srt-slurm recipe; points, DSpark K6, golden AL and env are unchanged. 将 dsv4-fp4-mi355x-atom-agentic-mtp 的原生 srt-slurm 配方镜像更新为 nightly_202609291501;测试点、DSpark K6、golden AL 和环境变量均不变。 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * chore: set perf-changelog pr-link to #3605 将 perf-changelog 的 pr-link 设为 #3605。 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * perf(amd): bump DSV4 ATOM AgentX image to nightly_202609291501 Update the ATOM image from rocm/atom-dev:nightly_202609161445 (ATOM 0.1.6rc1.dev477+g709c25be0) to rocm/atom-dev:nightly_202609291501 (digest sha256:e0d7134efff68c251acdc2612a9b6c238bc4da5d613832b2bd1251f34bd93355; ATOM 0.1.7.dev46+g74fd942b0, ROCm 7.2.4). Image-only change on the existing native srt-slurm recipe: all ten points (TP8/EP1 at concurrency 1, 2, 4, 8, 16; TP8/DPA8/EP8 native RCCL at 48, 64, 96, 128, 256), DSpark K6 with the thinking_on golden AL 3.77 for throughput and real acceptance for eval, KV and index-cache dtypes, graph capture and all environment settings are unchanged. Relevant ATOM changes in the range: V4 decode reuses the sparse prefill ASM (ROCm/ATOM#2271), greedy sampler picks via aiter.topk_select (ROCm/ATOM#2244), and FP8 block scales declared scale_fmt ue8m0 are now stored as E8M0 on gfx950 by default (ROCm/ATOM#2419; previously FP32 unless ATOM_FP8_BLOCKSCALE_USE_E8M0_SCALE=1). The upstream DeepSeek-V4 recipes are unchanged across the range, and every recipe flag and choice (all2all-backend rccl, dp-load-balance least_tokens, moe-backend standard) remains valid. No data-type or precision change to the DeepSeek-V4-Pro-0813 DSpark draft: no online quantization is configured, so it keeps its checkpoint precision; the E8M0 scale storage represents the checkpoint's power-of-two block scales exactly. kv-cache-dtype and index-cache-dtype touch cache storage only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Honglie Yi <hyi@crsuse2-m2m-v2-020.us-east2-a.compute.internal> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: seungrokj <144636725+seungrokj@users.noreply.github.com> Co-authored-by: seungrokj <seungrok.jung@amd.com>
5 tasks
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Four call sites reduced a
[rows, vocab]tensor withtorch.argmaxortorch.topk.aiter.topk_selectdispatches the same reduction over whichever of its backends is fastest for the shape, and takes bf16 without a cast in front of it -- the k=1 reduction folds int32 ordering keys, so it widens each element as it reads rather than materialising a[rows, vocab]fp32 copy.Sampler's all-greedy fast path_greedy_tokens, partly-greedy batchesprobs[greedy_mask].argmax(-1)gathers a[greedy, vocab]copy first, and that copy costs more than reducing the rows it would have dropped: 256 rows of 200064 with three quarters greedy, 134.6us against 71.9ParallelLMHead's draft greedy argmax attp<=1output_idxrather than a returned tensor -- a draft pass is recorded, and a capture bakes the address the answer lands atrejection_sample's target argmax, both greedy paths, and the relaxed path's top-N.to(torch.int32)The tie argument
Every call passes
tie="low", and that is the whole reason the answer is the picktorch.argmaxmade. The default promises no direction among equal scores, and neither didtorch.topk-- so a near-tie would resolve differently from one TP rank to the next, and in the rejection sampler the accept/reject pattern has to agree across ranks ornum_bonus_tokensdesyncs. It is free at k=1.The relaxed top-N is the one place where slot order among equal probabilities stays unspecified, exactly as it was under
torch.topk. Measured over 64-256 rows of three vocabularies: the id set per row is identical every time and the values are bitwise equal, sovalid_maskand the kernel's membership test are unaffected. Slot 0 -- which the kernel emits as the correction token -- moved on 2 rows of 256, and in both the two ids held the same probability.Test plan
GSM8K, DeepSeek-V4-Flash-DSpark tp2 MTP3 fp8
--level 0, 1319 questions, three runs:Against a recorded baseline of 0.9522 ± 0.0059: the mean is −0.43σ, every sample is inside one sigma, and all three clear the 0.94 threshold. This exercises sites 1, 2 and 4 -- GSM8K is greedy, and MTP3 drives the rejection sampler.
tests/test_lm_head_argmax_out.py(5) and the newtests/test_sampler_greedy_rows.py(19).The suite's 13 other failures were run with and without this change; the failure sets are identical, so they are not from here.
Site 3 has no end-to-end coverage.
ParallelLMHead's change is on thetp_size <= 1branch, and the gate above runs tp2, which takes the pack + all-gather branch instead. It is covered bytest_lm_head_argmax_out.pyonly. An e2e run would need tp1 with a drafter (--method dspark).