Skip to content

perf(sampler): take the greedy picks from aiter.topk_select - #2244

Merged
valarLip merged 2 commits into
mainfrom
perf/topk-select-adoption
Sep 19, 2026
Merged

valarLip merged 2 commits into
mainfrom
perf/topk-select-adoption

Conversation

@valarLip

Copy link
Copy Markdown
Collaborator

Draft: depends on an unmerged aiter PR. tie="low" and the output_idx= argument these call sites rely on are in ROCm/aiter#5526. Do not merge before that lands.

What this does

Four call sites reduced a [rows, vocab] tensor with torch.argmax or torch.topk. aiter.topk_select dispatches the same reduction over whichever of its backends is fastest for the shape, and takes bf16 without a cast in front of it -- the k=1 reduction folds int32 ordering keys, so it widens each element as it reads rather than materialising a [rows, vocab] fp32 copy.

site why
1 Sampler's all-greedy fast path the reduction itself
2 _greedy_tokens, partly-greedy batches probs[greedy_mask].argmax(-1) gathers a [greedy, vocab] copy first, and that copy costs more than reducing the rows it would have dropped: 256 rows of 200064 with three quarters greedy, 134.6us against 71.9
3 ParallelLMHead's draft greedy argmax at tp<=1 writes through output_idx rather than a returned tensor -- a draft pass is recorded, and a capture bakes the address the answer lands at
4 rejection_sample's target argmax, both greedy paths, and the relaxed path's top-N the reduction, and one fewer .to(torch.int32)

The tie argument

Every call passes tie="low", and that is the whole reason the answer is the pick torch.argmax made. The default promises no direction among equal scores, and neither did torch.topk -- so a near-tie would resolve differently from one TP rank to the next, and in the rejection sampler the accept/reject pattern has to agree across ranks or num_bonus_tokens desyncs. It is free at k=1.

The relaxed top-N is the one place where slot order among equal probabilities stays unspecified, exactly as it was under torch.topk. Measured over 64-256 rows of three vocabularies: the id set per row is identical every time and the values are bitwise equal, so valid_mask and the kernel's membership test are unaffected. Slot 0 -- which the kernel emits as the correction token -- moved on 2 rows of 256, and in both the two ids held the same probability.

Test plan

  • GSM8K, DeepSeek-V4-Flash-DSpark tp2 MTP3 fp8 --level 0, 1319 questions, three runs:

    flexible strict
    run 1 0.9484 0.9492
    run 2 0.9522 0.9530
    run 3 0.9484 0.9492
    mean 0.9497 0.9505

    Against a recorded baseline of 0.9522 ± 0.0059: the mean is −0.43σ, every sample is inside one sigma, and all three clear the 0.94 threshold. This exercises sites 1, 2 and 4 -- GSM8K is greedy, and MTP3 drives the rejection sampler.

  • tests/test_lm_head_argmax_out.py (5) and the new tests/test_sampler_greedy_rows.py (19).

  • The suite's 13 other failures were run with and without this change; the failure sets are identical, so they are not from here.

  • Site 3 has no end-to-end coverage. ParallelLMHead's change is on the tp_size <= 1 branch, and the gate above runs tp2, which takes the pack + all-gather branch instead. It is covered by test_lm_head_argmax_out.py only. An e2e run would need tp1 with a drafter (--method dspark).

Four call sites reduced a [rows, vocab] tensor with `torch.argmax` or
`torch.topk`. `aiter.topk_select` dispatches the same reduction over whichever
of its backends is fastest for the shape, and takes bf16 without a cast in
front of it -- the k=1 reduction folds int32 ordering keys, so it widens each
element as it reads rather than materialising a [rows, vocab] fp32 copy.

Every call passes `tie="low"`, and that is the whole reason the answer is the
pick `torch.argmax` made. The default promises no direction among equal scores,
and neither did `torch.topk`, so a near-tie would resolve differently from one
TP rank to the next -- and in the rejection sampler the accept/reject pattern
has to agree across ranks or `num_bonus_tokens` desyncs. It is free at k=1.

1. `Sampler`'s all-greedy fast path.

2. `_greedy_tokens`, for batches that are only partly greedy. The obvious
   spelling, `probs[greedy_mask].argmax(-1)`, gathers a [greedy, vocab] copy
   first, and that copy costs more than reducing the rows it would have
   dropped: 256 rows of 200064 with three quarters greedy, 134.6us against
   71.9.

3. `ParallelLMHead`'s draft greedy argmax at tp<=1, writing through
   `output_idx` rather than a returned tensor -- a draft pass is recorded, and
   a capture bakes the address the answer lands at.

4. `rejection_sample`'s target argmax on both greedy paths, and the relaxed
   path's top-N. The top-N keeps `sorted=True`, which is what puts the top-1 in
   slot 0 for `top1_probs`. Among EQUAL probabilities the slot order is
   unspecified, here as it was under `torch.topk`: measured over 64-256 rows of
   three vocabularies, the id set per row is identical every time and the
   values are bitwise equal, so `valid_mask` and the kernel's membership test
   are unaffected; slot 0 moved on 2 rows of 256, and in both the two ids held
   the same probability.

Also drops a now-redundant `.to(torch.int32)` -- the indices come back int32 --
and modernises one `Optional` in the same signature block.

Gates
-----
GSM8K, DeepSeek-V4-Flash-DSpark tp2 MTP3 fp8 `--level 0`, 1319 questions,
three runs: 0.9484 / 0.9522 / 0.9484 flexible, mean 0.9497 against a recorded
baseline of 0.9522 +- 0.0059. Every sample inside one sigma, and all three
clear the 0.94 threshold. That exercises 1, 2 and 4.

Unit tests: `tests/test_lm_head_argmax_out.py` and the new
`tests/test_sampler_greedy_rows.py`. The suite's 13 other failures were
compared with and without this change -- the sets are identical, so they are
not from here.
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2244 --add-label <label>

@valarLip
valarLip marked this pull request as ready for review September 18, 2026 16:59
@valarLip
valarLip merged commit b52b78b into main Sep 19, 2026
33 of 48 checks passed
@valarLip
valarLip deleted the perf/topk-select-adoption branch September 19, 2026 07:16
ganyi1996ppo added a commit that referenced this pull request Sep 28, 2026
…strides

Two defects in the flydsl index-score integration, both found by
measurement after rebasing onto main.

1. The ubatch work-map buffer was sized with the default cp_world=1
   while `_build_ubatch_metadata` builds it with the real CP pair. The
   shard's dispatch is denser than the whole context's, so the CP map is
   LONGER, not shorter -- 128 rows against 64 at bs=8, max_block=64 --
   and the build rejects the buffer:

       ValueError: work_map: expected packed [at least 128, 2] int32

   So ubatch plus indexer CP could not start. It fails loudly rather
   than writing out of bounds, which is why it never showed as
   corruption. The non-ubatch buffer next to it already passed the pair;
   this just does the same.

2. The three score strides were `tl.constexpr`. `S_HEAD` is
   `tokens * local`, so it moves with the batch, and specializing on it
   cost a ~430ms Triton recompile for every distinct decode batch size:
   8 of 8 sizes recompiled with a cold Triton cache, 0 of 8 after. That
   is the same reason `LOCAL_BLOCKS` and `GLOBAL_BLOCKS` are already
   runtime arguments in this kernel. Triton still folds the
   divisible-by-16 and equal-to-1 cases, which is where the addressing
   win is; nothing about the layout handling changes.

tests/test_minimax_m3_index_topk.py, tests/model_ops/test_indexer_cp_parity.py
and tests/test_paged_attention_dispatch.py pass (216 passed, 21 skipped).

Not verified end to end on a server: main now imports `aiter.topk_select`
(#2244, #2314) and the aiter installed here does not have it, so the
engine cannot start from this base at all -- with or without this branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
edwingao28 pushed a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 1, 2026
…V4 ATOM AgentX 镜像更新至 nightly_202609291501 (#3605)

* perf(amd): bump DSV4 ATOM AgentX image to nightly_202609291501

Image-only update of dsv4-fp4-mi355x-atom-agentic-mtp on its native
srt-slurm recipe; points, DSpark K6, golden AL and env are unchanged.

将 dsv4-fp4-mi355x-atom-agentic-mtp 的原生 srt-slurm 配方镜像更新为
nightly_202609291501;测试点、DSpark K6、golden AL 和环境变量均不变。

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* chore: set perf-changelog pr-link to #3605

将 perf-changelog 的 pr-link 设为 #3605。

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* perf(amd): bump DSV4 ATOM AgentX image to nightly_202609291501

Update the ATOM image from rocm/atom-dev:nightly_202609161445 (ATOM 0.1.6rc1.dev477+g709c25be0) to rocm/atom-dev:nightly_202609291501 (digest sha256:e0d7134efff68c251acdc2612a9b6c238bc4da5d613832b2bd1251f34bd93355; ATOM 0.1.7.dev46+g74fd942b0, ROCm 7.2.4). Image-only change on the existing native srt-slurm recipe: all ten points (TP8/EP1 at concurrency 1, 2, 4, 8, 16; TP8/DPA8/EP8 native RCCL at 48, 64, 96, 128, 256), DSpark K6 with the thinking_on golden AL 3.77 for throughput and real acceptance for eval, KV and index-cache dtypes, graph capture and all environment settings are unchanged.

Relevant ATOM changes in the range: V4 decode reuses the sparse prefill ASM (ROCm/ATOM#2271), greedy sampler picks via aiter.topk_select (ROCm/ATOM#2244), and FP8 block scales declared scale_fmt ue8m0 are now stored as E8M0 on gfx950 by default (ROCm/ATOM#2419; previously FP32 unless ATOM_FP8_BLOCKSCALE_USE_E8M0_SCALE=1). The upstream DeepSeek-V4 recipes are unchanged across the range, and every recipe flag and choice (all2all-backend rccl, dp-load-balance least_tokens, moe-backend standard) remains valid.

No data-type or precision change to the DeepSeek-V4-Pro-0813 DSpark draft: no online quantization is configured, so it keeps its checkpoint precision; the E8M0 scale storage represents the checkpoint's power-of-two block scales exactly. kv-cache-dtype and index-cache-dtype touch cache storage only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Honglie Yi <hyi@crsuse2-m2m-v2-020.us-east2-a.compute.internal>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: seungrokj <144636725+seungrokj@users.noreply.github.com>
Co-authored-by: seungrokj <seungrok.jung@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant