Skip to content

[Perf] Use finer gfx1250 index-score chunks for small decode grids - #2873

Open
xytpai wants to merge 3 commits into
ROCm:mainfrom
xytpai:perf/gfx1250-small-decode-score-chunks-1031
Open

xytpai wants to merge 3 commits into
ROCm:mainfrom
xytpai:perf/gfx1250-small-decode-score-chunks-1031

Conversation

@xytpai

@xytpai xytpai commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Use a finer block-axis split in the existing Triton MiniMax-M3 decode index scorer for gfx1250, batch1/2, small total index-block grids.

  • If batch * max_block <= 256, use one logical128-token block per CTA.
  • If the product is <=1024, use two blocks per CTA.
  • Only applies to the native tiled decode scorer, batch1/2 on gfx1250. indexer_context_scores (CP) keeps the original shared helper and grid. Other architectures, larger batches and longer contexts keep the old chunk-floor/grid-cap formula. Empty bounds still return one grid chunk.
  • Early return is algebraically identical to applying the corresponding1/2 block floor and round-trip chunk formula in this bounded region; it avoids extra Python integer work on the hot path.
  • No kernel math, query/key dtype conversion, stage count, top-k/forced-block policy, sparse-table layout or prefill launch policy changes. Decode retains stage3 and original query precision.

The policy is empirical, not a hardware-counter-proven root-cause claim. A stage1 experiment was slower and is explicitly not included.

Validation

Base 43eed2fde149e0d2e8986964af46fe1a02ff52ca; head 75bc566a03d3b5a96d576871fed55a1400409256.

  • Final release on two independent gfx1250 GPUs/processes/caches: 226 passed,1 skipped each for python3 -m pytest -q tests/test_minimax_m3_index_topk.py.
  • GPU-hidden: 130 passed,97 skipped. Black with --target-version py312, Ruff and git diff --check passed.
  • Sole GPU skip: optional AITER topk_per_row_small_k selector is absent from the AITER checkout; the Triton selector is exercised. No coverage claim for that unavailable alternative backend.
  • Original minimum-floor assertions are retained under gfx950; separate exact gfx942/gfx950/gfx1250 policy tests cover batch and256/1024 boundaries, empty bounds, positive grid, no empty last chunk and target-grid cap.
  • GPU comparisons include strided queries, shuffled physical page tables, BF16/FP8 caches, ordinary/speculative decode, zero-query ties, all effective scores bitwise equal, complete top-k/sparse-table/context outputs equal, independent sampled dot/max reference, and changed query/length graph replay on a non-default stream.
  • Separate46-shape randomized A/B/A validation on each GPU;32 additional query/head combinations on GPU1 (up to16 query rows ×4 heads or4 query rows ×8 heads).
  • Final early-return rewrite equals the frozen measured grid policy over 122,910 CPU cases; final GPU tests/eager timings were rerun afterward. Both forms select identical graph kernels/grids.

An exploratory128-token batch2 input accidentally generated a negative sequence length; baseline and candidate both returned -inf and the test oracle failed. The input generator was corrected to require max_query_len <= seq_len <= max_seq_len, and the complete54-shape boundary sweep was rerun. No numerical tolerance was relaxed.

Full score + top-k + sparse-table graph timing

Python3.12, Torch2.11 ROCm build, Triton3.9, FlyDSL0.4.1, image rocm/fw-bringup-mlperf:gfx1250-atom-20261023. No profiler.25 randomized A/B/A samples ×12 graph replays ×16 complete operator calls; warmups excluded. Includes score, selection and emitted sparse tables, not just the score kernel. Baseline and candidate use the same input values; graph creation/replay order randomized, original-path A/A and unchanged-grid negative controls retained.

Representative TP4-local shape: BF16 Q, FP8 index cache, four speculative query rows, one index head, batch1:

Context Old chunks New chunks GPU0 old µs GPU0 new µs GPU1 old µs GPU1 new µs
8192 22 64 8.541 7.844 8.542 7.831
32768 86 256 9.157 8.149 8.984 8.093
65536 171 256 9.607 8.865 9.352 8.896
105301 275 412 11.303 10.875 11.367 10.789
131072 342 512 11.833 11.325 11.662 11.090
262144 683 683 16.126 16.257 16.540 16.419

Of46 shapes,26 change the grid: GPU0 relative full-op differences were -14.13% to -3.79%, GPU1 -13.95% to+0.31% (the latter is a BF16 boundary case, effectively flat). Additional wide-query/head changed-grid cases improved ~5.14–10.31% in that independent GPU1 sweep. These ranges are descriptive, not broad guarantees. Unchanged-grid and A/A timing differences exist and are not claimed as improvements.

Eager host/completion timing

Final early-return policy,21 randomized paired rounds ×120 full operator calls, synchronized around the block;50 warmups/arm. On changed paths the CPU submission median saving reproduced at ~1.4–2.2µs/call on both GPUs; completed timings tracked it. These calls were host-bound, so GPU graph gains should not be added to the host savings.

For batch1/context105301/one head: GPU0 host52.42→50.70µs, GPU1 52.03→50.34µs. Batch2/context105301 is unchanged policy and showed ~0.11–0.16µs host regression; batch8/context105301 was approximately flat. All raw data retained.

The initial implementation changed the floor inside the full formula and cost~0.1–0.2µs extra host time despite GPU gains. It was replaced by the equivalent early-return formula, then revalidated. That failed version is retained, not reported as a win.

Scope / limitations

Whole-model controlled-C1 ABBA is complete; it did not establish a throughput gain (full results below). No production quality or original900s serving-load gain is claimed. Other architectures retain their original policy but were CPU-policy-tested, not GPU-benchmarked here. The available selector is Triton; alternative AITER selection and distributed indexer-CP transport were not validated. Graph timing has visible A/A noise, especially for tiny unchanged-grid cases.

Artifacts on f01: ~/bench/m3_decode_chunk_policy_20261031/ (release validation, JUnit, both graph raw JSONs, wide-head samples, old/final eager results, exact policy-equivalence check); exploratory stage/chunk sweeps in ~/bench/m3_decode_score_stages_20261031/.

PR Self-Review

Reviewed revision

38439469009d9b5236caeaff85398fd212150e7d

Behavior and intent

The gfx1250 small-batch index-score grid gets more parallel CTAs within explicitly bounded total-block ranges. The previous uniform3-block floor was slower there in controlled full-operator measurements. Only launch chunking and its host formula change, not numeric operations or selector semantics.

Validation and unverified scope

python3 -m pytest -q tests/test_minimax_m3_index_topk.py:226 passed/1 optional-selector skip on each of two gfx1250 GPUs;130 passed/97 skipped GPU-hidden. Black/Ruff/diff checks pass. Python3.12/Torch2.11/Triton3.9 in the ROCm image above. Unverified: whole-model performance/accuracy, other-GPU runtime performance, optional AITER selector, distributed indexer-CP transport. Explicit negative cases and host regressions are listed above.

PR-specific question

The new gfx1250 exact-policy boundary assertions fail before this PR where the old3-block floor is selected; numeric/graph checks are preservation tests expected to pass on old valid inputs. Original gfx950 floor tests remain intact via an explicit architecture fixture. Existing arithmetic tests alone cannot detect the unnecessary launch partitioning because both chunk layouts produce the same valid scores.

Changed tests

tests/test_minimax_m3_index_topk.py: preserves original-architecture floor coverage, adds explicit architecture/batch/total-block boundaries and no-empty-grid invariants, and exercises changed-input decode score/selection/page-table outputs with independent sampled reference and graph replay. It reuses existing module constants and established test structure without changing tolerances of existing tests.

Complete controlled-C1 model ABBA

Verified four-arm report.
Frozen M3 TP4/Simple/FP8-KV/EAGLE3 stack, identical AITER/cache sources; only this ATOM chunk policy differs. One outstanding request and max_num_seqs=1, retained102784-token prefix + fresh256-token suffix +768 output tokens. Six warmup and12 measured decode requests plus10 short/10 long prefills per process.

All four ranks had exactly one256-token prefill and only4-token decode forward shapes per measured C1 request. Source/script hashes and actual container import-source hashes verified; all32 measured input/output hashes match; no compiler/cache-load events recorded in measured windows.

Descriptive mean-of-process-medians: C1 throughput +0.20%, short prefill latency +0.51%, long-prefix prefill latency +6.01%. All outcomes are retained. These sequential four-process data do not establish a significant whole-model speedup. Prefill policy and math are unchanged by this patch; that fact alone is not proof of what caused the measured prefill slowdown. Process/system drift remains possible, and no favorable subsets are substituted for the original results.

The evidence for the optimization is the controlled full index-score/selection/sparse-table operator measurements. Synthetic acceptance is performance-only, not generated-text accuracy.

CP scope correction after review

Revision 85a814b684bd1932a1dd32735bc74ed5d212612d fixes an unintended scope expansion in the first PR revision. The reviewer correctly noted that indexer_context_scores also imports _decode_score_chunks; the original change would have retuned its distinct CP scorer without CP performance evidence.

The shared _decode_score_chunks is now restored exactly to pre-PR code. A separate _native_decode_score_chunks is called only by minimax_m3_index_topk_decode. The CP source file is unchanged. Native decode chunk counts match the previous measured policy in 122,910 cases.

Validation of this revision:

  • python3 -m pytest -q tests/test_minimax_m3_index_topk.py tests/model_ops/test_indexer_cp_parity.py: 263 passed, 1 optional-selector skip on each of two gfx1250 GPUs; CPU-hidden 139 passed, 125 skipped.
  • New CP tests verify its original grid on real kernel launches and changed-input graph replay; existing CP score→candidate-exchange→merge parity tests also pass. Transport in the canonical parity tests is simulated in one process, not a new distributed CP performance measurement.
  • Old PR-revision negative control: 2 failed, 7 passed, catching unintended gfx1250 batch1/2 CP grid changes while unchanged controls pass.
  • Final native eager recheck still saves about 1.4–2.1µs on changed paths. Unchanged-policy batch2/context105301 and batch8 controls showed about 0.23µs/0.14µs extra host cost in this run; these are retained rather than claimed as improvements.
  • Black/Ruff and diff checks pass.

The earlier native graph/model results are not reruns of this scope-only revision. Native grid values are proven identical, with final GPU numerical/eager checks above; no CP performance claim is made, and CP now has no policy change to benchmark. This addresses the reviewer’s concern rather than extrapolating native-scorer timing to CP.

Comment clarification in 384394690: the shared helper describes the legacy policy used by CP, with an explicit pointer to the separately measured native gfx1250 exception. Executable AST unchanged; CPU regression 139 passed / 125 GPU skips, Black/Ruff passed. No new performance or GPU-test rerun claimed for this documentation-only edit.

@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2873 --add-label <label>

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few small things..

ci green.

Comment thread atom/model_ops/minimax_m3/index_topk.py Outdated
# In this bounded gfx1250 region the existing target-grid cap cannot
# bind (at most 1024 blocks across one or two requests). One block/CTA
# below 256 total blocks, otherwise two, also leaves no empty tail CTA.
if (

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this helper is also used by indexer_context_scores, so cp score launch changes too on gfx1250. intended? any perf number there?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right: the first revision unintentionally changed the CP scorer's grid through the shared helper, and I had no CP performance evidence for that tuning. Fixed in 85a814b6: _decode_score_chunks is restored exactly to pre-PR code; only native minimax_m3_index_topk_decode calls the new _native_decode_score_chunks. indexer_context_parallel.py is unchanged.

Added a CPU CP-grid negative control (old PR revision: 2 fail/7 pass), plus real CP kernel grid interception and changed-input graph replay. Ran the complete native index-topk and canonical CP composition/parity suites on both gfx1250 GPUs: 263 passed/1 optional-selector skip each; CPU 139 passed/125 skipped. Native chunk values match the previously benchmarked policy over 122,910 cases; final eager savings remain about 1.4–2.1µs on changed paths. No CP performance claim or transport benchmark is made; CP policy now stays unchanged.

@xytpai

xytpai commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

Controlled-C1 model ABBA: decode index-score chunk policy

Independent validation of ATOM #2873 on a frozen model stack. Both arms share exactly the same AITER/kernel/cache code; only the ATOM gfx1250 small-batch decode index-score chunk policy differs. This does not replace earlier unrelated C2/fused-route results.

Both arms set max_num_seqs=1. Exactly one outstanding HTTP request, each measured decode request: same102784-token retained prefix + fresh256-token suffix +768 generated tokens. Six warmup requests, then12 measured C1 requests/process; ten short and ten long prefill samples. M3 TP4/Simple/FP8-KV/EAGLE3, synthetic acceptance0.5933.

All four ranks verified: exactly one256-token prefill and only4-token decode forward shapes per measured C1 request. Request windows do not overlap. Input hashes/import paths/source and script hashes pass. No profiler enabled; equal forward/compiler event hooks in both arms.

Order Arm Short median ms Long median ms C1 output tok/s
1 baseline_a 84.053 88.441 418.34
2 gated_b 86.095 90.982 419.84
3 gated_c 84.029 95.596 418.71
4 baseline_d 85.216 87.557 418.54

Descriptive mean-of-process-medians changes (not statistical significance):

  • short_s: +0.51%.
  • long_s: +6.01%.
  • c1_tokens_s: +0.20%.

Recorded timed compiler-listener events: {'baseline_a': 0, 'gated_b': 0, 'gated_c': 0, 'baseline_d': 0}.
No recorded FlyDSL/Triton compilation or cache-load events in measured request windows.

Output hashes matching baseline A: {'baseline_a': 32, 'gated_b': 32, 'gated_c': 32, 'baseline_d': 32} out of32 measured requests. This is not an accuracy/quality oracle; acceptance remains synthetic.

Arm Rank0 median of per-request decode-forward medians (host ms)
baseline_a 5.6973
gated_b 5.7065
gated_c 5.7284
baseline_d 5.7340

Limitations: sequential ABBA, only two baseline/two candidate processes, process drift and server scheduling overhead remain. Host forward timing is not GPU-event kernel latency. This C1 protocol cannot establish C2/concurrent serving gains or the original900s load result. No quality claim.

Artifacts on f01: ~/bench/m3_decode_chunk_model_20261031/verified.json, raw per-request/rank event logs, manifest and launch scripts. Verification: python3 ~/bench/m3_decode_chunk_model_20261031/verify.py.

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few small things..

ci green.



def _native_decode_score_chunks(batch: int, max_block: int) -> int:
"""Measured native decode policy; CP keeps the shared legacy policy.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this comment now contradicts the native branch below.. add a line there so it's not stale

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 384394690: the shared helper now explicitly scopes its chunk floor to the legacy/CP policy and points to _native_decode_score_chunks for the measured one/two-block gfx1250 exception. Removed the blanket 'every measured regression' statement. Executable AST is unchanged after stripping docstrings; 139 CPU tests passed / 125 GPU cases skipped; Black/Ruff pass. No logic or new performance claim in this edit.

@zufayu
zufayu requested review from JiaoliangYu and removed request for ZhangLirong-amd November 9, 2026 02:00
@zufayu zufayu assigned JiaoliangYu and unassigned ZhangLirong-amd Nov 9, 2026
lirui927 pushed a commit that referenced this pull request Nov 10, 2026
… candidate exchange (#3116)

Squash of #3116 at 52f35c2, by @lirui927.
On top of #2873 here, its two CP chunk-policy tests now check the
#3116 score grid instead of _decode_score_chunks (the CP scorer no longer
uses it); they still fail if CP reaches the native chunk policy.

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (2)

  1. Fix the module comment that still says one-block chunks are always a loss. atom/model_ops/minimax_m3/index_topk.py:85-87 still states "at one block per chunk that fixed cost is the whole cost". _native_decode_score_chunks (:683-688) now launches exactly one-block chunks on gfx1250. valar-cop flagged this same contradiction at :676; only that copy was updated. Scope the comment to the shared/CP policy and point it at the native exception.
  2. Make the gfx1250 thresholds named constants next to DECODE_SCORE_TARGET_GRID / DECODE_SCORE_MIN_BLOCKS. :683-688 hardcodes batch <= 2, 256 and 1024, while every sibling launch knob in this file is a module constant with a one-line note on where its value comes from. Hoist these three the same way, e.g. DECODE_SCORE_FINE_MAX_BATCH, DECODE_SCORE_ONE_BLOCK_GRID, DECODE_SCORE_TWO_BLOCK_GRID. While there, drop the "legacy" / "measured" PR-history wording at :647, :668 and :676-681, and describe each policy by what it does.

Root cause and placement are fine: _decode_index_score_tiled_kernel is ATOM's own Triton kernel, so ATOM is the only place that can choose its grid. The gfx1250 gating follows the existing _prefill_score_launch_config pattern. Evidence is op-level only: ~1 µs (7–14%) on the full index op, while the C1 model ABBA (+0.20%) is within noise. No code bugs found. Pre-checkin CI is green; the heavy model tests have not run, and CI hardware will not exercise this gfx1250-only branch.


Reviewed at 3843946.

@valar-cop

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants