Repository navigation
Conversation
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
| # In this bounded gfx1250 region the existing target-grid cap cannot | ||
| # bind (at most 1024 blocks across one or two requests). One block/CTA | ||
| # below 256 total blocks, otherwise two, also leaves no empty tail CTA. | ||
| if ( |
There was a problem hiding this comment.
this helper is also used by indexer_context_scores, so cp score launch changes too on gfx1250. intended? any perf number there?
There was a problem hiding this comment.
You're right: the first revision unintentionally changed the CP scorer's grid through the shared helper, and I had no CP performance evidence for that tuning. Fixed in 85a814b6: _decode_score_chunks is restored exactly to pre-PR code; only native minimax_m3_index_topk_decode calls the new _native_decode_score_chunks. indexer_context_parallel.py is unchanged.
Added a CPU CP-grid negative control (old PR revision: 2 fail/7 pass), plus real CP kernel grid interception and changed-input graph replay. Ran the complete native index-topk and canonical CP composition/parity suites on both gfx1250 GPUs: 263 passed/1 optional-selector skip each; CPU 139 passed/125 skipped. Native chunk values match the previously benchmarked policy over 122,910 cases; final eager savings remain about 1.4–2.1µs on changed paths. No CP performance claim or transport benchmark is made; CP policy now stays unchanged.
Controlled-C1 model ABBA: decode index-score chunk policyIndependent validation of ATOM #2873 on a frozen model stack. Both arms share exactly the same AITER/kernel/cache code; only the ATOM gfx1250 small-batch decode index-score chunk policy differs. This does not replace earlier unrelated C2/fused-route results. Both arms set max_num_seqs=1. Exactly one outstanding HTTP request, each measured decode request: same102784-token retained prefix + fresh256-token suffix +768 generated tokens. Six warmup requests, then12 measured C1 requests/process; ten short and ten long prefill samples. M3 TP4/Simple/FP8-KV/EAGLE3, synthetic acceptance0.5933. All four ranks verified: exactly one256-token prefill and only4-token decode forward shapes per measured C1 request. Request windows do not overlap. Input hashes/import paths/source and script hashes pass. No profiler enabled; equal forward/compiler event hooks in both arms.
Descriptive mean-of-process-medians changes (not statistical significance):
Recorded timed compiler-listener events: Output hashes matching baseline A:
Limitations: sequential ABBA, only two baseline/two candidate processes, process drift and server scheduling overhead remain. Host forward timing is not GPU-event kernel latency. This C1 protocol cannot establish C2/concurrent serving gains or the original900s load result. No quality claim. Artifacts on f01: |
|
|
||
|
|
||
| def _native_decode_score_chunks(batch: int, max_block: int) -> int: | ||
| """Measured native decode policy; CP keeps the shared legacy policy. |
There was a problem hiding this comment.
this comment now contradicts the native branch below.. add a line there so it's not stale
There was a problem hiding this comment.
Fixed in 384394690: the shared helper now explicitly scopes its chunk floor to the legacy/CP policy and points to _native_decode_score_chunks for the measured one/two-block gfx1250 exception. Removed the blanket 'every measured regression' statement. Executable AST is unchanged after stripping docstrings; 139 CPU tests passed / 125 GPU cases skipped; Black/Ruff pass. No logic or new performance claim in this edit.
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (2)
- Fix the module comment that still says one-block chunks are always a loss.
atom/model_ops/minimax_m3/index_topk.py:85-87still states "at one block per chunk that fixed cost is the whole cost"._native_decode_score_chunks(:683-688) now launches exactly one-block chunks on gfx1250. valar-cop flagged this same contradiction at:676; only that copy was updated. Scope the comment to the shared/CP policy and point it at the native exception. - Make the gfx1250 thresholds named constants next to
DECODE_SCORE_TARGET_GRID/DECODE_SCORE_MIN_BLOCKS.:683-688hardcodesbatch <= 2,256and1024, while every sibling launch knob in this file is a module constant with a one-line note on where its value comes from. Hoist these three the same way, e.g.DECODE_SCORE_FINE_MAX_BATCH,DECODE_SCORE_ONE_BLOCK_GRID,DECODE_SCORE_TWO_BLOCK_GRID. While there, drop the "legacy" / "measured" PR-history wording at:647,:668and:676-681, and describe each policy by what it does.
Root cause and placement are fine: _decode_index_score_tiled_kernel is ATOM's own Triton kernel, so ATOM is the only place that can choose its grid. The gfx1250 gating follows the existing _prefill_score_launch_config pattern. Evidence is op-level only: ~1 µs (7–14%) on the full index op, while the C1 model ABBA (+0.20%) is within noise. No code bugs found. Pre-checkin CI is green; the heavy model tests have not run, and CI hardware will not exercise this gfx1250-only branch.
Reviewed at 3843946.
|
ci triage for 3843946:
|
Summary
Use a finer block-axis split in the existing Triton MiniMax-M3 decode index scorer for gfx1250, batch1/2, small total index-block grids.
batch * max_block <= 256, use one logical128-token block per CTA.indexer_context_scores(CP) keeps the original shared helper and grid. Other architectures, larger batches and longer contexts keep the old chunk-floor/grid-cap formula. Empty bounds still return one grid chunk.The policy is empirical, not a hardware-counter-proven root-cause claim. A stage1 experiment was slower and is explicitly not included.
Validation
Base
43eed2fde149e0d2e8986964af46fe1a02ff52ca; head75bc566a03d3b5a96d576871fed55a1400409256.python3 -m pytest -q tests/test_minimax_m3_index_topk.py.--target-version py312, Ruff and git diff --check passed.topk_per_row_small_kselector is absent from the AITER checkout; the Triton selector is exercised. No coverage claim for that unavailable alternative backend.An exploratory128-token batch2 input accidentally generated a negative sequence length; baseline and candidate both returned -inf and the test oracle failed. The input generator was corrected to require
max_query_len <= seq_len <= max_seq_len, and the complete54-shape boundary sweep was rerun. No numerical tolerance was relaxed.Full score + top-k + sparse-table graph timing
Python3.12, Torch2.11 ROCm build, Triton3.9, FlyDSL0.4.1, image
rocm/fw-bringup-mlperf:gfx1250-atom-20261023. No profiler.25 randomized A/B/A samples ×12 graph replays ×16 complete operator calls; warmups excluded. Includes score, selection and emitted sparse tables, not just the score kernel. Baseline and candidate use the same input values; graph creation/replay order randomized, original-path A/A and unchanged-grid negative controls retained.Representative TP4-local shape: BF16 Q, FP8 index cache, four speculative query rows, one index head, batch1:
Of46 shapes,26 change the grid: GPU0 relative full-op differences were -14.13% to -3.79%, GPU1 -13.95% to+0.31% (the latter is a BF16 boundary case, effectively flat). Additional wide-query/head changed-grid cases improved ~5.14–10.31% in that independent GPU1 sweep. These ranges are descriptive, not broad guarantees. Unchanged-grid and A/A timing differences exist and are not claimed as improvements.
Eager host/completion timing
Final early-return policy,21 randomized paired rounds ×120 full operator calls, synchronized around the block;50 warmups/arm. On changed paths the CPU submission median saving reproduced at ~1.4–2.2µs/call on both GPUs; completed timings tracked it. These calls were host-bound, so GPU graph gains should not be added to the host savings.
For batch1/context105301/one head: GPU0 host52.42→50.70µs, GPU1 52.03→50.34µs. Batch2/context105301 is unchanged policy and showed ~0.11–0.16µs host regression; batch8/context105301 was approximately flat. All raw data retained.
The initial implementation changed the floor inside the full formula and cost~0.1–0.2µs extra host time despite GPU gains. It was replaced by the equivalent early-return formula, then revalidated. That failed version is retained, not reported as a win.
Scope / limitations
Whole-model controlled-C1 ABBA is complete; it did not establish a throughput gain (full results below). No production quality or original900s serving-load gain is claimed. Other architectures retain their original policy but were CPU-policy-tested, not GPU-benchmarked here. The available selector is Triton; alternative AITER selection and distributed indexer-CP transport were not validated. Graph timing has visible A/A noise, especially for tiny unchanged-grid cases.
Artifacts on f01:
~/bench/m3_decode_chunk_policy_20261031/(release validation, JUnit, both graph raw JSONs, wide-head samples, old/final eager results, exact policy-equivalence check); exploratory stage/chunk sweeps in~/bench/m3_decode_score_stages_20261031/.PR Self-Review
Reviewed revision
38439469009d9b5236caeaff85398fd212150e7dBehavior and intent
The gfx1250 small-batch index-score grid gets more parallel CTAs within explicitly bounded total-block ranges. The previous uniform3-block floor was slower there in controlled full-operator measurements. Only launch chunking and its host formula change, not numeric operations or selector semantics.
Validation and unverified scope
python3 -m pytest -q tests/test_minimax_m3_index_topk.py:226 passed/1 optional-selector skip on each of two gfx1250 GPUs;130 passed/97 skipped GPU-hidden. Black/Ruff/diff checks pass. Python3.12/Torch2.11/Triton3.9 in the ROCm image above. Unverified: whole-model performance/accuracy, other-GPU runtime performance, optional AITER selector, distributed indexer-CP transport. Explicit negative cases and host regressions are listed above.PR-specific question
The new gfx1250 exact-policy boundary assertions fail before this PR where the old3-block floor is selected; numeric/graph checks are preservation tests expected to pass on old valid inputs. Original gfx950 floor tests remain intact via an explicit architecture fixture. Existing arithmetic tests alone cannot detect the unnecessary launch partitioning because both chunk layouts produce the same valid scores.
Changed tests
tests/test_minimax_m3_index_topk.py: preserves original-architecture floor coverage, adds explicit architecture/batch/total-block boundaries and no-empty-grid invariants, and exercises changed-input decode score/selection/page-table outputs with independent sampled reference and graph replay. It reuses existing module constants and established test structure without changing tolerances of existing tests.Complete controlled-C1 model ABBA
Verified four-arm report.
Frozen M3 TP4/Simple/FP8-KV/EAGLE3 stack, identical AITER/cache sources; only this ATOM chunk policy differs. One outstanding request and max_num_seqs=1, retained102784-token prefix + fresh256-token suffix +768 output tokens. Six warmup and12 measured decode requests plus10 short/10 long prefills per process.
All four ranks had exactly one256-token prefill and only4-token decode forward shapes per measured C1 request. Source/script hashes and actual container import-source hashes verified; all32 measured input/output hashes match; no compiler/cache-load events recorded in measured windows.
Descriptive mean-of-process-medians: C1 throughput +0.20%, short prefill latency +0.51%, long-prefix prefill latency +6.01%. All outcomes are retained. These sequential four-process data do not establish a significant whole-model speedup. Prefill policy and math are unchanged by this patch; that fact alone is not proof of what caused the measured prefill slowdown. Process/system drift remains possible, and no favorable subsets are substituted for the original results.
The evidence for the optimization is the controlled full index-score/selection/sparse-table operator measurements. Synthetic acceptance is performance-only, not generated-text accuracy.
CP scope correction after review
Revision
85a814b684bd1932a1dd32735bc74ed5d212612dfixes an unintended scope expansion in the first PR revision. The reviewer correctly noted thatindexer_context_scoresalso imports_decode_score_chunks; the original change would have retuned its distinct CP scorer without CP performance evidence.The shared
_decode_score_chunksis now restored exactly to pre-PR code. A separate_native_decode_score_chunksis called only byminimax_m3_index_topk_decode. The CP source file is unchanged. Native decode chunk counts match the previous measured policy in 122,910 cases.Validation of this revision:
python3 -m pytest -q tests/test_minimax_m3_index_topk.py tests/model_ops/test_indexer_cp_parity.py: 263 passed, 1 optional-selector skip on each of two gfx1250 GPUs; CPU-hidden 139 passed, 125 skipped.The earlier native graph/model results are not reruns of this scope-only revision. Native grid values are proven identical, with final GPU numerical/eager checks above; no CP performance claim is made, and CP now has no policy change to benchmark. This addresses the reviewer’s concern rather than extrapolating native-scorer timing to CP.
Comment clarification in
384394690: the shared helper describes the legacy policy used by CP, with an explicit pointer to the separately measured native gfx1250 exception. Executable AST unchanged; CPU regression 139 passed / 125 GPU skips, Black/Ruff passed. No new performance or GPU-test rerun claimed for this documentation-only edit.