Skip to content

[FlyDSL] M3 index score: NVFP4 decode scorer (score_flydsl_nvfp4) - #7813

Closed
ganyi1996ppo wants to merge 11 commits into
mainfrom
ganyi/nvfp4-index-decode-pr
Closed

ganyi1996ppo wants to merge 11 commits into
mainfrom
ganyi/nvfp4-index-decode-pr

Conversation

@ganyi1996ppo

@ganyi1996ppo ganyi1996ppo commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds the decode block scorer for the MiniMax-M3 lightning indexer over an NVFP4 index cache, next to the existing MXFP4 one (score_flydsl_fp4): same work-map machinery, same score layout, new page format. Exports minimax_m3_index_score_nvfp4_flydsl and minimax_m3_index_score_nvfp4_flydsl_supported. Only gfx950 (MI355X). A follow-up PR adds the prefill scorer and builds on this one.

Why NVFP4 at all, when it is slower than MXFP4. NVFP4 is MiniMax-M3.1's native fp4 KV/index format, and the benchmark rules do not accept a non-native fp4 cache, so the MXFP4 scorer cannot serve the model as it is. The cost is measured in the Performance section below: the NVFP4 arms take 12-15% more time than score_flydsl_fp4 (geomean e4m3/mx 1.125, exact/mx 1.150) and 72 vs 68 B per token.

Page format

One NVFP4 index page is 9216 B: 128 tokens x 64 B of e2m1 codes (two 4-bit codes per byte, low nibble = even element) = 8192 B, followed by 128 tokens x 8 B of e4m3 scales = 1024 B. One scale per 16 elements, no per-tensor scale. Q is e4m3 (bf16 is accepted). The scorer reads the two regions as separate views (k_data [pages, 128, 64] uint8 and k_scale [pages, 128, 8] uint8, each with its own page stride), so a single [pages, 9216] allocation works. Scale bytes must be positive e4m3 and not 0x7F; contents are a caller contract, shapes, dtypes and strides are validated.

Numerics: k_round

  • k_round="e4m3" (default): K = e4m3_rn_satfinite(e2m1 x scale), the rounding vLLM's NVFP4 indexer applies. With an e4m3 Q it runs on the fp8 MFMA: each CTA builds a table of the 8 e4m3 values of every scale byte in LDS and widens a K dword with v_perm_b32. Scores are bit-identical to vLLM's NVFP4 decode indexer (60 cases, 208,999 live slots, 100% bit-equal on one shared page buffer, live-slot sets identical; table below).
  • k_round="exact": K = e2m1 x scale, not rounded. The product is exact in f16 and so is an e4m3 Q, so with an e4m3 Q the K dwords are converted straight to f16 pairs and scored on the f16 MFMA; a bf16 Q uses the bf16 MFMA. This is not bit-identical to vLLM. Measured top-k block overlap with vLLM's scores (random-normal K and Q quantized to NVFP4, so a synthetic distribution; rows with at least k live blocks; the e4m3 arm is 1.0 everywhere):
case scorer arm k top-k overlap with vLLM rows counted
decode S4H1 B2 ctx<=16384 exact 8 0.95312 8
decode S4H1 B2 ctx<=16384 exact 16 0.96094 8
decode S4H1 B2 ctx<=16384 exact 32 0.97656 8
decode S4H1 B2 ctx<=16384 e4m3 8 1.00000 8
decode S4H1 B2 ctx<=16384 e4m3 16 1.00000 8
decode S4H1 B2 ctx<=16384 e4m3 32 1.00000 8
decode S8H4 B4 ctx<=40000 exact 8 0.94727 128
decode S8H4 B4 ctx<=40000 exact 16 0.95850 128
decode S8H4 B4 ctx<=40000 exact 32 0.97217 128
decode S8H4 B4 ctx<=40000 e4m3 8 1.00000 128
decode S8H4 B4 ctx<=40000 e4m3 16 1.00000 128
decode S8H4 B4 ctx<=40000 e4m3 32 1.00000 128
decode S4H4 B2 ctx<=65536 exact 8 0.92969 32
decode S4H4 B2 ctx<=65536 exact 16 0.95898 32
decode S4H4 B2 ctx<=65536 exact 32 0.96191 32
decode S4H4 B2 ctx<=65536 e4m3 8 1.00000 32
decode S4H4 B2 ctx<=65536 e4m3 16 1.00000 32
decode S4H4 B2 ctx<=65536 e4m3 32 1.00000 32

What changes

  • aiter/ops/flydsl/kernels/minimax_m3_index_score.py (+614/-65): the NVFP4 arms, the e4m3 table and widen helpers (module level, so the prefill scorer reuses them), validation, score_flydsl_nvfp4, index_score_supported_nvfp4. Page id in an SGPR (no waterfall loop in the K loads), causal mask skipped on tiles whose tokens are all visible, f16 scale pair from one u24 multiply, and an NVFP4 depth table for S8H4 batches 3..16 selected with geometry(..., nvfp4=True) / build_work_map(..., nvfp4=True) so MXFP4 work maps do not move. The table gives, on S8H4 (same process, e4m3 arm / exact arm, depth table vs shared map): batch 8 ctx 64K 0.83 / 0.83, batch 16 ctx 32K 0.79 / 0.79, batch 8 ctx 128K 0.95 / 0.92, batch 2..4 unchanged (1.00 +- 0.01).
  • aiter/ops/flydsl/__init__.py (+10): the two lazy exports.
  • op_tests/test_flydsl_minimax_m3_index_score.py (+528/-29): see Tests.
    No csrc, no tuned CSV: the aiter-config-shape rules do not apply (the diff has no files under aiter/configs).

Performance (decode, vs vLLM)

One MI355X, same process, interleaved, 7 repetitions, two independent runs (r1/r2 columns), 32 shapes: S = query tokens per request, H = index heads, B = batch, ctx = context. mx is aiter's MXFP4 scorer, vllm is the MiniMax-M3 NVFP4 decode indexer of our vLLM tree (models/minimax_m3/amd/ops/nvfp4_index_decode, unmodified). TB/s is effective K-page bandwidth (live pages x 128 x 72 B per token for NVFP4, x 68 B for MXFP4, over the time); K is re-read from the same buffer every iteration, so the small shapes are warm-cache numbers. Ratios are by mean time (below 1 = faster than vLLM). The nvfp4 columns do not include the work-map build, as for MXFP4.

exact/vLLM time ratio: geomean 0.823, worst 0.940, best 0.648, slower than vLLM in 0 of 32 shapes
e4m3/vLLM time ratio: geomean 0.805, worst 0.905, best 0.619, slower than vLLM in 0 of 32 shapes
shapes where exact is slower than vLLM (mean of the two runs):
none
shapes where e4m3 is slower than vLLM (mean of the two runs):
none

Shapes where an NVFP4 arm is slower than vLLM: none in this table. Against aiter's own MXFP4 scorer the NVFP4 arms are slower, as expected for 72 B vs 68 B per token and the extra decode work: geomean e4m3/mx 1.125, exact/mx 1.150 by time.

Raw table: 32 shapes, two runs
S H B ctx mx us (r1/r2) exact us (r1/r2) e4m3 us (r1/r2) vllm us (r1/r2) mx TB/s exact TB/s e4m3 TB/s vllm TB/s exact/vLLM e4m3/vLLM
4 1 1 4096 3.67/3.63 4.04/4.0 4.07/4.02 4.94/4.93 0.08 0.07 0.07 0.06 0.815 0.820
4 1 1 16384 3.71/3.69 4.1/4.06 4.1/4.08 4.89/4.84 0.30 0.29 0.29 0.24 0.839 0.841
4 1 1 65536 3.77/3.75 4.18/4.14 4.22/4.16 5.09/5.04 1.19 1.13 1.13 0.93 0.821 0.827
4 1 1 131072 4.06/4.03 4.65/4.61 4.7/4.64 5.59/5.51 2.20 2.04 2.02 1.70 0.834 0.841
4 1 2 4096 3.61/3.59 4.0/3.94 4.02/3.98 4.96/4.94 0.15 0.15 0.15 0.12 0.802 0.808
4 1 2 16384 3.74/3.71 4.12/4.09 4.15/4.11 5.03/4.98 0.60 0.57 0.57 0.47 0.820 0.825
4 1 2 65536 4.04/3.97 4.66/4.61 4.71/4.66 5.59/5.58 2.22 2.04 2.01 1.69 0.830 0.839
4 1 2 131072 4.66/4.64 5.76/5.74 5.55/5.51 7.31/7.36 3.83 3.28 3.41 2.57 0.784 0.754
4 1 8 4096 3.76/3.74 4.16/4.11 4.16/4.14 4.9/4.86 0.59 0.57 0.57 0.48 0.847 0.850
4 1 8 16384 4.04/3.98 4.66/4.59 4.68/4.62 5.42/5.43 2.22 2.04 2.03 1.74 0.853 0.857
4 1 8 65536 7.06/7.05 8.03/7.95 7.89/7.84 10.87/10.84 5.05 4.73 4.80 3.48 0.736 0.725
4 1 8 131072 12.02/11.93 13.01/12.96 12.49/12.45 16.19/16.21 5.95 5.82 6.05 4.66 0.802 0.770
4 1 32 4096 3.94/3.86 4.65/4.58 4.68/4.62 5.5/5.46 2.28 2.04 2.03 1.72 0.842 0.849
4 1 32 16384 6.95/6.95 8.25/8.21 8.07/8.0 11.15/11.18 5.13 4.59 4.70 3.38 0.737 0.720
4 1 32 65536 21.63/21.75 23.35/23.15 23.17/23.53 27.9/27.76 6.57 6.50 6.47 5.43 0.835 0.839
4 1 32 131072 42.51/42.1 45.95/46.38 46.44/46.7 51.44/51.52 6.74 6.54 6.48 5.87 0.897 0.905
8 4 1 4096 3.74/3.71 4.42/4.4 4.21/4.17 5.14/5.1 0.07 0.07 0.07 0.06 0.861 0.818
8 4 1 16384 3.89/3.85 4.58/4.56 4.35/4.32 5.14/5.12 0.29 0.26 0.27 0.23 0.891 0.845
8 4 1 65536 3.97/3.98 4.74/4.73 4.49/4.48 5.3/5.3 1.12 1.00 1.05 0.89 0.893 0.846
8 4 1 131072 4.22/4.28 5.03/4.98 5.02/4.99 5.82/5.79 2.10 1.89 1.89 1.63 0.862 0.862
8 4 2 4096 3.8/3.78 4.5/4.47 4.27/4.24 5.13/5.09 0.15 0.13 0.14 0.12 0.878 0.833
8 4 2 16384 3.91/3.89 4.62/4.57 4.39/4.35 5.21/5.19 0.57 0.51 0.54 0.45 0.884 0.840
8 4 2 65536 4.27/4.29 4.97/4.94 4.99/4.97 5.83/5.79 2.08 1.90 1.89 1.62 0.853 0.857
8 4 2 131072 5.54/5.52 6.72/6.66 6.13/6.09 7.66/7.64 3.22 2.82 3.09 2.47 0.875 0.799
8 4 8 4096 3.94/3.93 4.65/4.62 4.43/4.4 5.12/5.1 0.57 0.51 0.53 0.46 0.907 0.864
8 4 8 16384 4.27/4.36 4.98/4.92 5.0/4.89 5.74/5.69 2.06 1.91 1.91 1.65 0.866 0.865
8 4 8 65536 9.36/9.22 11.61/11.62 10.75/10.73 12.37/12.35 3.84 3.25 3.52 3.05 0.940 0.869
8 4 8 131072 13.29/13.35 16.25/16.19 14.52/14.51 23.47/23.43 5.35 4.66 5.20 3.22 0.692 0.619
8 4 32 4096 4.37/4.31 5.0/5.01 5.06/5.03 5.7/5.69 2.05 1.89 1.87 1.66 0.879 0.886
8 4 32 16384 8.13/8.11 9.3/9.31 8.69/8.67 12.32/12.14 4.39 4.06 4.35 3.09 0.761 0.710
8 4 32 65536 23.19/23.24 26.39/25.93 25.93/25.9 40.33/40.41 6.14 5.77 5.83 3.74 0.648 0.642
8 4 32 131072 48.06/48.13 53.08/52.74 52.28/52.28 81.64/81.74 5.93 5.71 5.78 3.70 0.648 0.640

Tests

python3 op_tests/test_flydsl_minimax_m3_index_score.py (about 20 to 40 s on one MI355X; exits non-zero on any failed check): the existing MXFP4 checks, then the NVFP4 ones, then the sweep tables. The NVFP4 part follows the aiter op-test shape: no pytest in the added code (check_nvfp4_* functions run by the file's own runner, as for MXFP4), @benchmark with run_perftest, checkAllclose, TFLOPS and TB/s per candidate, a markdown table, and get_gfx_runtime() gating in main(). The torch oracle dequantizes the cache itself and a torch writer produces the pages, because this change has no NVFP4 write kernel. Matrix: oracle match on page and tile boundaries, ragged and empty requests; bit-exact on exact-sum data; tail pages; scale range and saturation; the depth table; e4m3 Q equals bf16 Q on the exact arm; split vs single-region views; repeat launches bitwise identical; the default k_round is "e4m3"; compiled kernels are not shared between modes; rejections; both k_round values and both Q dtypes; and a sweep through every non-CP launch geometry the selector can reach (10 geometry points x bf16/e4m3 Q x e4m3/exact K, 40 checks, about 21 s on its own, run by check_fp4_every_geometry_matches_oracle, the same function the MXFP4 builder calls). A wider regression matrix (183 NVFP4 decode checks, hundreds of shapes, bit comparison against vLLM) was run outside the PR on this tree; it is not part of the diff.

Checked on this tree with flydsl 0.5.0 (the version requirements.txt pins), gfx950:

  • 183 NVFP4 decode checks of the wide matrix pass; the 100 existing MXFP4 checks pass; the 76 sha256 hashes of MXFP4 decode and prefill outputs are identical to origin/main.
  • NVFP4 decode outputs (both arms, both Q dtypes, 128 cases) are byte-identical to the development branches this PR was cut from.
  • e4m3 arm vs vLLM: 60 cases, 208,999 live slots, 100% bit-equal.

Dependencies and scope

flydsl==0.5.0 as pinned, gfx950 only. Not in this PR: the NVFP4 index write kernel (the tests write pages with torch), the prefill scorer (next PR), ATOM integration.

PR Self-Review

Reviewed revision

499b0e0 (head after the second 11-15 review round; the full sha is the PR head)

Behavior and intent

Adds score_flydsl_nvfp4 and index_score_supported_nvfp4: decode block scoring over an NVFP4 index cache (9216 B pages: e2m1 codes plus one e4m3 scale per 16 elements). Before this PR aiter could only score the MXFP4 cache, so the MiniMax-M3 NVFP4 index cache had no aiter scorer. The default k_round="e4m3" arm is bit-identical to vLLM's NVFP4 decode indexer; k_round="exact" keeps K unrounded and is not bit-identical. The predicate returns False for any k_round the scorer rejects, including None.

Validation and unverified scope

Run: python3 op_tests/test_flydsl_minimax_m3_index_score.py on one MI355X (gfx950, flydsl 0.5.0): exit 0 on the head above. Re-checked after the review fixes: 76 sha256 hashes of MXFP4 decode and prefill outputs identical to origin/main, and the NVFP4 decode output hashes (both arms, both Q dtypes) identical to the pre-fix tree. Run before the review fixes, on the same kernel code: 183 NVFP4 decode checks and 60 cases / 208,999 live slots 100% bit-equal to vLLM for the e4m3 arm (not re-run, the fixes touch no kernel code). Unverified: gfx942 and other flydsl versions (gfx950 only by design), real selector/model data (synthetic random-normal K and Q), the "exact" arm's agreement with vLLM beyond the top-k overlap table. The hgemm CI failure (test_flydsl_hgemm.py) is outside this diff.

Re-run at the revision above after the review fixes (3 commits on 80ff1e7: docstrings and comments, test comments, and one refactor folding fp8_dwords_to_f16 into fp8_dwords_widen(raw, lo, dwords, dtype)):

  • python3 op_tests/test_flydsl_minimax_m3_index_score.py --ctx-len 8192 on one MI355X (gfx950, flydsl 0.5.0): exit 0, 153 checks PASS, 0 FAIL, before and after the refactor (at 6e6f677).
  • After the 11-15 review round (6a9e5a8: comment and docstring fixes, f16_arm named flag, docs of nvfp4=True on the map builders; 9a5d21b: the NVFP4 every-geometry sweep, expect_value_error on the int8 and wide-stride scale cases, single_region_views shared): the default run on one MI355X passes 193 checks, 0 FAIL, including the new sweep of 40 checks (10 non-CP geometry points x bf16/e4m3 Q x e4m3/exact K); the sweep alone takes about 21 s. The kernel change in that round is only the f16_arm flag (equal to the expression it replaces at that branch) and comments; the ISA and the performance tables were not re-measured after it.
  • Second 11-15 review round (c4c66f3: the dead bf16 branches of the f16 arm replaced by fx.Float16, the LDS/barrier comments, score_flydsl's public docstring; 499b0e0: one format-neutral check_fp4_every_geometry_matches_oracle for both builders instead of an NVFP4 wrapper): the final ISA of 56 NVFP4 e4m3-Q kernels (both k_round, 28 of them the f16 arm), compiled without a GPU before and after, is byte-identical by sha256; the default run of the op test on one MI355X passes 193 checks, 0 FAIL, before and after (100 MXFP4 and 93 NVFP4 checks, same names). The performance tables were not re-measured.
  • The refactor does not change code generation: the final ISA (FLYDSL_DEBUG_DUMP_ASM) of all 65 kernels the test builds, including the NVFP4 e4m3-Q arms for both k_round values, is byte-identical before and after (diff -rq).
  • black and ruff clean on both changed files (ruff 0.17.3 locally; CI pins 0.16.0, which I did not run).
  • Not re-run: the 183-check wide matrix and the vLLM bit comparison; the fixes touch no kernel numerics (ISA identical), but those runs were not repeated.

PR-specific question

Before this PR every NVFP4 check in the changed test fails at the first call (score_flydsl_nvfp4 does not exist). Beyond that, the checks pin things a plain oracle match would not: bit-exact scores on exact-sum data, tail pages, scale range and saturation, empty and ragged requests, e4m3 Q equals bf16 Q on the exact arm, repeat launches bitwise identical, the k_round default and the predicate for k_round=None, and rejection of bad shapes/strides/dtypes. The tuned work-map check fails if the MXFP4 exact-grid check in _validate_map is removed. No canonical test covers the NVFP4 page format; the existing MXFP4 test only rejects such a cache.

Changed op tests

op_tests/test_flydsl_minimax_m3_index_score.py (+528/-29): the canonical index-score test, extended in place (no new file) with the NVFP4 oracle, correctness matrix and benchmark; the MXFP4 checks check what they checked before, apart from the arch gate (now get_gfx_runtime()); single_region_views and the every-geometry check (check_fp4_every_geometry_matches_oracle, called by both the MXFP4 and the NVFP4 builder) are now shared with the NVFP4 side. It is not redundant with any other test: no other test builds NVFP4 pages or calls the NVFP4 scorer.

🤖 Generated with Claude Code

ganyi1996ppo and others added 2 commits November 11, 2026 01:08
Decode block scorer over an NVFP4 index cache: the same e2m1 data view as the
MXFP4 path, but one e4m3 scale byte per 16 elements (8 per token, 9216 B per
128-token page) and no per-tensor scale; Q is e4m3 or bf16.

Two numerics, selected by k_round:
- "e4m3" (default): K = e4m3_rn_satfinite(e2m1 * scale) through a per-CTA LDS
  table of the 8 e4m3 values of every scale byte, scored on the fp8 MFMA with
  an e4m3 Q; bit-identical to vLLM's NVFP4 indexer.
- "exact": K = e2m1 * e4m3 scale, unrounded. The product is exact in f16 and
  so is an e4m3 Q, so with an e4m3 Q the K dwords are converted straight to f16
  pairs and scored on the f16 MFMA; a bf16 Q uses the bf16 MFMA. Not
  bit-identical to vLLM, which rounds K to e4m3.

Page ids are read into an SGPR so the K buffer descriptors need no waterfall
loop; the causal mask is skipped on tiles whose tokens are all visible; the f16
scale pair is built with one u24 multiply; and a depth table for S8H4 batches
3..16 is selected through geometry(..., nvfp4=True) so MXFP4 work maps do not
move. The e4m3 table and widen helpers are module-level so the prefill scorer
can reuse them.

Co-Authored-By: Claude <noreply@anthropic.com>
Extends the existing decode index-score op test with the NVFP4 scorer. No pytest
in the added code: `check_nvfp4_*` functions run through the file's own runner,
like the MXFP4 checks next to them.
- A torch oracle (e2m1 x e4m3 scale, unrounded and rounded to e4m3) and a torch
  NVFP4 page writer, since this change has no NVFP4 write kernel.
- A minimal matrix, both k_round values and both Q dtypes: oracle match on page
  and tile boundaries, ragged and empty requests; bit-exact on exact-sum data;
  tail pages; scale range and saturation; the NVFP4 depth table; e4m3 Q equals
  bf16 Q on the exact arm; split vs single-region views; repeat launches
  bitwise identical; the default k_round is "e4m3"; modes do not share a
  compiled kernel; rejections.
- An @benchmark table (us, TFLOPS, TB/s, err per k_round) in the decode sweep.
The wider regression matrices stay out of the test file.

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter standard tests on MI35X
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:mi300x Run Aiter standard tests on MI300X in PRs; main branch always runs them
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
ci:triton-355 Run the full Triton test suite on MI35X, not only the tests the change affects
ci:windows Windows JIT op smoke test on the Radeon R9700 ROCm runner
ci:multi-gpu Aiter multi-GPU tests on the 8-GPU runner
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 7813 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

Reviews:
A reviewer is usually assigned within one working day after the PR is ready for review (see Code Review Process). If nobody has responded for about a week, mention the area contact listed in CONTACTS.md. Want to help review? See REVIEWERS.md.

@aiter-bot

aiter-bot commented Nov 11, 2026 •

Copy link
Copy Markdown

WAITING ON AUTHOR

PR Self-Review

Revision 499b0e0f453f · mode shadow

Waiting for the PR author. The readiness report will appear after the self-review answers are complete for this revision.

Questions to answer in the PR description

  1. What behavior does this PR change, and why is the existing behavior insufficient?
  2. What did you validate, and what remains unverified? Include commands, hardware/configuration and results.
  3. What behavior would each changed op test fail on before this PR, and why is that coverage not already provided by an existing canonical test?
  4. For every changed op_tests/ file, what behavior does it cover and why is it not redundant? Write None if no op test changed.

Edit the PR Self-Review section in the PR description. Include the current revision SHA; this comment refreshes automatically.

Missing or invalid answers

  • FAIL — Reviewed revision must contain at least the first 7 characters of the current head SHA

Shadow mode reports what would block but keeps the status successful. Set AITER_PR_SELF_REVIEW_MODE=enforce only after reviewing real false positives.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The NVFP4 predicate accepts inputs with k_round=None that its scorer rejects, risking invalid dispatch, and the work-map rejection test never reaches the intended validation.

2 open issues
What changed in this PR

Adds NVFP4 decode scoring for the MiniMax-M3 lightning indexer on gfx950, alongside the existing MXFP4 scorer, reusing its work-map machinery and score layout.

Changes:

  • Implements default e4m3-rounded and exact NVFP4 scoring modes with format validation and tuned launch geometry.
  • Adds lazy exports for the scorer and support predicate.
  • Extends correctness checks and performance sweeps, and switches test architecture checks to get_gfx_runtime().
File Description
op_tests/​test_flydsl_minimax_m3_index_score.py Adds NVFP4 reference generation, correctness checks, and benchmarks.
aiter/​ops/​flydsl/​kernels/​minimax_m3_index_score.py Implements NVFP4 decoding, scoring, validation, and launch selection.
aiter/​ops/​flydsl/​__init__.py Exports the NVFP4 scorer and support predicate.

Review effort: Balanced

…ork-map test non-vacuous

index_score_supported_nvfp4 passed k_round=None straight to the shared validator,
which reads None as an MXFP4 cache, so the predicate said True for arguments that
score_flydsl_nvfp4 rejects. Return False unless k_round is one of NVFP4_ROUNDINGS.

The tuned-work-map refusal check fed NVFP4 scale views to the MXFP4 scorer, which
refused them on shape before looking at the work map. Use real MXFP4 views, check the
shared map is accepted and the tuned one is refused with a message naming work_map.

Host-side and test-only; kernel outputs are unchanged.

Co-Authored-By: Claude <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 12, 2026 01:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The benchmark’s shared/MXFP4 work map causes both candidates to bypass the NVFP4 depth-table optimization, so their timings do not represent the default scorer’s launch.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Use the NVFP4 work map for benchmark candidates

op_tests/​test_flydsl_minimax_m3_index_score.py:2834

The benchmark builds a shared/MXFP4 work map, so _validate_metadata retains the shared geometry when it differs from the NVFP4-tuned default, such as S=8, H=4, B=8 at 64K context. Both benchmark candidates therefore bypass the new depth-table optimization and report timings for a different launch than the default scorer. Build the map with nvfp4=True; if shared-map timings are also needed, report them as separate candidates.

2 resolved since last review

Review effort: Balanced

bench_score_nvfp4 passed a shared (MXFP4) work map, which score_flydsl_nvfp4
keeps as-is, so the benchmark never ran the NVFP4 depth-table launch that the
default scorer picks (S8H4, batch 3..16). Build the map with nvfp4=True.

Co-Authored-By: Claude <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 13, 2026 06:29
@ganyi1996ppo

Copy link
Copy Markdown
Contributor Author

Re the "previously missed issue" in Copilot's review of cf80410 (bench_score_nvfp4 builds a shared work map; no inline thread, so answering here).

Valid, fixed in 80ff1e7: the benchmark now calls build_work_map(seq, mb, S, H, nvfp4=True), so both candidates launch the NVFP4 depth-table geometry that the default scorer picks. _validate_metadata keeps a shared map's geometry when one is passed (kernel.py:2072-2082), so the old benchmark timed a different grid for S8H4.

Measured on one MI355X (median of 5 x 10 iters, B8 S8 H4, uniform ctx, shared map vs nvfp4=True map):

  • ctx 64K: e4m3 11.10 us (344 rows) vs 9.15 us (512 rows); exact 11.79 vs 9.63 us
  • ctx 128K: e4m3 15.30 us (688 rows) vs 14.44 us (1024 rows); exact 16.57 vs 15.41 us

I did not add the shared-map timing as a separate candidate (the depth table only changes S8H4 batches 3..16; MXFP4 and other shapes are identical either way). Say if you want it.

python3 op_tests/test_flydsl_minimax_m3_index_score.py --ctx-len (decode checks and benchmarks only) exits 0 with the change; black and ruff 0.16.0 clean. Not run here: the prefill perf sweep (needs the CK submodule to JIT-build module_msa_sparse_attention, absent in my tree; CI has it).

The two earlier threads (k_round predicate, work-map rejection test) were already fixed in cf80410 and are unchanged by this commit.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new gfx950 MFMA paths and LDS lookup decoding require human review of hardware correctness and performance evidence, as no head-matched runtime report is supplied and GPU validation is unavailable in this environment.

No open issues 🎉

Review effort: Balanced

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (5)

  1. State in the description why NVFP4 is needed despite the numbers. The only consumer is draft ATOM#3310; its GSM8K shows nvfp4 at or below MXFP4 (TP2 strict-match 0.8916 vs 0.9030), and this PR's own table has the NVFP4 arms 12-15% slower than score_flydsl_fp4 with 72 vs 68 B/token. The reason is that NVFP4 is MiniMax-M3.1's native fp4 KV/index format and the benchmark rules do not accept a non-native fp4 cache; say that, so the five-PR series reads as required rather than as a slower duplicate of the MXFP4 path.
  2. Fix the NVFP4 module docstring, it describes only one of the three arms. minimax_m3_index_score.py:50-53 says K is widened at cvt(scale=1.0) and Q "is widened to bf16"; the e4m3-Q arms run the fp8 MFMA (k_round="e4m3") or the f16 MFMA ("exact") with 2^120 / 2^8 undo scales. Same in the build_index_score docstring ("q_fp8 (MXFP4 only)"). Also :56 "scale bytes must be positive": zero is accepted and tested, say "non-negative, not 0x7F".
  3. Correct the depth-table comment. minimax_m3_index_score.py:448 says the shared table gives "depth 3 for batches 3..16"; _SERVED_DECODE_DEPTH[(8, 4)] (:441) gives depth 2 for batches 3-4 and depth 3 only for 5..16. In the test, exact_sum_case says scales are in [0.5, 2) for bytes 0x30..0x47 (that range is [0.5, 4)), and check_nvfp4_bit_exact says it "records how often they differ" (it only asserts).
  4. Fold fp8_dwords_to_f16 into fp8_dwords_to_bf16. minimax_m3_index_score.py:155-187 are the same loop with a different intrinsic and element type; the bf16 helper's own docstring says two copies are a silent-wrong-numbers risk. Take the output dtype as a parameter.
  5. Drop references aiter readers cannot follow, and refresh the self-review SHA. "like the vLLM nvfp4 indexer", "vLLM's _quant4", "the P0 microtest", "regression matrix ... lives outside this file" (kernel docstrings and test :2449-2533, :2860): keep the numeric definition (RNE e4m3, saturate at 448, scale rule) and name the test that pins it. aiter-bot's PR Self-Review still fails because "Reviewed revision" is cf80410, not 80ff1e7.

Copilot's ask (gfx950 MFMA hardware correctness, perf numbers): MI35X CI shard 6 ran the NVFP4 matrix (oracle, bit-exact, tail, saturation, depth table, rejections: all PASS), and the description has a 32-shape before/after table vs vLLM and MXFP4, so that ask is answered. Copilot's earlier k_round=None predicate and vacuous work-map test findings are fixed in cf80410; the NVFP4 work map benchmark finding is fixed in 80ff1e7.


Reviewed at 80ff1e7. Stacked follow-up: #7814 (prefill).

ganyi1996ppo and others added 3 commits November 15, 2026 03:57
Module and build_index_score docstrings described one arm; spell out the
bf16-Q, e4m3-Q/"e4m3" (fp8 MFMA) and e4m3-Q/"exact" (f16 MFMA) routes, say
scale bytes are non-negative and not 0x7F, fix the depth-table comment
(depth 2 for batches 3-4, depth 3 for 5..16) and drop references to vLLM
internals and an unnamed microtest. Comments only.

Co-Authored-By: Claude <noreply@anthropic.com>
…rences

exact_sum_case scale range is [0.5, 3.75] (bytes 0x30..0x47) and its sum
bound 2^14; check_nvfp4_bit_exact only asserts; replace references to vLLM
internals and an out-of-tree regression matrix with the rule itself and the
tests that pin it. Comments and docstrings only.

Co-Authored-By: Claude <noreply@anthropic.com>
fp8_dwords_to_bf16 and fp8_dwords_to_f16 were the same loop with a
different intrinsic and element type. One helper, fp8_dwords_widen, takes
the output dtype; the e4m3-Q NVFP4 loader drops its duplicated branch.
Generated ISA of all 65 kernels the op test builds is byte-identical.

Co-Authored-By: Claude <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 15, 2026 10:09
@ganyi1996ppo

Copy link
Copy Markdown
Contributor Author

Thanks @valarLip. All five items are addressed at head 6e6f677 (3 commits on top of 80ff1e7); the description is updated too.

  1. Why NVFP4: added to the description (Summary): NVFP4 is MiniMax-M3.1's native fp4 KV/index format and the benchmark rules do not accept a non-native fp4 cache; the 12-15% / 72 vs 68 B per token cost is stated next to it.
  2. Docstrings: 6e46918. The module docstring now lists the three arms (bf16 Q with the cvt/narrow path and the bf16 MFMA; e4m3 Q + "e4m3" with the LDS table, v_perm_b32 and one fp8 scaled MFMA; e4m3 Q + "exact" with the 2^8 undo scale and the f16 MFMA). build_index_score no longer says "(MXFP4 only)". Scale bytes are described as non-negative (zero allowed), not 0x7F.
  3. Depth-table comment and test comments: 6e46918 (kernel) and aece88e (test). The comment now says the shared S8H4 table gives depth 2 for batches 3-4 and 3 for 5..16. exact_sum_case says scales are in [0.5, 3.75] (bytes 0x30..0x47) and I corrected the sum bound next to it (2^12 -> 2^14: 128 x 6 x 3.75 x 3.75 = 10800). check_nvfp4_bit_exact now says it asserts that the data separates the two references.
  4. Fold the f16 helper: 6e6f677. One fp8_dwords_widen(raw, lo, dwords, dtype) for bf16 and f16; the e4m3-Q loader's duplicated branch is gone. I renamed it because a dtype parameter on a name ending in _bf16 reads wrong; say if you want the old name back. The final ISA of all 65 kernels the op test builds is byte-identical before and after (diff -rq), and the op test is 153 PASS / 0 FAIL both times on one MI355X.
  5. References readers cannot follow: 6e46918 and aece88e. The vLLM / _quant4 / "P0 microtest" / "regression matrix lives outside this file" references are replaced by the rule (RNE e4m3, saturate at 448, clamp to +-448 before the convert because it does not saturate; writer rule scale = RNE e4m3 of min(amax x RN(1/6), 448) per 16 elements, then RNE e2m1) and the tests that pin it (check_nvfp4_matches_oracle, check_nvfp4_scale_range_and_saturation). The self-review "Reviewed revision" is now 6e6f677.

Not re-run: the 183-check wide matrix and the vLLM bit comparison (the fixes touch no kernel numerics, ISA identical); CI's pinned ruff 0.16.0 (I used 0.17.3 locally).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The gfx950 MFMA and NVFP4 changes require independent GPU validation of correctness, rounding, and performance that this source-only review environment cannot provide.

No open issues 🎉

Review effort: Balanced

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (3)

  1. Run the NVFP4 scorer through the reachable launch geometries, as the MXFP4 suite does. build_nvfp4_checks (op_tests/test_flydsl_minimax_m3_index_score.py:2862) only uses the F*/L*/causal_straddle/ragged_zero cases (B <= 4) plus one S8H4 B8 tuned-map case. So for NVFP4 the token split at tw2, ppw4 and the non-served ppw2/ppw4 nt2 arms never run. Those are the arms that page through the new SGPR page id, the causal-skip and the e4m3 LDS table. Add check_nvfp4_every_geometry_matches_oracle over the non-CP GEOM_POINTS x QDTS x K_ROUNDS, reusing check_mxfp4_every_geometry_matches_oracle (:2165) with nvfp4_cache_of. While there, also call expect_value_error(... run_nvfp4 ...) on the int8 and wide-stride scale cases (:2818-2825), which only check the predicate today.
  2. Fix the comments the 11-15 rewrite got wrong.
    • minimax_m3_index_score.py:311-313: check_nvfp4_matches_oracle is an ATOL/RTOL checkAllclose (check_scores, test :1248). The bit-equality check is check_nvfp4_bit_exact.
    • :353-354: e2m1 x e4m3 needs six significant bits, not five (1.5 x 1.875 = 101101b).
    • :2445-2447: re-wrap the broken "and not / 0x7F." docstring line.
    • :579-581: the geometry docstring says an nvfp4=True map is "only valid for score_flydsl_nvfp4". That is true only for S8H4 batches 5..16; everywhere else the two maps are identical.
    • build_work_map / work_map_size (:1765, :1818) and score_flydsl_nvfp4 do not say that nvfp4=True (passed through **cp) is how a caller gets the tuned depth table.
  3. Small cleanups.
    • :954: elif const_expr(nvfp4 and q_fp8) is exactly f16_arm there; use the named flag like every other branch.
    • nvfp4_single_region_views (test :2534) is single_region_views (:2108) with 4 -> 8; give it a scale-width parameter instead of a second copy.

CI: all 30 checks pass (27 skipped). The premise is accepted: NVFP4 is M3.1's native fp4 index format and is now stated in the description.


Re-reviewed at 6e6f677. Already fixed: the "why NVFP4" paragraph in the description; the three-arm module and build_index_score docstrings and the "non-negative, not 0x7F" wording (6e46918); the S8H4 depth-table comment, exact_sum_case range [0.5, 3.75] with the 2^14 bound, and the check_nvfp4_bit_exact docstring; fp8_dwords_to_f16 folded into fp8_dwords_widen (6e6f677); vLLM / _quant4 / P0 / out-of-tree references removed; self-review SHA refreshed. Copilot's k_round=None predicate, vacuous work-map test and NVFP4 benchmark-map findings stay fixed.

ganyi1996ppo and others added 2 commits November 15, 2026 11:54
…vfp4=True on the map builders

Comments only plus one flag rename: f16_arm is exactly `nvfp4 and q_fp8` at that branch.

Co-Authored-By: Claude <noreply@anthropic.com>
…ch geometry

Reuses the MXFP4 every-geometry check (token split, ppw 2/4, non-served nt2 arms), checks
ValueError on the int8 and wide-stride scale views, and folds nvfp4_single_region_views
into single_region_views (scale width from the view).

Co-Authored-By: Claude <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 15, 2026 18:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new gfx950 numerical paths require domain-reviewer scrutiny because author-reported results lack independent, exact-head GPU validation of correctness and base/head latency.

No open issues 🎉

Review effort: Balanced

ganyi1996ppo added a commit that referenced this pull request Nov 15, 2026
Co-Authored-By: Claude <noreply@anthropic.com>
@ganyi1996ppo

Copy link
Copy Markdown
Contributor Author

@valarLip, re-review of 11-15 17:47, point by point. Pushed head is now 9a5d21b (fast-forward of 6e6f677; fixes in 6a9e5a8 and 9a5d21b). The description is updated to match.

  1. NVFP4 scorer only ran B <= 4 cases: fixed in 9a5d21b. check_nvfp4_every_geometry_matches_oracle runs the 10 non-CP GEOM_POINTS x bf16/e4m3 Q x e4m3/exact K (40 checks) through check_mxfp4_every_geometry_matches_oracle, which now takes an optional k_round and then uses nvfp4_cache_of + run_nvfp4. The scorer has no CP shard, so the CP points are left out. The geometries reached for NVFP4 include tw2, tw4, ppw4 and the non-served ppw2/ppw4 nt2 arms. One note: the S8H4 b16 point resolves to ppw 2 for NVFP4 (the tuned table); ppw 3 is only reached with an explicit shared work map, which the existing S8H4 B8 tuned-map check runs. The int8 and wide-stride scale cases in check_nvfp4_rejections now also check that run_nvfp4 raises ValueError. On one MI355X all 40 pass (21 s with first compile) and the full default run passes 193 checks, 0 FAIL. I did not mutation-test the sweep: no kernel code changed in this round.
  2. Comments: all five were right; fixed in 6a9e5a8. check_nvfp4_matches_oracle -> check_nvfp4_bit_exact. "five significant bits" -> six (1.5 x 1.875 = 10.1101b; I also checked every e2m1 x e4m3 product, the maximum is six). The "and not / 0x7F" line is re-wrapped. The geometry docstring now says the two maps differ only at S8H4 batches 5..16 and are identical elsewhere. build_work_map, work_map_size and score_flydsl_nvfp4 now say that nvfp4=True (through **cp) gets the tuned depth table, and that work_map=None resolves it too.
  3. Cleanups: elif const_expr(nvfp4 and q_fp8) is f16_arm at that point, so it now uses the flag (6a9e5a8). nvfp4_single_region_views is gone; single_region_views reads the scale width from the scale view, so no extra parameter is needed (9a5d21b). That differs slightly from the parameter you suggested; say so if you prefer the explicit one.

Not re-measured after this round: ISA and the performance tables (the only kernel change is the f16_arm flag, equal to the expression it replaces, plus comments).

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (3)

  1. Drop the dead bf16 branches left in the f16 arm. Naming f16_arm at minimax_m3_index_score.py:956 left the same redundancy one level down. load_q8_lifted is only called under elif const_expr(f16_arm) (:957), but :932 still says wdt = fx.Float16 if f16_arm else fx.BFloat16, and :960, inside that same branch, builds the MFMA with fx.Float16 if f16_arm else fx.BFloat16. Use fx.Float16 in both places. Also fix the load_q8_lifted docstring (:921): "the four bf16 B-fragments" should say f16.
  2. Give the shared every-geometry check a format-neutral name, not a wrapper. check_nvfp4_every_geometry_matches_oracle (test :2569) only forwards to check_mxfp4_every_geometry_matches_oracle (:2166), and that MXFP4-named function now holds the NVFP4 branch too. Rename it (for example check_fp4_every_geometry_matches_oracle(point, qdt, k_round=None)), call it from both build_mxfp4_checks and build_nvfp4_checks, and delete the one-line wrapper.
  3. Update the PR description to match head. "Changed op tests" says "the MXFP4 checks are unchanged apart from the arch gate". That is no longer true: single_region_views and the every-geometry check are now shared with NVFP4. The Tests paragraph still says "about 20 s" and its matrix list leaves out the 40-check NVFP4 geometry sweep (you measured about 21 s for the sweep alone).

CI: the decode op test passes at head (shard 6, including all 40 new every_geometry_* NVFP4 checks). The only failure is test_flydsl_hgemm.py ("graph replay mismatch"), which does not import this module, so it is unrelated to this PR.


Re-reviewed at 9a5d21b. Already fixed: the NVFP4 sweep over the 10 non-CP GEOM_POINTS x Q dtype x k_round (S8H4 b16 now runs the tuned ppw2 geometry); expect_value_error on run_nvfp4 for the int8 and wide-stride scales (the "k_scale" and "packed" substrings match the real messages); the check_nvfp4_bit_exact, "six significant bits", 0x7F re-wrap, geometry nvfp4 and nvfp4=True-through-**cp docstrings; f16_arm at :956; nvfp4_single_region_views folded into single_region_views. Earlier rounds' items and Copilot's findings stay fixed.

ganyi1996ppo and others added 2 commits November 15, 2026 18:42
…LDS/barrier comments, restore score_flydsl docstring

load_q8_lifted is only called under f16_arm, so its dtype and the MFMA dtype are plain
Float16. The "no LDS, no barrier" comments now name the fp8 arm's scale table and its
barrier. Final ISA of 56 NVFP4 e4m3-Q kernels is identical before and after.

Co-Authored-By: Claude <noreply@anthropic.com>
…r both builders

Co-Authored-By: Claude <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 00:49
@ganyi1996ppo

Copy link
Copy Markdown
Contributor Author

@valarLip, re-review of 11-15 21:16, point by point. Pushed head is now 499b0e0 (fast-forward of 9a5d21b): c4c66f3 (kernel and comments) and 499b0e0 (test). The description is updated to match.

  1. Dead bf16 branches in the f16 arm: fixed in c4c66f3. load_q8_lifted is only called under elif const_expr(f16_arm), so the fragment dtype and the MFMA dtype are now plain fx.Float16, and its docstring says "four f16 B-fragments". The generated code is unchanged: I compiled 56 NVFP4 e4m3-Q kernels (both k_round, S1H1/S4H4/S8H4/S16H4, token split, ppw 1..4, spread; 28 of them the f16 arm) with no GPU before and after, and the final ISA is byte-identical for all 56 by sha256.
  2. Format-neutral name for the every-geometry check: fixed in 499b0e0. The shared function is check_fp4_every_geometry_matches_oracle(point, qdt, k_round=None), called from both build_mxfp4_checks and build_nvfp4_checks; the one-line wrapper is deleted. The check list is unchanged: 100 MXFP4 and 93 NVFP4 checks (40 of them the NVFP4 sweep), same names before and after.
  3. Description: updated. "Changed op tests" now says the MXFP4 checks check what they did before and that single_region_views and the every-geometry check are shared; the Tests paragraph says about 20 to 40 s and lists the 40-check NVFP4 geometry sweep (about 21 s on its own); the per-file line counts are recomputed (+614/-65 for the kernel file, +528/-29 for the test).

Two more things I fixed in c4c66f3, from our own review of this head: the comments that said "no LDS, no barrier" now name the fp8 arm's 1 KiB scale table and its barrier, and score_flydsl's public docstring is back to describing its own arguments instead of pointing at the private _score_impl.

On one MI355X the default run of the op test passes 193 checks, 0 FAIL, before and after this round. Not re-measured: the performance tables (the kernel change is the fx.Float16 constants, which do not change the ISA, and comments), and CI's ruff 0.16.0.

@valarLip @valar-cop re-review at 499b0e0.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new gfx950 NVFP4 and MFMA paths require native correctness and performance validation, but no head-matched report was supplied and this environment cannot run the GPU tests.

No open issues 🎉

Review effort: Balanced

@valar-cop valar-cop left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

ci green.

ganyi1996ppo added a commit that referenced this pull request Nov 16, 2026
Co-Authored-By: Claude <noreply@anthropic.com>

@valarLip valarLip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Action required before merge (3)

  1. Compute the f16-arm scale pair once per byte, as the bf16 arm does. minimax_m3_index_score.py:1180-1184 calls e4m3_scale_raw_f16x2(w16) twice and e4m3_scale_raw_f16x2(w16 >> 8) twice; the else branch right below (:1187-1189) already does s_lo/s_hi then [s_lo, s_lo, s_hi, s_hi]. Use the same shape in both arms.
  2. Split the _validate_metadata NVFP4 guard so its message names the real problem. :2043 raises "nvfp4: k_round must be one of ..." also when k_round is valid and k_scale is None. Raise a separate "nvfp4 needs k_scale" error for that case.
  3. Say "non-negative" everywhere the scale contract is stated. The module docstring (:54) and score_flydsl_nvfp4 say non-negative (zero allowed, and zero is tested), but the PR description ("Scale bytes must be positive e4m3"), e4m3_to_f32's docstring (:301, "scales are positive") and the test comment at :2688 ("0x00..0x7E: scales are positive") still say positive.

Optional, not blocking: nvfp4 is a bool in geometry/_decode_depth_table and the K-rounding string in build_index_score/_get/_validate_metadata; a distinct name for the geometry flag would read better. The test's K_ROUNDS (:2474) duplicates kernel.NVFP4_ROUNDINGS.

CI: 31 pass, 27 skipped, no failures.


Re-reviewed at 499b0e0. Already fixed: the dead fx.BFloat16 branches in the f16 arm (load_q8_lifted and its MFMA are now plain fx.Float16, docstring says f16); one format-neutral check_fp4_every_geometry_matches_oracle(point, qdt, k_round=None) called from both builders, wrapper deleted; the description's "Changed op tests" and Tests paragraphs (shared helpers, 20-40 s, the 40-check sweep, line counts +614/-65 and +528/-29 match the diff). Earlier rounds' items and Copilot's findings stay fixed.

@ganyi1996ppo

Copy link
Copy Markdown
Contributor Author

Closing as landed: this PR's code is already on main. #7814 was stacked on this branch (its head contains 499b0e0f45), so its squash merge dad1b66282 brought in minimax_m3_index_score.py and test_flydsl_minimax_m3_index_score.py exactly as they are at this head; they are byte-identical to main now.

@valarLip, thanks for the reviews. Three items from your last round (06:20, at 499b0e0f45) did not make it into main, and I am not following them up in this PR: (1) hoisting the f16-arm scale pair to once per byte, (2) a separate "nvfp4 needs k_scale" error in _validate_metadata, (3) "non-negative" wording in the e4m3_to_f32 docstring, the LUT_ENTRIES comment and the test comment. I checked (1) on CPU: the generated ISA is identical before and after (112 kernels, same hashes), so it is a readability change only. I have a tested patch for all three (2 files, +21/-16, cherry-picks cleanly onto current main); if you want it, I will send it as a small follow-up PR against main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants