Repository navigation
[MiniMax-M3.1] NVFP4 KV cache and index cache (--kv-cache-dtype nvfp4, --index-cache-dtype nvfp4) - #3310
[MiniMax-M3.1] NVFP4 KV cache and index cache (--kv-cache-dtype nvfp4, --index-cache-dtype nvfp4)#3310ganyi1996ppo wants to merge 6 commits into
Conversation
…idation, pool layout NVFP4 (e2m1 + one e4m3 scale per 16) K/V pages and index pages for MiniMax-M3.1, next to the MXFP4 ones: the two flags, their validation (fail closed, same refusals as the MXFP4 caches, plus: MXFP4 and NVFP4 never pair, nvfp4 is M3 only), the KV pool layout (one 2 * Hkv * 9216 B field a page) and the 9216 B index cache spec. FP4_CACHE_DTYPES names the two FP4 dtypes where the MXFP4 code tested "fp4". Co-Authored-By: Claude <noreply@anthropic.com>
…FP4 selector and sparse PA rope_cache takes the NVFP4 mode of aiter's fused_qknorm_idxrqknorm (kv_cache_dtype and index_cache_dtype "nvfp4"): norm / RoPE and the quantization of K, V and the index key into the pages in one pass, the layer's fp32 [1] global K / V scales riding along. The selector returns the logical block ids (no page-16 table is emitted) and the NVFP4 sparse-PA decode / prefill kernels read them with the request block table. The index scorer record says which FP4 index scorer to run (`IndexScoreBuffers.nvfp4`). Needs aiter ROCm/aiter#7813, #7814, #7824, #7825, #7826; their symbols are imported where the NVFP4 path is first reached, so nothing changes for the other caches. Co-Authored-By: Claude <noreply@anthropic.com>
…cles on gfx950 Pages are written by aiter's fused NVFP4 writer from random qkv rows; the oracles read the pages back through a token-major reader built from the documented page offsets, so the test pins the layout the writer and the readers share. Skips without gfx950 or an aiter that has the NVFP4 kernels. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
Conflicts (#3104 indexer CP on the MXFP4 index cache): keep main's CP shard fields in IndexScoreBuffers next to the NVFP4 flag, and keep the indexer-CP gate for NVFP4 only (indexer_cp_unsupported_reason(nvfp4_caches=...)), since aiter's NVFP4 decode scorer takes no cp_world/cp_rank. Co-Authored-By: Claude <noreply@anthropic.com>
…vfp4_{decode,prefill}
Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The fused-writer test skips every failure, potentially hiding writer/read-path regressions in all five end-to-end cases; only the known unsupported-NVFP4-mode error should trigger a skip.
1 open issue
What changed in this PR
Adds NVFP4 KV and index caches to MiniMax-M3.1’s native ATOM inference path, extending the existing MXFP4 support with a second FP4 format backed by AITER kernels.
Changes:
- Adds
nvfp4cache options, matching-dtype defaults, and startup validation for unsupported configurations. - Implements packed page layouts and connects cache writes, index scoring, and sparse attention.
- Adds cache-layout, validation, and read-path tests, plus configuration and dependency documentation.
| File | Description |
|---|---|
tests/test_minimax_m3_nvfp4_read.py |
Checks selection and attention against dequantized-page references. |
tests/test_minimax_m3_nvfp4_cache.py |
Tests page geometry, pool allocation, and configuration restrictions. |
tests/test_minimax_m3_nvfp4_attention.py |
Tests prefill row mapping and scorer dispatch. |
tests/test_minimax_m3_mxfp4_cache.py |
Allows the shared config helper to select architectures. |
docs/configuration_guide.md |
Documents NVFP4 flags, restrictions, and required AITER kernels. |
atom/models/minimax_m3/config.py |
Extends FP4 validation to NVFP4 and rejects incompatible cache pairings. |
atom/model_ops/minimax_m3/sparse_attn_nvfp4.py |
Adds sparse-attention wrappers and prefill row mapping. |
atom/model_ops/minimax_m3/nvfp4_pages.py |
Defines NVFP4 page geometry, views, and KV pool layout. |
atom/model_ops/minimax_m3/index_topk.py |
Dispatches index scoring to the matching FP4 kernels. |
atom/model_ops/minimax_m3/index_buffers.py |
Allocates and identifies NVFP4 scorer buffers. |
atom/model_ops/attentions/aiter_attention.py |
Allocates index pages and per-layer global K/V scales. |
atom/model_ops/attention_mha.py |
Connects NVFP4 writes, logical block selection, and attention. |
atom/model_engine/arg_utils.py |
Exposes NVFP4 cache choices in the CLI. |
atom/config.py |
Resolves NVFP4 defaults, draft cache types, and model/parallelism restrictions. |
Review effort: Balanced
| except Exception as e: # noqa: BLE001 - an aiter without the NVFP4 write mode | ||
| pytest.skip(f"aiter's fused writer has no NVFP4 mode: {e}") |
valar-cop
left a comment
There was a problem hiding this comment.
a few small things..
gpu re-run of the read test and one server start with --kv-cache-dtype nvfp4 against aiter main?
ci green.
| if spec is not None and spec.draft_kv_dtype == "fp4": | ||
| # "fp4" is left only on a draft in the target's pool (MTP), which has no | ||
| # MXFP4 attention impl to write its rows; an own-pool draft is bf16 and passes. | ||
| if spec is not None and spec.draft_kv_dtype in FP4_CACHE_DTYPES: |
There was a problem hiding this comment.
name and messages still say MXFP4, but now gates nvfp4 too. please update _fp4_kv_refusal, _check_fp4_kv, _declare_kv_pools and index_scorer docstrings.
| weight(), weight(), 1, slots, kv, kv, idx, 128, q_out, iq_out, slots, | ||
| kv_cache_dtype="nvfp4", index_cache_dtype="nvfp4", k_scale=ks, v_scale=vs, | ||
| ) # fmt: skip | ||
| except Exception as e: # noqa: BLE001 - an aiter without the NVFP4 write mode |
There was a problem hiding this comment.
catching every exception as skip hides real failures. gate on the nvfp4 write mode up front, or catch only the unsupported-mode error?
|
|
||
| from atom.model_ops.attentions.pool_layout.entry_arena import EntryField | ||
|
|
||
| PAGE_TOKENS = 128 |
There was a problem hiding this comment.
PAGE_TOKENS duplicates geometry.SPARSE_BLOCK_SIZE, import it instead of hard coding 128?
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (6)
- Re-run the GPU evidence on aiter main and this head, and add perf numbers. Every GPU result in the description (read test, server runs, GSM8K, memory) was taken on the pre-merge aiter heads + ATOM 5aebd79; aiter main has since changed the API under you (ops renamed,
sparse_pa_nvfp4_decodenow uses a lazily allocatedDecodeWorkspacethat is refused inside graph capture). Posttests/test_minimax_m3_nvfp4_read.py+ one TP4 server start (CUDA-graph capture) + GSM8K on aiter main 0b0381a1a8 or later, plus decode TPOT / output throughput vsfp8and MXFP4 (fp4) at the same TP/concurrency. Also state the reason in the description: M3.1's native fp4 KV/index format is NVFP4 and external benchmark rules reject non-native fp4 caches. - Stop turning writer failures into skips.
tests/test_minimax_m3_nvfp4_read.py:84-92catches everyExceptionaroundfused_qknorm_idxrqknormand skips, so a wrong layout or a launch error skips all 5 cases. All the NVFP4 symbols are on aiter main now, so drop thattryand the_missing_kernelsgetattr probe; keep only the gfx950 skip (Copilot + valar-cop raised this too; not addressed). - Declare state instead of stashing attributes.
attention_mha.py:1771-1775stashesprefill_md.nvfp4_row_infothroughgetattr(..., None). Make it a field ofMiniMaxM3SparsePrefillMetadatabuilt once inmake_sparse_prefill_metadata.aiter_attention.py:647-656(build_kv_cache_tensor) setsimpl.nvfp4_k_scale/v_scalefrom the builder, but the impl's__init__never declares them; declare them there, next toself.kv_layout = Nvfp4KvLayout(). - Fix the names, messages and docs that still say MXFP4 (valar-cop asked, not addressed): rename
_mxfp4_kv_refusal(models/minimax_m3/config.py:10);_check_fp4_kvdocstring (config.py:1987); the user-visible"MXFP4 index decode: ..."error and comments that now fire for NVFP4 (aiter_attention.py:1125-1133, 467); theindex_scorerdocstrings in sparse_attn.py:103,121,172; the module docstring of index_topk.py:11-18. docs/configuration_guide.md: theindex_cache_dtyperow still says "anything else is refused" on M3, and the minimum-aiter list names #7813 (still open) and #7825 (closed). Cite what landed instead (#7814 dad1b66282, #7824 47d23011bd, #7826 8f574bdf51). - Delete duplication.
nvfp4_pages.PAGE_TOKENS = 128duplicatesgeometry.SPARSE_BLOCK_SIZE(valar-cop);sparse_pa_nvfp4_decode/prefillin sparse_attn_nvfp4.py are 1:1 pass-throughs of the aiter ops, so call aiter directly and keep onlyprefill_row_positions; the(selected, None, None) if nvfp4 else selectedunpack is duplicated in_sparse_prefill/_sparse_decodeand should be one helper (or always return the 3-tuple). - Refresh the PR description. "Draft until the aiter PRs land" and the unchecked aiter box are stale; also list what is untested (DSpark/EAGLE own-pool draft with nvfp4, prefix-cache hits).
Reviewed at 2421500. aiter deps checked: all 4 FlyDSL symbols + the fused writer's nvfp4 mode are on aiter main 0b0381a1a8 with matching signatures. Light CI green; heavy CI not run yet. Full findings list kept with the reviewer.
|
ci triage for 2421500:
|

Motivation
#2962 added the MXFP4 KV and index caches for MiniMax-M3.1. This adds the NVFP4 pair (e2m1 data, one e4m3 scale per 16 elements, one fp32 global scale per layer for K and V) as a second FP4 option:
--kv-cache-dtype nvfp4 --index-cache-dtype nvfp4. The kernels live in aiter (five open PRs, below); this PR is the ATOM side: flags and validation, pool and index-cache layout, the write call, and the selector / sparse-PA wiring. Draft until the aiter PRs land.Technical Details
Hard dependencies (aiter; all five are now in aiter main, see the status below; the page format is the contract between the writer #7824 and the readers):
minimax_m3_index_score_nvfp4_flydsl(decode index scorer)minimax_m3_index_score_prefill_nvfp4_flydsl(prefill index scorer; stacked on #7813)fused_qknorm_idxrqknorm(kv_cache_dtype="nvfp4"+index_cache_dtype="nvfp4"): norm / RoPE and quantization of K, V and index-K into the pages in one passsparse_pa_nvfp4_decodesparse_pa_nvfp4_prefill(stacked on #7825)Status 2026-11-16: all five are in aiter main. #7824 landed as 47d23011bd. #7826 landed as 8f574bdf51 and, being stacked on #7825, carries the decode kernel too (#7825 was closed as landed). #7814 landed as dad1b66282 and carries #7813's decode scorer (#7813 itself is still open for review nits; its code is on main). Checked with
git grepon main f858b48e68 for the symbols this PR imports (score_flydsl_nvfp4,score_prefill_flydsl_nvfp4,sparse_pa_nvfp4_decode,sparse_pa_nvfp4_prefill); the Python signatures this PR calls are unchanged by the merged versions [read]. The merged code went through further review rounds after the heads listed under Test Result (for example the decode split scratch is now aDecodeWorkspace, allocated on first use and refused during graph capture); the GPU results below were not re-run on aiter main [not measured]. They are imported where the NVFP4 path is first reached, so the other caches never load them.ATOM changes (14 files, +1001 / -72 against #2962; commits: config and validation, writer + selector + sparse PA wiring, read-path tests, docs):
atom/config.py,arg_utils.py: the two flags takenvfp4;FP4_CACHE_DTYPES = ("fp4", "nvfp4")replaces the"fp4"tests of [MiniMax-M3.1] MXFP4 index cache and MXFP4 KV cache (--index-cache-dtype fp4, --kv-cache-dtype fp4) #2962 (draft dtype, scorer allocation, CP gate). The index dtype follows the KV dtype when omitted.atom/models/minimax_m3/config.py(startup validation, not the model definition): every [MiniMax-M3.1] MXFP4 index cache and MXFP4 KV cache (--index-cache-dtype fp4, --kv-cache-dtype fp4) #2962 refusal applies to nvfp4 through the same code (vLLM/SGLang bridges, DCP, PCP, sequence parallelism,--block-size != 128, non-gfx950, non-bf16 model, KV offload /--kv-transfer-config, a draft left with FP4 KV, layers that are not all indexed sparse). New: MXFP4 and NVFP4 never pair, an NVFP4 index needs an NVFP4 KV, nvfp4 is refused off MiniMax-M3.nvfp4_pages.py(new): page geometry,Nvfp4KvLayout(one2 * Hkv * 9216B field a page),index_page_views.aiter_attention.py: the 9216 B index-cache spec and the layer's fp32[1]K / V scales (1.0: the checkpoint carries none).attention_mha.py:rope_cachecalls the fused op with the nvfp4 dtypes (same call as MXFP4); the selector emits logical block ids (no page-16 table) and the NVFP4 sparse PA reads them.index_topk.py/index_buffers.py:IndexScoreBuffers.nvfp4picks the NVFP4 page views and scorers.sparse_attn_nvfp4.py(new): thin wrappers and the prefill row positions.mono/dispatch.py), so nvfp4 runs the regular decode path. The indexer-CP flag is cleared with the existing warning under any FP4 cache (the gate argument now covers nvfp4).On an image whose aiter predates those PRs (image aiter 627461276, which has none of the symbols): [measured] with
--kv-cache-dtype nvfp4the server passes config validation and memory sizing, then aborts at CUDA-graph capture with aiter'sfused_qknorm_idxrqknorm.cu:1069 sparse insert mode expects index_cache_dtype='auto', fp8 or mxfp4(TP2, one run). All other caches are untouched; the new tests skip.Known limitations (read from the code, not measured unless stated):
Test Plan
tests/run against main with aiter / triton / flydsl made unimportable (the non-GPU CI job's condition).Test Result
All GPU runs used a local aiter stack: image aiter 627461276 plus the net diff of the five PR heads (#7813 80ff1e7ee4, #7814 44fee0620b, #7824 2f0bda9b0e, #7825 af14f1c96d, #7826 f3d1915843), not aiter main. Model
amd/MiniMax-M3.1-Flash-Quark-MXFP4with the recipe's online ptpc_fp8 config,--block-size 128,--max-model-len 135000, mono off, no speculative decoding, prefix caching at its default. MI355X (gfx950).Lint / CPU:
tests/(aiter / triton / flydsl blocked, no GPU,--continue-on-collection-errors): main 9671 passed / 26 failed / 32 collection errors, this branch 9695 passed / 25 failed / 32 collection errors. The 32 errors and 24 of the failures are the same set on both. The remaining ones: on main twotest_benchmark_catalogfailures (git worktree not visible in the container), on the branchtest_atomesh_spur_dispatch::test_dispatch_waits_until_controller_reports_running, which also fails on main when run alone (a 15 s script timeout). The container is not the CI runner (missing optional packages), so these are comparisons against main, not the CI result.GPU:
op_tests/test_fused_qknorm_nvfp4.py(writer pages vs a torch reference, byte for byte): exit 0, every case 0 mismatching bytes.test_flydsl_minimax_m3_sparse_pa_nvfp4.py(decode + prefill, correctness cases only),test_flydsl_minimax_m3_index_score.pyandtest_flydsl_minimax_m3_index_score_prefill_fp4.py(-b 1 --q-len 512 --ctx-len 8192): exit 0, no failing row. These are the aiter PRs' own tests on the local stack, one run each.tests/test_minimax_m3_nvfp4_read.py(3 decode cases, S 1 and 4, Hkv 1 and 2, global scales 1.0 / 0.37 / 1.9; 2 prefill cases): 5 passed. The selected block sets equal the oracle's exactly (the oracle rounds K to e4m3, the scorers' defaultk_round); attention is checked against an fp64 reference on the selected blocks with a relative-L2 bound of 2.5e-3.nvfp4_write.pyof the source branch): same qkv and slots, whole kv and index pages compared, 90 cases (tokens 1/7/64/300/2048 x Hkv 1/2/4 x global scale 1.0/0.37/3.0 x scattered/sequential slots), 0 mismatching bytes. The Triton writer is not part of this PR.--gpu-memory-utilization 0.85: 107,301 vs 59,172 blocks at TP4.--kv-cache-dtype nvfp4 --index-cache-dtype nvfp4finish and give coherent text. Against fp8 caches, one prompt (chat0) is identical and the other six diverge after 45 to 193 characters of the reasoning text; this is not an equality claim.flexible-extract is within 0.3 points across all rows; strict-match is 1.0 to 2.2 points lower for nvfp4 than for fp8 (about 1 to 2.6 standard errors, single runs, run-to-run noise not measured). The cause of the strict-match gap (answer formatting vs content) was not investigated.
Not measured: throughput / latency of any kind; accuracy beyond GSM8K (no long-context benchmark); TP8 or other TPs; speculative decoding with nvfp4; prefix-cache behaviour with nvfp4 pages specifically (prefix caching was on, no hit-rate check); aiter main (the stack above is the image aiter plus the five PRs); the CI GPU image; repeated runs.
Submission Checklist
docs/configuration_guide.md: flags, conditions, minimum aiter PRs)Update 2026-11-15 (merge of main and aiter's rename)
Merged main (3298107) into the branch (a merge commit, no rebase) and followed aiter's rename of the sparse-PA ops (aiter #7825/#7826 review):
minimax_m3_sparse_pa_nvfp4_{decode,prefill}->sparse_pa_nvfp4_{decode,prefill}.IndexScoreBufferskeeps main'scp_world/cp_rankand this PR'snvfp4flag. The indexer-CP gate keeps refusing NVFP4 (the NVFP4 decode scorer takes no CP shard) but no longer MXFP4, soindexer_cp_unsupported_reasontakesnvfp4_cachesin place of the removedmxfp4_caches; a new test checks that an NVFP4 cache clears the flag with the warning.sparse_attn_nvfp4.pyimports the aiter ops under the new names; the_KERNELSlist oftests/test_minimax_m3_nvfp4_read.pyand the docstring ofnvfp4_pages.pyfollow.tests/test_minimax_m3_*.py,test_minimax_m31_shared_fusion.py,test_indexer_cp_gate.py,test_indexer_cp_gather_ragged.py): branch 401 passed / 4 failed / 108 skipped, main 373 passed / 4 failed / 103 skipped; the same 4 tests fail on both (test_minimax_m31_shared_fusion.py,'FusedMoE' object has no attribute 'quant_method', container environment); the +28 passed are this PR's NVFP4 tests.🤖 Generated with Claude Code