Repository navigation
Conversation
The Triton prefill scorer allocated its fp32 block scores [index heads, rows, blocks] on every forward. In a chunked long prefill the width grows by 64 blocks a chunk, so the caching allocator kept a new, slightly larger segment each chunk. The memory profile never saw them (its prefill is short). On a PP4 stage (TP1, 4 index heads, 8K chunks) one 702K-token prompt left 21-27 GiB of cached segments, and under AgentX traffic at --gpu-memory-utilization 0.9 an RCCL kernel then failed to get scratch memory and the server aborted (HSA_STATUS_ERROR_OUT_OF_RESOURCES). The builder now allocates one LogitsWorkspace for the bf16 / FP8 index caches, sized like the MXFP4 record's (index heads x a step's rows x the model length's blocks), before the memory profile, and the prefill metadata hands it to the Triton scorer. The MXFP4 path is unchanged and shares the sizing (prefill_score_capacity). The vLLM and SGLang bridges pass no workspace and keep allocating per step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
Path scope:
|
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues were identified for the prefill score workspace changes.
No open issues 🎉
What changed in this PR
This PR moves MiniMax-M3’s native bf16/FP8 prefill index scores into a fixed workspace, avoiding growing per-chunk allocations during long prefills. It extends the workspace approach already used by MXFP4 without changing scoring kernels or selection logic.
Changes:
- Allocate a configuration-sized workspace during metadata-builder initialization so memory profiling accounts for it.
- Pass the workspace through sparse-prefill metadata to the Triton scorer, preserving allocation fallback for bridges.
- Add capacity checks and GPU regression tests for workspace reuse with both selectors.
| File | Description |
|---|---|
tests/test_minimax_m3_mxfp4_decode_batch.py |
Updates the prefill fixture with the optional workspace field. |
tests/test_minimax_m3_index_topk.py |
Tests reused, poisoned workspaces against fresh score buffers. |
tests/test_minimax_m3_index_buffers.py |
Tests capacity bounds, verify-row sizing, and MXFP4 exclusion. |
atom/model_ops/minimax_m3/sparse_attn.py |
Carries the workspace in sparse-prefill metadata. |
atom/model_ops/minimax_m3/index_topk.py |
Lets the Triton prefill scorer write into the supplied workspace. |
atom/model_ops/minimax_m3/index_buffers.py |
Adds workspace allocation and shares prefill-capacity calculation. |
atom/model_ops/attentions/aiter_attention.py |
Allocates the workspace at initialization and attaches it to metadata. |
atom/model_ops/attention_mha.py |
Passes the prefill workspace to index scoring. |
Review effort: Balanced
| block_table: torch.Tensor | ||
| max_query_len: int | ||
| max_seq_len: int | ||
| score_workspace: LogitsWorkspace | None = None |
There was a problem hiding this comment.
bridge paths still allocate per chunk and stay exposed to the OOM abort, please track the follow-up or pass the workspace there too
valarLip
left a comment
There was a problem hiding this comment.
⚠️ Action required before merge (5)
- Key the bf16/FP8 workspace off the scorer record, not off
== "fp4".index_buffers.py:65(allocate_prefill_scores): #3310 makesallocate_index_scorersreturn a record fornvfp4too, so after it lands NVFP4 would also get this up-to-1 GiB workspace and never use it. Allocate only when"" not in self._index_scorers(the switch the builder already uses,aiter_attention.py:1118). Then the "workspace xorindex_scorer" rule is set in one place. - Coordinate with #2493 before merging. #2493 makes aiter's FlyDSL
minimax_m3_index_score_prefill_flydslthe default bf16/FP8 prefill scorer, and that call takes noout=. Once it lands, Triton runs only in tests: this workspace stays resident but unused, and the per-chunk growth comes back. Whichever PR merges second must pass the workspace into the FlyDSL call, which needs aiter to acceptout=. Please agree on the order with @ganyi1996ppo and say it in the description. - Address valar-cop's bridge comment (
sparse_attn.py:75). Open an issue for the vLLM (plugin/vllm/attention/minimax_m3_attnetion.py:691) and SGLang (plugin/sglang/minimax_m3_bridge.py:648) paths, which still allocate on every call and can still hit the OOM abort, and link it from the PR. Or pass the workspace on both paths. - Fix the docstrings. Verify steps take the decode sparse path (
aiter_attention.py:1111), so "or a verify step" (index_buffers.py:55) andtest_verify_rows_past_the_chunk_widen_itdescribe a case this prefill-only workspace never sees. Either drop themax_bs * max_qlenterm from the bf16/FP8 sizing or say it is only there to match MXFP4. "the bridges, which allocate per step" (sparse_attn.py:77) should say per layer call. - Small cleanups. Type
score_workspace/outasLogitsWorkspace | None(index_topk.py:1159,:1279). Optionally add a GPU check that a second chunk with the workspace allocates nothing (torch.cuda.memory_allocatedbefore and after), because no test fails on main for the growth itself.
Root cause is in ATOM: the Triton scorer allocated per forward, and LogitsWorkspace (sparse_indexer_chunk.py:205) exists for exactly this. The before/after memory, TTFT and accuracy numbers are convincing. Pre-checkin is green. Heavy CI has not run yet; it needs ci:atom before approval.
Reviewed at b6cd839. Full findings list kept with the reviewer.
|
ci triage for b6cd839:
|
Problem
With a bf16 or FP8 index cache, MiniMax-M3's prefill index scorer (
_triton_prefill_score) allocated its fp32 block scores,[index heads, rows, context blocks], on every forward. In a chunked long prefill the width grows with every chunk (64 blocks per 8K chunk), so the caching allocator never reuses the previous, smaller block and reserves a new segment each chunk. The memory profile never sees them: its warmup prefill is short.On a PP4 prefill (TP1 per stage, 4 index heads, 8K chunks, 1M context,
--gpu-memory-utilization 0.9), one 702K-token prompt raises reserved memory by 21-27 GiB per stage while allocated memory stays flat: 36-54 free cached segments a stage, 8 MiB apart, all from that line. Under AgentX traffic the device then runs out, an RCCL p2p kernel cannot get its scratch memory, and the stage aborts withHSA_STATUS_ERROR_OUT_OF_RESOURCES, taking the server down.--index-cache-dtype fp4is not affected: since #2962 its scorer writes into the builder'sLogitsWorkspace.Fix
allocate_prefill_scoresgives the bf16 / FP8 index caches the same kind of workspace: oneLogitsWorkspaceofprefill_score_capacityfp32 scores, index heads x max(max_num_batched_tokens,max_num_seqsx query width) rows x themax_model_lenblocks. That is the bound the MXFP4 record already uses for its prefill rows, andallocate_index_scorersnow takes it from the same function. The builder allocates it in its constructor, so the memory profile counts it.make_sparse_prefill_metadata(score_workspace=)carries it to the sparse layers, andminimax_m3_index_topk(score_workspace=)hands it to_triton_prefill_score, which writes into it instead of allocating. The kernels and the selection are unchanged. A step wider than the workspace fails inLogitsWorkspace.logits, as on MXFP4, instead of allocating.Resident cost
It scales with
max_model_len: 1 GiB per GPU at TP1 with--max-num-batched-tokens 8192 --max-model-len 1048576(4 index heads), also 1 GiB at TP4 with 32768 (1 head a rank), and 128 MiB at TP4 with a 131072 model length. KV blocks: 86,299 -> 85,827 (-0.55%) on the PP4 configuration above, 61,327 -> 61,283 (-0.07%) on the TP4 GSM8K servers below.Validation
MI355X (gfx950), image
rocm/atom-dev:nightly_202611151511with each tree mounted: base = main 246df9d, this PR on top of it. MiniMax-M3.1-Flash-Quark-MXFP4,--kv_cache_dtype fp8 --index-cache-dtype fp8, ptpc FP8,--block-size 128. PP4 =-tp 1 -pp 4 --enforce-eager --max-num-batched-tokens 8192 --max-model-len 1048576 --gpu-memory-utilization 0.9, prefix caching on, Mooncake producer, the AgentX prefill-only setup the crash was found in. Base and PR ran at the same time on GPUs 0-3 and 4-7, from fresh servers.PP4, one 702K-token prompt:
PP4, AgentX trace replay (AIPerf
inferencex-agentx-mvp, one output token, 300 s), fresh servers for each concurrency:HSA_STATUS_ERROR_OUT_OF_RESOURCESinncclDevKernel_GenericThe C16 base run survived because the first allocation to fail was PyTorch's, which frees its cache and retries. At C32 an RCCL kernel's scratch allocation failed first.
Long-context retrieval through chat, thinking off (a document of
Room r has access code c.lines, the question for 10 rooms from 1% to 99% depth), on the same PP4 servers:At 296K both miss the same two edge rooms; their wrong answers change from run to run on either server.
GSM8K chat (5-shot multi-turn, thinking on, temperature 0,
num_concurrent64), TP4 aggregated servers (--max-num-batched-tokens 32768 --max-model-len 131072, CUDA graphs, no prefix caching), 8 runs per arm, the two arms swapping GPU sets after the third:The differences are within one standard error of the means. Responses change from run to run on one server about as much as between the arms (302 of 1,319 identical between two base runs, 294-307 between base and PR runs).
Unit and GPU tests (M3 suites plus the MHA builder and transfer tests, 16 files): base 325 passed, this PR 333 passed (the 8 new tests), 32 skipped on both, no failures.
test_chunks_scored_into_one_stale_workspace_select_as_fresh_buffers(GPU): the chunks of one prefill share a workspace filled with +inf before each call. Both selectors (Triton, and aiter's from 2048 blocks), with bf16 and FP8 caches, return exactly what a fresh buffer gives. With the scorer redirected so the selection reads the stale workspace, all four cases fail.tests/test_minimax_m3_index_buffers.py(CPU, runs in CI): the workspace holds a full chunk at the model length and refuses one row more; a verify step's rows widen it; MXFP4 gets none.Not in this PR
max_num_seqsx query width rows (16 MiB at TP1, 128 sequences, 1M context).🤖 Generated with Claude Code