Repository navigation
[AMD] Add experimental MiniMax-M3 FlyDSL attention - #41397
Open
ThomasNing wants to merge 1 commit into
Open
ThomasNing wants to merge 1 commit into
ThomasNing wants to merge 1 commit into
Conversation
ThomasNing
marked this pull request as ready for review
September 27, 2026 06:54
ThomasNing
requested review from
Fridge003,
HaiShaw,
Qiaolin-Yu,
Ying1123,
alphabetc1,
hanming-lu,
hebiao064,
hnyls2002,
huangtingwei9988,
hzh0425,
ishandhanani,
ispobock,
merrymercy,
sogalin,
xiezhq-hermann,
yctseng0211 and
yizhang2077
as code owners
September 27, 2026 06:54
Add opt-in SHUFFLE KV and direct AITER FlyDSL attention with graph-owned dense planning. Include manual validation and gfx950 serving reproduction; retain known numerical failures and small-batch regressions as draft qualification blockers. Remove the ATOM reference from the FlyDSL PR.
ThomasNing
force-pushed
the
thomas/minimax-m3-flydsl-amd
branch
from
October 1, 2026 00:32
2b62d26 to
1ee3fb2
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Experimental, opt-in, and not qualified for general use.
Add an opt-in native SGLang attention path for MiniMax-M3 using the AITER FlyDSL paged-attention kernel from AITER #4332 and GPU work planner from AITER #5546.
This calls AITER directly and does not modify its attention kernel. The PR is ready for review of the SGLang integration and its qualification gaps, using concrete gfx950 measurements.
AITER APIs and SGLang-specific integration
The AITER #4332 API provides FlyDSL paged attention. AITER #5546 provides GPU-side dynamic partition planning and graph-safe refresh. SGLang uses the planner only for dense, uneven-context graph decode; sparse calls keep static partitions.
The SGLang-specific integration connects these AITER APIs to MiniMax's cache and attention lifecycle: SHUFFLE 5D main KV writes, the existing NHD indexer/top-k path, conversion of per-head sparse selections to physical page tables, graph metadata refresh, and guards for unsupported modes. The AITER kernel is unchanged.
Modifications
SGLANG_MINIMAX_FLYDSL_DECODE(default false) andSGLANG_MINIMAX_FLYDSL_PLAN(default true; used only when the feature is enabled).test/manual/minimax_m3/flydsl/.Dependencies: measured with AITER
94dca7bc674650b6de2609eeae2bcdd58e970bf3and FlyDSL0.3.4.1. The feature requires this newer kernel/compiler stack; applying the SGLang patch alone does not upgrade dependencies. The example Docker build has not been validated; measurements used source/dependency overlays in an existing ROCm container.Accuracy Tests
Gfx950 GPU adapter checks: 43/48 pass. Passing coverage includes dense decode, graph replay with changed int64 lengths, sparse decode, and 12 exact comparisons against independently CPU-built page tables.
Five short sparse-prefill cases fail the original
atol=0.01, rtol=0.08tolerance against dequantized FP32 attention (page16/head2; page64/head1–2; page128/head1–2). In these cases the adapter exactly matches direct AITER with independently constructed metadata. This points to FP8 arithmetic rather than page-table construction, but does not establish acceptable model accuracy. The tolerance has not been loosened. The target TP4/page16/one-KV-head group passes its six numerical checks.MiniMax-M3-MXFP8 GSM8K chat/thinking smoke check, first 64 questions, temperature 0, maximum 4096 output tokens:
This is not full accuracy qualification or proof of equivalence. A larger paired evaluation is still needed. All three modes also passed dummy-weight startup, decode graph, mixed-length request, and chunked-prefill smoke checks.
Speed Tests and Profiling
Four MI355X/gfx950 GPUs, TP4,
MiniMaxAI/MiniMax-M3-MXFP8revisionc5454eb03678d8710e54a4e0fc681b9f3b4a3dba, BF16 activations, FP8 KV, page16, chunk8192, decode graphs enabled, radix cache disabled. NCCL all-reduce; custom all-reduce disabled. No EAGLE/speculative decoding.Two repetitions per mode/workload/concurrency, 8–40 requests per point, fixed seed and forced output lengths. All 60 measurements / 1,272 timed requests succeeded, with matching per-request input/output lengths across modes. Loading, graph capture, and warmup are excluded from timed results.
Planned FlyDSL versus existing SGLang, medians of run-level metrics:
At concurrency20, median TTFT changes 3574.75 → 3325.88 ms for fixed8K and 2165.59 → 1973.47 ms for mixed lengths.
Caveats:
Remaining work before merge
Checklist
The runtime implementation is the GPU-tested version based on
fc9e1c8d296216ff1e216dfbe7286ef392448d28; PR preparation changed formatting only (runtime ASTs checked unchanged). It merges cleanly with upstreamdf1fc6ab6441abbdf21944958b131c03e0921253.CI States
Latest PR Test (Base): ❌ Run #36796705310
Latest PR Test (Extra): ❌ Run #36796705080
Latest PR Test (AMD ROCm 10): ❌ Run #36796705230