Skip to content

[AMD] Add experimental MiniMax-M3 FlyDSL attention - #41397

Open
ThomasNing wants to merge 1 commit into
sgl-project:mainfrom
ThomasNing:thomas/minimax-m3-flydsl-amd
Open

ThomasNing wants to merge 1 commit into
sgl-project:mainfrom
ThomasNing:thomas/minimax-m3-flydsl-amd

Conversation

@ThomasNing

@ThomasNing ThomasNing commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Experimental, opt-in, and not qualified for general use.

Add an opt-in native SGLang attention path for MiniMax-M3 using the AITER FlyDSL paged-attention kernel from AITER #4332 and GPU work planner from AITER #5546.

This calls AITER directly and does not modify its attention kernel. The PR is ready for review of the SGLang integration and its qualification gaps, using concrete gfx950 measurements.

AITER APIs and SGLang-specific integration

The AITER #4332 API provides FlyDSL paged attention. AITER #5546 provides GPU-side dynamic partition planning and graph-safe refresh. SGLang uses the planner only for dense, uneven-context graph decode; sparse calls keep static partitions.

The SGLang-specific integration connects these AITER APIs to MiniMax's cache and attention lifecycle: SHUFFLE 5D main KV writes, the existing NHD indexer/top-k path, conversion of per-head sparse selections to physical page tables, graph metadata refresh, and guards for unsupported modes. The AITER kernel is unchanged.

Modifications

  • Add SGLANG_MINIMAX_FLYDSL_DECODE (default false) and SGLANG_MINIMAX_FLYDSL_PLAN (default true; used only when the feature is enabled).
  • Write the main FP8 KV cache directly in SHUFFLE 5D while keeping the existing NHD index cache, indexer, and top-k reuse policy.
  • Adapt sparse selections into physical page tables without copying the whole KV cache. Handle per-head selections, partial final pages, and causal prefill rows.
  • Use static FlyDSL splits for sparse attention and optionally plan dense graph decode on the GPU. Persist int32 graph lengths and capture their refresh from SGLang's int64 inputs once per forward.
  • Gather/dequantize used KV for dense prefill; process sparse prefill as bounded chunks of causal query rows.
  • Reject unsupported speculation/cache-transfer/parallelism configurations when opted in.
  • Add an optional Docker build argument to skip the default AITER backports when selecting a newer AITER pin that already contains them. Default image dependencies and backport behavior are unchanged.
  • Include manual GPU validation, serving/benchmark reproduction, and machine-readable result summaries under test/manual/minimax_m3/flydsl/.

Dependencies: measured with AITER 94dca7bc674650b6de2609eeae2bcdd58e970bf3 and FlyDSL 0.3.4.1. The feature requires this newer kernel/compiler stack; applying the SGLang patch alone does not upgrade dependencies. The example Docker build has not been validated; measurements used source/dependency overlays in an existing ROCm container.

Accuracy Tests

Gfx950 GPU adapter checks: 43/48 pass. Passing coverage includes dense decode, graph replay with changed int64 lengths, sparse decode, and 12 exact comparisons against independently CPU-built page tables.

Five short sparse-prefill cases fail the original atol=0.01, rtol=0.08 tolerance against dequantized FP32 attention (page16/head2; page64/head1–2; page128/head1–2). In these cases the adapter exactly matches direct AITER with independently constructed metadata. This points to FP8 arithmetic rather than page-table construction, but does not establish acceptable model accuracy. The tolerance has not been loosened. The target TP4/page16/one-KV-head group passes its six numerical checks.

MiniMax-M3-MXFP8 GSM8K chat/thinking smoke check, first 64 questions, temperature 0, maximum 4096 output tokens:

Mode Correct Invalid
Existing SGLang 64/64 0
FlyDSL static 63/64 0
FlyDSL planned 64/64 0

This is not full accuracy qualification or proof of equivalence. A larger paired evaluation is still needed. All three modes also passed dummy-weight startup, decode graph, mixed-length request, and chunked-prefill smoke checks.

Speed Tests and Profiling

Four MI355X/gfx950 GPUs, TP4, MiniMaxAI/MiniMax-M3-MXFP8 revision c5454eb03678d8710e54a4e0fc681b9f3b4a3dba, BF16 activations, FP8 KV, page16, chunk8192, decode graphs enabled, radix cache disabled. NCCL all-reduce; custom all-reduce disabled. No EAGLE/speculative decoding.

Two repetitions per mode/workload/concurrency, 8–40 requests per point, fixed seed and forced output lengths. All 60 measurements / 1,272 timed requests succeeded, with matching per-request input/output lengths across modes. Loading, graph capture, and warmup are excluded from timed results.

Planned FlyDSL versus existing SGLang, medians of run-level metrics:

Workload Concurrency Output throughput (tok/s) Throughput change Median TPOT change
8K input / 256 output 1 74.43 → 70.99 -4.6% 6.2% slower
8K input / 256 output 2 136.01 → 129.29 -4.9% 6.6% slower
8K input / 256 output 10 312.54 → 317.80 +1.7% 0.4% lower
8K input / 256 output 15 379.14 → 388.80 +2.5% 0.9% lower
8K input / 256 output 20 430.83 → 447.45 +3.9% 2.3% lower
Mixed 1K–32K input / 8–256 output 20 128.79 → 138.72 +7.7% 7.8% lower

At concurrency20, median TTFT changes 3574.75 → 3325.88 ms for fixed8K and 2165.59 → 1973.47 ms for mixed lengths.

Caveats:

  • Small-batch serving regresses; this is not an unconditional speedup.
  • The isolated sparse kernel has 21.8% lower latency at batch20, but that is not a whole-model speedup. It excludes indexer/KV writes/other layers/communication and repeatedly uses warm tensors.
  • No separate planner benefit is established: second-repetition static/planned throughput differs by at most 0.3% at concurrency10–20. The first static sweep was slower; shared JIT caches and fixed mode order confound attribution.
  • Mixed-length concurrency2 is unstable; its aggregate +13.7% throughput is not reliable evidence of improvement.
  • The short pilot cannot establish production tail latency or statistical significance for small gains. No full-model kernel profile was collected.
  • This pilot uses MXFP8, no speculative decoding, and NCCL all-reduce. Results do not predict other weight formats, speculation modes, or communication paths.

Remaining work before merge

  • Resolve/qualify the five numerical failures and run full paired model accuracy evaluation.
  • Investigate small-batch regressions.
  • Add CI integration once the required AITER/FlyDSL dependency stack is available, and qualify gfx942 separately (only gfx950 was tested).
  • Validate the Docker build and decide how to roll out the dependency update.
  • Run longer fully warmed traces with varied mode order before drawing conclusions about the planner or small gains.

Checklist

  • Repository pre-commit checks pass on all changed files.
  • Include manual GPU validation and reproducible benchmark scripts.
  • Document launch, routing, dependencies, measured results, and limitations.
  • Provide accuracy and speed results, including failures and regressions.
  • Complete numerical/model accuracy qualification and registered CI coverage.

The runtime implementation is the GPU-tested version based on fc9e1c8d296216ff1e216dfbe7286ef392448d28; PR preparation changed formatting only (runtime ASTs checked unchanged). It merges cleanly with upstream df1fc6ab6441abbdf21944958b131c03e0921253.


CI States

Latest PR Test (Base): ❌ Run #36796705310
Latest PR Test (Extra): ❌ Run #36796705080
Latest PR Test (AMD ROCm 10): ❌ Run #36796705230

Add opt-in SHUFFLE KV and direct AITER FlyDSL attention with graph-owned dense planning. Include manual validation and gfx950 serving reproduction; retain known numerical failures and small-batch regressions as draft qualification blockers.

Remove the ATOM reference from the FlyDSL PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

amd documentation Improvements or additions to documentation memory-pool

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant