Repository navigation
[Klaud Cold] minimaxm31flash-fp4-gb300-sglang-agentic-dspark: day-zero MiniMax-M3.1-Flash GB300 SGLang recipe #4443
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
592ae7c
5af102c
d3bc0ba
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,164 @@ | ||
| base: | ||
| schema: 2 | ||
| name: minimaxm31flash-fp4-gb300-sglang-agentic | ||
| resources: | ||
| gpu_type: gb300 | ||
| gpus_per_node: 4 | ||
| frontend: | ||
| type: sglang | ||
| enable_multiple_frontends: false | ||
| observability: | ||
| enabled: false | ||
| tachometer: | ||
| enabled: false | ||
| engine: sglang | ||
| # Cold loads of the 250 GB checkpoint plus JIT MSA kernel builds and graph capture outlast the default. | ||
| health_check: | ||
| max_attempts: 1440 | ||
| interval_seconds: 5 | ||
| roles: | ||
| agg: | ||
| nodes: 1 | ||
| workers: 1 | ||
| gpus: 4 | ||
| # Serve flags follow the cookbook's GB300 cells at | ||
| # https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3.1. | ||
| args: | ||
| served-model-name: MiniMaxAI/MiniMax-M3.1-Flash | ||
| trust-remote-code: true | ||
| reasoning-parser: minimax-m3 | ||
| tool-call-parser: minimax-m3 | ||
| # Seven-token DSpark, the bundled draft's dspark_block_size. Throughput runs | ||
| # switch to synthetic acceptance at the golden acceptance length. | ||
| speculative-algorithm: DSPARK | ||
| speculative-dspark-block-size: 7 | ||
| speculative-use-block-verification: true | ||
| cuda-graph-backend-prefill: breakable | ||
| tokenizer-worker-num: 8 | ||
| watchdog-timeout: 3600 | ||
| enable-metrics: true | ||
| env: | ||
| PYTHONNOUSERSITE: '1' | ||
| PYTHONUNBUFFERED: '1' | ||
| SGLANG_TIMEOUT_KEEP_ALIVE: '1800' | ||
| FMHA_SM100_ALLOW_JIT: '1' | ||
| benchmark: | ||
| env: | ||
| # Warmup can leave bytes unacknowledged past AIPerf's 30 s default. | ||
| AIPERF_HTTP_TCP_USER_TIMEOUT: '900000' | ||
| AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:' | ||
|
|
||
| # TP4 Balanced cell (TRT-LLM routed MoE). The scheduler batch is 2x CONC (AgentX | ||
| # subagent fan-out), capped at the cookbook's 128; the decode graph batch matches it. | ||
|
|
||
| override_tp4_c1: | ||
| roles: | ||
| agg: | ||
| args: | ||
| tensor-parallel-size: 4 | ||
| moe-runner-backend: flashinfer_trtllm_routed | ||
| chunked-prefill-size: 32768 | ||
| max-running-requests: 2 | ||
| cuda-graph-max-bs-decode: 2 | ||
| mem-fraction-static: 0.8 | ||
| benchmark: | ||
| env: | ||
| CONC: '1' | ||
| KV_OFFLOADING: none | ||
|
|
||
| override_tp4_c4: | ||
| roles: | ||
| agg: | ||
| args: | ||
| tensor-parallel-size: 4 | ||
| moe-runner-backend: flashinfer_trtllm_routed | ||
| chunked-prefill-size: 32768 | ||
| max-running-requests: 8 | ||
| cuda-graph-max-bs-decode: 8 | ||
| mem-fraction-static: 0.8 | ||
| benchmark: | ||
| env: | ||
| CONC: '4' | ||
| KV_OFFLOADING: none | ||
|
|
||
| override_tp4_c8: | ||
| roles: | ||
| agg: | ||
| args: | ||
| tensor-parallel-size: 4 | ||
| moe-runner-backend: flashinfer_trtllm_routed | ||
| chunked-prefill-size: 32768 | ||
| max-running-requests: 16 | ||
| cuda-graph-max-bs-decode: 16 | ||
| mem-fraction-static: 0.8 | ||
| benchmark: | ||
| env: | ||
| CONC: '8' | ||
| KV_OFFLOADING: none | ||
|
|
||
| override_tp4_c16: | ||
| roles: | ||
| agg: | ||
| args: | ||
| tensor-parallel-size: 4 | ||
| moe-runner-backend: flashinfer_trtllm_routed | ||
| chunked-prefill-size: 32768 | ||
| max-running-requests: 32 | ||
| cuda-graph-max-bs-decode: 32 | ||
| mem-fraction-static: 0.8 | ||
| benchmark: | ||
| env: | ||
| CONC: '16' | ||
| KV_OFFLOADING: none | ||
|
|
||
| override_tp4_c32: | ||
| roles: | ||
| agg: | ||
| args: | ||
| tensor-parallel-size: 4 | ||
| moe-runner-backend: flashinfer_trtllm_routed | ||
| chunked-prefill-size: 32768 | ||
| max-running-requests: 64 | ||
| cuda-graph-max-bs-decode: 64 | ||
| mem-fraction-static: 0.8 | ||
| benchmark: | ||
| env: | ||
| CONC: '32' | ||
| KV_OFFLOADING: none | ||
|
|
||
| # High-Throughput cell: TP4 with EP4 (MegaMoE). The scheduler batch and decode graph | ||
| # batch are the cookbook's 128. | ||
|
|
||
| override_tp4_ep4_c64: | ||
| roles: | ||
| agg: | ||
| args: | ||
| tensor-parallel-size: 4 | ||
| expert-parallel-size: 4 | ||
| chunked-prefill-size: 32768 | ||
| max-running-requests: 128 | ||
| cuda-graph-max-bs-decode: 128 | ||
| mem-fraction-static: 0.8 | ||
| env: | ||
| SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: '32768' | ||
| benchmark: | ||
| env: | ||
| CONC: '64' | ||
| KV_OFFLOADING: none | ||
|
|
||
| override_tp4_ep4_c128: | ||
| roles: | ||
| agg: | ||
| args: | ||
| tensor-parallel-size: 4 | ||
| expert-parallel-size: 4 | ||
| chunked-prefill-size: 32768 | ||
| max-running-requests: 128 | ||
| cuda-graph-max-bs-decode: 128 | ||
| mem-fraction-static: 0.8 | ||
|
Comment on lines
+149
to
+158
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 (optional) The conc 128 point admits only 128 requests, 1x CONC. Subagent fan-out above that waits in the SGLang queue and is measured as TTFT. Every other point here, and the vLLM sibling at c128, admits 2x CONC. In Why this was flaggedThe matrix point Verification: In |
||
| env: | ||
| SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: '32768' | ||
| benchmark: | ||
| env: | ||
| CONC: '128' | ||
| KV_OFFLOADING: none | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8880,6 +8880,29 @@ minimaxm31flash-fp4-gb300-vllm-agentic-dspark: | |
| - { tp: 4, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 16, 32], srt-recipe: agentic.yaml } | ||
| - { tp: 4, ep: 4, dp-attn: true, spec-decoding: mtp, kv-offloading: none, conc-list: [64, 96, 128], router: { name: vllm-router, version: "0.1.14" }, srt-recipe: agentic.yaml } | ||
|
|
||
| # MiniMax-M3.1-Flash AgentX on GB300 with SGLang: day-zero image from the SGLang | ||
| # cookbook (https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3.1), | ||
| # manifest digest verified 2026-10-24. Serve flags follow the cookbook's | ||
| # GB300 Balanced (TP4) and High-Throughput (TP4, EP4). | ||
| # Seven-token DSpark (bundled draft); synthetic acceptance follows | ||
| # infx/golden_al_distribution/minimaxm31flash_dspark.yaml. | ||
|
Comment on lines
+8883
to
+8888
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 (optional) This adds header and provenance comments that AGENTS.md's mandatory code-comment rules forbid, so later readers get history in place of technical notes. The class is section/header or cookbook-provenance comments on new lines. Fix: delete these comments and keep only one-line technical notes on specific values (the health_check comment is fine), which covers the 5 sites listed. Same pattern at 5 sites (nvidia-master.yaml:8875-8880, nvidia-master.yaml:8893, nvidia-master.yaml:8895, agentic.yaml:24-25, agentic.yaml:51-52 and 130-131). Why this was flaggedAGENTS.md "Code comments (mandatory)" applies to recipes and configs. It says: no header comments above a recipe, override or section; no comments about "where it was derived from"; only short technical explanations. nvidia-master.yaml:8875-8880 is a block above the new key. It describes the config, cites the SGLang cookbook URL, and records "manifest digest verified 2026-10-24", which is provenance and a date after today's 2026-10-23. nvidia-master.yaml:8893 "# Balanced: TP4." and :8895 "# High-Throughput: TP4 + EP4 (MegaMoE)." are section labels that restate the row. The recipe has "Serve flags follow the cookbook's GB300 cells at " (agentic.yaml:24-25). It also has the section headers "# TP4 Balanced cell ..." (51-52) and "# High-Throughput cell ..." (130-131) above the override groups. The sibling vLLM GB300 recipe has none of these. Merging adds forbidden comment styles that the repo's own rules say belong in git history and perf-changelog.yaml. Verification: AGENTS.md:60-66 "Code comments (mandatory)" says the rules apply to "every file, including recipes, configs". They forbid header comments "above a recipe, override, or section" and history comments about "where it was derived from". nvidia-master.yaml:8875-8880 is a header block above the new key. Nothing breaks at runtime. |
||
| minimaxm31flash-fp4-gb300-sglang-agentic-dspark: | ||
| image: lmsysorg/sglang:dev-minimax-m31-flash@sha256:614b1509bcb729c653528d448bff9850f8977fcad9d12ba56bd6dbe3d7985e87 | ||
| model: MiniMaxAI/MiniMax-M3.1-Flash | ||
| model-prefix: minimaxm31flash | ||
| runner: cluster:gb300-nv | ||
| precision: fp4 | ||
| framework: sglang | ||
| multinode: false | ||
| srt-recipe-dir: minimaxm31flash/sglang/gb300-fp4-mtp | ||
| scenarios: | ||
| agentic-coding: | ||
| - search-space: | ||
| # Balanced: TP4. | ||
| - { tp: 4, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 16, 32], srt-recipe: agentic.yaml } | ||
| # High-Throughput: TP4 + EP4 (MegaMoE). | ||
| - { tp: 4, ep: 4, spec-decoding: mtp, kv-offloading: none, conc-list: [64, 128], srt-recipe: agentic.yaml } | ||
|
|
||
| # H200 AgentX arm for DeepSeek-V4.1-Flash. Upstream marks h200 verified and says | ||
| # the GB200 NVL4 TP4 layout becomes TP8 on 8-GPU nodes, so this is TP8. | ||
| # `precision: fp4` labels the checkpoint's MXFP4 routed experts, as on the | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 The conc 64 and 128 results will be published as High-Throughput MegaMoE numbers, but those runs never actually use MegaMoE. Neither EP4 override sets
moe-a2a-backend: megamoe, so SGLang falls back to its default MoE path. That leavesSGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANKwith no effect. Fix: addmoe-a2a-backend: megamoeto the args of both EP4 overrides (and the MoE runner the cookbook's High-Throughput cell uses), which covers the 2 sites listed. Same pattern at 2 sites (agentic.yaml:135, agentic.yaml:152).Why this was flagged
The matrix point
{ tp: 4, ep: 4, ... conc-list: [64, 128] }at nvidia-master.yaml binds tooverride_tp4_ep4_c64andoverride_tp4_ep4_c128(agentic.yaml:132, agentic.yaml:149). Their args set onlytensor-parallel-size: 4andexpert-parallel-size: 4, with nomoe-a2a-backendand nomoe-runner-backend. Every other SGLang MegaMoE recipe in the repo setsmoe-a2a-backend: megamoenext to this same env var, for example dsv41flash/sglang/gb200-fp4-mtp/agentic.yaml:189 with :196, and dsv4/sglang/b200-fp4-mtp/agentic.yaml:232 with :246. So the server starts with the default a2a backend, and the DeepGEMM MegaMoE token cap at agentic.yaml:143 and :160 is never read. Nothing in the binder adds that flag, so the conc 64/128 results land in the changelog labelled 'TP4 + EP4 (MegaMoE)' even though they come from a different MoE path.Verification: The args of
override_tp4_ep4_c64(agentic.yaml:132-147) andoverride_tp4_ep4_c128(:149-164) set nomoe-a2a-backendand nomoe-runner-backend, and neither does the base block. The only MegaMoE-related setting is the env varSGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: '32768'(:143, :160). Every other SGLang recipe that sets this env var also turns MegaMoE on explicitly.