Skip to content

[Klaud Cold] minimaxm31flash-fp4-gb300-sglang-agentic-dspark: day-zero MiniMax-M3.1-Flash GB300 SGLang recipe - #4443

Closed
Oseltamivir wants to merge 3 commits into
mainfrom
feat/minimaxm31flash-gb300-sglang
Closed

Oseltamivir wants to merge 3 commits into
mainfrom
feat/minimaxm31flash-gb300-sglang

Conversation

@Oseltamivir

Copy link
Copy Markdown
Collaborator

Summary

Day-zero MiniMax-M3.1-Flash (MiniMaxAI/MiniMax-M3.1-Flash, NVFP4) AgentX on GB300 with SGLang, following the SGLang cookbook: https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3.1 (config source: docs/src/snippets/configs/MiniMaxAI/minimax-m3.1.jsx).

  • Image: lmsysorg/sglang:dev-minimax-m31-flash, pinned by digest (verified on Docker Hub 2026-10-24).
  • Serve flags: the cookbook's GB300 Balanced (TP4, TRT-LLM routed MoE) for conc 1–32 and High-Throughput (TP4 + EP4 MegaMoE) for conc 64–128.
  • Common flags: --speculative-algorithm DSPARK --speculative-use-block-verification, --cuda-graph-backend-prefill breakable, minimax-m3 reasoning and tool parsers, --tokenizer-worker-num 8.
  • Speculative decoding: seven-token DSpark (--speculative-dspark-block-size 7, the bundled draft's dspark_block_size). It's set explicitly so throughput runs resolve synthetic acceptance from minimaxm31flash_dspark.yaml ([Klaud Cold] Add MiniMax-M3.1-Flash DSpark golden AL curve and SPEED-Bench collector / [Klaud Cold] 新增 MiniMax-M3.1-Flash DSpark 黄金 AL 曲线与 SPEED-Bench 采集脚本 #4425): AL 4.04 at 7 tokens, thinking on (adaptive, the default).
  • Scheduler batch: the TP4 arm uses 2× CONC for AgentX subagent fan-out, capped at the cookbook's 128, with the decode graph batch matching it.
  • Checked locally: infx generate binds every matrix point to exactly one recipe variant, with SGLANG_SIMULATE_ACC_LEN=4.04.

vLLM siblings: #4419 (GB300), #4420 (MI355X), #4421 (B300), #4422 (H100), #4423 (H200), #4426/#4431 (B200), #4433 (GB200).

…1-Flash SGLang AgentX on GB300

Serve flags follow the SGLang cookbook's MiniMax-M3.1 cells with the day-zero
dev-minimax-m31-flash image. Seven-token DSpark with synthetic acceptance from
the minimaxm31flash_dspark golden curve.
@Oseltamivir
Oseltamivir requested a review from a team October 23, 2026 17:38
@Oseltamivir Oseltamivir added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Oct 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@Oseltamivir Oseltamivir added full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) and removed full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) labels Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Comment on lines +135 to +143
args:
tensor-parallel-size: 4
expert-parallel-size: 4
chunked-prefill-size: 32768
max-running-requests: 128
cuda-graph-max-bs-decode: 128
mem-fraction-static: 0.8
env:
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: '32768'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The conc 64 and 128 results will be published as High-Throughput MegaMoE numbers, but those runs never actually use MegaMoE. Neither EP4 override sets moe-a2a-backend: megamoe, so SGLang falls back to its default MoE path. That leaves SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK with no effect. Fix: add moe-a2a-backend: megamoe to the args of both EP4 overrides (and the MoE runner the cookbook's High-Throughput cell uses), which covers the 2 sites listed. Same pattern at 2 sites (agentic.yaml:135, agentic.yaml:152).

Why this was flagged

The matrix point { tp: 4, ep: 4, ... conc-list: [64, 128] } at nvidia-master.yaml binds to override_tp4_ep4_c64 and override_tp4_ep4_c128 (agentic.yaml:132, agentic.yaml:149). Their args set only tensor-parallel-size: 4 and expert-parallel-size: 4, with no moe-a2a-backend and no moe-runner-backend. Every other SGLang MegaMoE recipe in the repo sets moe-a2a-backend: megamoe next to this same env var, for example dsv41flash/sglang/gb200-fp4-mtp/agentic.yaml:189 with :196, and dsv4/sglang/b200-fp4-mtp/agentic.yaml:232 with :246. So the server starts with the default a2a backend, and the DeepGEMM MegaMoE token cap at agentic.yaml:143 and :160 is never read. Nothing in the binder adds that flag, so the conc 64/128 results land in the changelog labelled 'TP4 + EP4 (MegaMoE)' even though they come from a different MoE path.

Verification: The args of override_tp4_ep4_c64 (agentic.yaml:132-147) and override_tp4_ep4_c128 (:149-164) set no moe-a2a-backend and no moe-runner-backend, and neither does the base block. The only MegaMoE-related setting is the env var SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: '32768' (:143, :160). Every other SGLang recipe that sets this env var also turns MegaMoE on explicitly.

Comment thread inferencex-e2e/perf-changelog.yaml Outdated
- "Serve flags follow the cookbook's GB300 Balanced (TP4) and High-Throughput (TP4, EP4): DSpark with block verification, breakable prefill graphs, minimax-m3 reasoning and tool parsers, 8 tokenizer workers."
- "Seven-token DSpark (the bundled draft's dspark_block_size); throughput runs use synthetic acceptance from the minimaxm31flash_dspark golden curve (AL 4.04 at 7 draft tokens, #4425)."
- "Search space: Balanced TP4 at conc 1-32; TP4 + EP4 at conc 64-128."
pr-link: TBD

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The changelog check on this PR will fail, and so will the merge-prep step that rewrites the link. The new block uses pr-link: TBD, which is not an accepted placeholder. PR_LINK_PLACEHOLDERS in validate_perf_changelog.py:22 only allows XXX or https://github.com/SemiAnalysisAI/InferenceX/pull/XXX. Fix: set the line to pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX or to this PR's canonical URL. Both validators accept either form.

Why this was flagged

The appended entry at perf-changelog.yaml:11018 has pr-link: TBD. On a PR run, compare_entries calls validate_added_pr_link (validate_perf_changelog.py:193). That function raises new PR entry must use '<url>' or an XXX placeholder; found 'TBD' (line 147-149) because TBD is not in PR_LINK_PLACEHOLDERS (line 22-25). On main, it fails CANONICAL_PR_LINK with new main-branch entry has invalid pr-link: 'TBD'. prepare_perf_changelog_merge.py:108-110 also refuses with appended entry N has unexpected pr-link 'TBD', so the merge helper cannot swap in the real link. No other entry in the file uses TBD. The changelog validation job fails, and result collection for minimaxm31flash-fp4-gb300-sglang-agentic-dspark is blocked until the link is fixed.

Verification: Trigger: the PR goes through the reuse-assisted merge with the pr-link still TBD. merge_with_reuse.py:631 always calls canonicalize_changelog, which reaches validate_added_pr_link (validate_perf_changelog.py:193). Since TBD is not in PR_LINK_PLACEHOLDERS (lines 22-25), it raises, so merge prep stops with an error. The PR-run CI check does not fail.

Comment on lines +149 to +158
override_tp4_ep4_c128:
roles:
agg:
args:
tensor-parallel-size: 4
expert-parallel-size: 4
chunked-prefill-size: 32768
max-running-requests: 128
cuda-graph-max-bs-decode: 128
mem-fraction-static: 0.8

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) The conc 128 point admits only 128 requests, 1x CONC. Subagent fan-out above that waits in the SGLang queue and is measured as TTFT. Every other point here, and the vLLM sibling at c128, admits 2x CONC. In override_tp4_ep4_c128, max-running-requests: 128 caps admission at CONC, while c64 with the same server admits 2x. Fix: admit 2x CONC (256) at c128 and keep cuda-graph-max-bs-decode: 128, as the dsv41flash GB300 SGLang DEP2 c128 variant does, or drop the c128 point. 256 x 8 verify tokens is far below the 32768-token MegaMoE per-rank budget.

Why this was flagged

The matrix point { tp: 4, ep: 4, ... conc-list: [64, 128] } (nvidia-master.yaml:8896) binds c128 to override_tp4_ep4_c128, which sets max-running-requests: 128 and CONC: '128' (agentic.yaml:149-163). AgentX replays sessions at --concurrency CONC (infx/bench/agentic/replay.py:142). Sessions with subagents put more requests in flight than CONC. That is why this recipe's own comment (agentic.yaml:51-52) and its siblings size admission at 2x CONC. At c128, any burst above 128 in-flight requests waits in the scheduler's waiting queue, and the wait shows up in TTFT and end-to-end latency. c64 runs the same server with 2x headroom, so c64 and c128 are not measured under the same admission rule. The vLLM GB300 sibling DEP4 c128 admits 64 per rank x 4 = 256 (vllm/gb300-fp4-mtp/agentic.yaml), so the published SGLang c128 row is biased against SGLang. That cap is not a technical limit here.

Verification: In agentic.yaml, override_tp4_ep4_c128 sets max-running-requests: 128 together with CONC: '128' (lines 149-163). That admits only 1x CONC. In the same file, override_tp4_ep4_c64 keeps max-running-requests: 128 at CONC: '64', which is 2x. Requests above 128 wait in SGLang's queue, and that wait is counted as TTFT.

Comment on lines +8875 to +8880
# MiniMax-M3.1-Flash AgentX on GB300 with SGLang: day-zero image from the SGLang
# cookbook (https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3.1),
# manifest digest verified 2026-10-24. Serve flags follow the cookbook's
# GB300 Balanced (TP4) and High-Throughput (TP4, EP4).
# Seven-token DSpark (bundled draft); synthetic acceptance follows
# infx/golden_al_distribution/minimaxm31flash_dspark.yaml.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) This adds header and provenance comments that AGENTS.md's mandatory code-comment rules forbid, so later readers get history in place of technical notes. The class is section/header or cookbook-provenance comments on new lines. Fix: delete these comments and keep only one-line technical notes on specific values (the health_check comment is fine), which covers the 5 sites listed. Same pattern at 5 sites (nvidia-master.yaml:8875-8880, nvidia-master.yaml:8893, nvidia-master.yaml:8895, agentic.yaml:24-25, agentic.yaml:51-52 and 130-131).

Why this was flagged

AGENTS.md "Code comments (mandatory)" applies to recipes and configs. It says: no header comments above a recipe, override or section; no comments about "where it was derived from"; only short technical explanations. nvidia-master.yaml:8875-8880 is a block above the new key. It describes the config, cites the SGLang cookbook URL, and records "manifest digest verified 2026-10-24", which is provenance and a date after today's 2026-10-23. nvidia-master.yaml:8893 "# Balanced: TP4." and :8895 "# High-Throughput: TP4 + EP4 (MegaMoE)." are section labels that restate the row. The recipe has "Serve flags follow the cookbook's GB300 cells at " (agentic.yaml:24-25). It also has the section headers "# TP4 Balanced cell ..." (51-52) and "# High-Throughput cell ..." (130-131) above the override groups. The sibling vLLM GB300 recipe has none of these. Merging adds forbidden comment styles that the repo's own rules say belong in git history and perf-changelog.yaml.

Verification: AGENTS.md:60-66 "Code comments (mandatory)" says the rules apply to "every file, including recipes, configs". They forbid header comments "above a recipe, override, or section" and history comments about "where it was derived from". nvidia-master.yaml:8875-8880 is a header block above the new key. Nothing breaks at runtime.

Keep main's entries and re-add the GB300 SGLang changelog and master-config blocks.
@Oseltamivir

Copy link
Copy Markdown
Collaborator Author

replaced by #4456

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant