Skip to content

[Klaud Cold] minimaxm31flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg: MiniMax-M3.1-Flash DEP8 + CPU offload on two GB300 trays - #4436

Closed
Oseltamivir wants to merge 5 commits into
mainfrom
feat/minimaxm31flash-gb300-dynamo-agg
Closed

Oseltamivir wants to merge 5 commits into
mainfrom
feat/minimaxm31flash-gb300-dynamo-agg

Conversation

@Oseltamivir

@Oseltamivir Oseltamivir commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

MiniMax-M3.1-Flash AgentX on two GB300 trays (8 GPUs), aggregated through Dynamo vLLM, for the high-concurrency end of the curve. Low and mid concurrency stay on the single-tray TP4 recipe (#4419).

Layout Concurrency DSpark tokens (golden AL) KV offload
DEP8 (TP1 × DP8 + EP8, dp_launch_mode: per_node) 72, 96, 128 3 (2.88) vLLM SimpleCPUOffload, 179 GiB per DP rank, lazy

Why there are no TP8/TEP8 points

The model has 64 query heads over 4 KV heads. vLLM's NVFP4 KV-cache attention kernels need 16 query heads per KV head on each rank. Attention TP8 leaves 8, and the engine refuses to start:

MiniMax-M3 does not support an NVFP4 KV cache with tensor_parallel_size=8 yet: each rank holds 8 query heads over 1 KV head(s), and the NVFP4 attention kernels need 16 per KV head.

DEP8 runs attention at TP1, so it is unaffected.

Validation

infx generate binds all 3 points to one variant each with the expected TP/DP/EP, draft tokens, golden AL and offload size, and srtctl accepts them.

@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@Oseltamivir Oseltamivir added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Oct 23, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Comment thread inferencex-e2e/perf-changelog.yaml Outdated
description:
- "Add MiniMax-M3.1-Flash AgentX on two GB300 trays (8 GPUs) aggregated through Dynamo vLLM, with the single-node GB300 serve flags (NVFP4 KV and indexer cache, 600 s chat-template render timeout) and the day-0 arm64 vLLM image."
- "TP8 c1-c32 and TEP8 (TP8 + EP8) c48-c64 with 7-token DSpark (golden AL 4.04); DEP8 (TP1 x DP8 + EP8, per-node DP launch) c72-c128 with 3-token DSpark (golden AL 2.88) and vLLM Simple CPU KV offload (179 GiB per DP rank, lazy). FlashInfer PrimiTS drafter throughout."
pr-link: TBD

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The changelog check fails on this PR, and a main-branch merge would also be rejected. The new block's pr-link is TBD, which the validator does not accept. Accepted values are XXX, https://github.com/SemiAnalysisAI/InferenceX/pull/XXX, or the canonical PR URL. Fix: set pr-link: to this PR's canonical https://github.com/SemiAnalysisAI/InferenceX/pull/<N> URL, or to XXX until the number is known.

Why this was flagged

The new entry at inferencex-e2e/perf-changelog.yaml:10998 sets pr-link: TBD. On PR runs, validate_added_pr_link in inferencex-e2e/infx/workflows/validate_perf_changelog.py:147 accepts only PR_LINK_PLACEHOLDERS (XXX and https://github.com/SemiAnalysisAI/InferenceX/pull/XXX, lines 22-25) or the exact expected PR URL. Anything else raises "new PR entry must use ... or an XXX placeholder". On main-branch runs, line 142 requires CANONICAL_PR_LINK, so TBD raises "new main-branch entry has invalid pr-link". Changelog validation therefore fails, and the entry cannot trigger result collection for minimaxm31flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg. On the base branch there is no such entry and the check passes.

Verification: The new block in inferencex-e2e/perf-changelog.yaml ends with pr-link: TBD. PR_LINK_PLACEHOLDERS (validate_perf_changelog.py lines 22-25) contains only "XXX" and ".../pull/XXX". At merge time, compare_entries(base, head, pr_number) at prepare_perf_changelog_merge.py:88 calls validate_added_pr_link with an int pr_number and raises on 'TBD'. CONTRIBUTING.md:169 names merge_with_reuse as the supported merge path, so it fails for this PR.

Comment on lines +3 to +6
# MiniMax-M3.1-Flash AgentX on two GB300 trays (8 GPUs), aggregated through
# Dynamo vLLM. Serve flags follow the single-node GB300 recipe
# (single_node/srt-slurm-recipes/minimaxm31flash/vllm/gb300-fp4-mtp); the
# Dynamo setup follows minimaxm3/vllm/gb200-fp4/agentx.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Maintainers get new header and provenance comments that AGENTS.md "Code comments (mandatory)" forbids in recipes and master configs. The recipe top block and the master-entry block describe the topology and say where flags were taken from. Fix: delete the comment blocks that sit above the file, override or entry, and keep only per-line technical reasons such as the existing block-size and render-timeout notes. That covers the 5 sites listed. Same pattern at 5 sites (agg-variants.yaml:3-6, agg-variants.yaml:95, agg-variants.yaml:107, agg-variants.yaml:121-124, nvidia-master.yaml:5611-5614).

Why this was flagged

AGENTS.md:64 says "Never add broad, overarching comments at the top of a file or above a recipe, override, or section. Do not describe what the file, topology, or config is." AGENTS.md:65 also bans comments about where code was derived from. agg-variants.yaml:3-6 describes the file ("MiniMax-M3.1-Flash AgentX on two GB300 trays...") and gives provenance ("Serve flags follow the single-node GB300 recipe...; the Dynamo setup follows minimaxm3/vllm/gb200-fp4/agentx"). The comments above override_tp8 (line 95), override_tep8 (line 107) and override_dep8_vllm_simple (lines 121-124) describe each override's topology. nvidia-master.yaml:5611-5614 describes the entry's layouts and says "as in the single-node GB300 recipe". The base branch has none of these comments, so the diff adds every one of them.

Verification: This happens on every merge, because the comments are newly added in this diff. AGENTS.md:64 forbids header comments "above a recipe, override, or section" that describe "what the file, topology, or config is". AGENTS.md:65 forbids comments about "where it was derived from". Nothing breaks at runtime.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

@Oseltamivir Oseltamivir changed the title [Klaud Cold] minimaxm31flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg: MiniMax-M3.1-Flash TP8/TEP8/DEP8 on two GB300 trays [Klaud Cold] minimaxm31flash-fp4-gb300-dynamo-vllm-agentic-dspark-agg: MiniMax-M3.1-Flash DEP8 + CPU offload on two GB300 trays Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant