Repository navigation
[AgentX] MiniMax-M3.1-Flash B300 SGLang: add TP2 configs / [AgentX] MiniMax-M3.1-Flash B300 SGLang:新增 TP2 配置 - #4524
Conversation
Adds a TP2 config key with L2 HiCache at conc 32 to 48; conc 44 to 48 use longest-prefix-match scheduling. 为 MiniMax-M3.1-Flash B300 SGLang AgentX 新增 TP2 + L2 HiCache 配置,覆盖 conc 32 到 48,其中 conc 44 到 48 使用 longest-prefix-match 调度。
|
Thanks for the contribution!
中文感谢你的贡献!
|
| # the decode graph batch matches it. | ||
|
|
||
| zip_override_tp2_hicache: | ||
| roles: | ||
| agg: | ||
| args: | ||
| tensor-parallel-size: 2 | ||
| moe-runner-backend: flashinfer_trtllm_routed | ||
| chunked-prefill-size: 32768 | ||
| max-running-requests: [64, 72, 80] | ||
| cuda-graph-max-bs-decode: [64, 72, 80] | ||
| mem-fraction-static: 0.87 | ||
| enable-hierarchical-cache: true | ||
| hicache-size: 295 | ||
| hicache-write-policy: write_back | ||
| benchmark: | ||
| env: | ||
| CONC: ['32', '36', '40'] | ||
| KV_OFFLOADING: dram | ||
|
|
There was a problem hiding this comment.
🟡 nit (optional): New override and search-space comments are section headers that restate code, which AGENTS.md forbids. sweep:^# TP2 with L2 HiCache|^# Higher-concurrency points|# TP2 with L2 HiCache: 295 GB Examples: agentic-tp2.yaml:72 repeats schedule-policy: lpm, and nvidia-master.yaml:9484 repeats the tp, hicache size and conc-list values. Fix: delete these header comments. Keep only a one-line why at a specific line where needed, such as next to max-running-requests for the 2x CONC choice. [also at: inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/minimaxm31flash/sglang/b300-fp4-mtp/agentic-tp2.yaml:72 - nit: Two new comments only restate the config next to them, which the repo's comment rules forbid. "Higher-concurrency points use longest-prefix-match scheduling." just repeats schedule-policy: lpm, and the master-config comment "TP2 with L2 HiCache: 295 GB per rank at conc 32-48." repeats the row below it.]
Why this was flagged
AGENTS.md "Comments (mandatory, every file)" bans "header comments describing a file, topology, config, recipe, override, or section" and says "never restate the code". agentic-tp2.yaml:52-53 sits above zip_override_tp2_hicache and repeats write_back and 295 GB, which are already in the override's hicache-write-policy and hicache-size lines. agentic-tp2.yaml:72 sits above zip_override_tp2_hicache_lpm and only restates schedule-policy: lpm. nvidia-master.yaml:9484 "# TP2 with L2 HiCache: 295 GB per rank at conc 32-48." restates the search-space row below it. The sibling agentic.yaml has similar section comments, which is precedent but not a sanctioned exception. Nothing breaks at runtime; this is a convention slip that reviewers following AGENTS.md would flag.
Verification: AGENTS.md lines 54-58 say: "No header comments describing a file, topology, config, recipe, override, or section" and "never restate the code". agentic-tp2.yaml:51-52 and :72 are section headers that repeat hicache-size: 295 and schedule-policy: lpm. nvidia-master.yaml:9484 restates the row below it. Nothing functional breaks.
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX config ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#46506 has been merged. This PR's AgentX config ( This PR has no authorized |
…sglang-agentx-tp2 # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=40739166138 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=40739166138 |
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#46547 has been merged. This PR's AgentX config ( This PR has no authorized |
…sglang-agentx-tp2 # Conflicts: # inferencex-e2e/perf-changelog.yaml
…sglang-agentx-tp2
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( This PR's |
|
[by Claude Code] sgl-project/sglang#46948 has been merged. This PR's AgentX config ( This PR's |
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( This PR's |
|
[by Claude Code] sgl-project/sglang#47027 has been merged. This PR's AgentX config ( This PR's |
…sglang-agentx-tp2 # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
/use 40739166138 |
|
@nvpohanh staged run 40739166138. This run remains available across future |
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( The commands mirror the launch measured in run 40739166138, pinned by this PR's |
|
[by Claude Code] sgl-project/sglang#47214 has been merged. This PR's AgentX config ( The commands mirror the launch measured in run 40739166138, pinned by this PR's |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
❌❌❌ REJECTED ❌❌❌@nvpohanh Blocking: both recipes override only the DSpark draft's KV cache to FP8 ( ❌ Check 13 (Draft precision): FAIL — Draft KV-cache dtype override below the framework default, aimed only at the draft. Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (In-PR sweep + evals): PASS — run 40739166138 on the pinned head ✅ Check 2 (Eval accuracy): PASS — gsm8k ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#47214 merged 2026-10-29. The ✅ Check 4 (Reuse command): PASS — COLLABORATOR ✅ Check 5 (Checklist template): PASS — all 20 items of the current ✅ Check 6 (Upstream image/order/provenance): PASS — the image is upstream ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (Architecture hacks): PASS — there is no ✅ Check 9 (Chat template): PASS — AgentX trace replay goes through the shared AIPerf chat-completions client, with the ✅ Check 10 (Engine patches): PASS — neither recipe has a setup script, patch, source edit or wheel install. The launch command is a stock ✅ Check 11 (Golden AL): PASS — the job logs show ➖ Check 12 (Append-only): N/A — the new changelog entry does not set ✅ Check 14 (Pareto coverage): PASS — curve minimaxm31flash/agentic-coding/b300_sglang/fp4, run 40739166138 attempt 1, P90 E2EL vs ✅ Check 15 (PR description): PASS — the body matches the pinned code: TP2 conc 32-48, 370 GB HiCache per rank, the 749 GB allocation matches ✅ Check 16 (NVIDIA overview): PASS — the NVIDIA overview lists MiniMax-M3.1-Flash with SGLang populated. ➖ Check 17 (Metrics publication): N/A — this is direct SGLang serving with no Dynamo TRT-LLM path. ✅ Check 18 (Recipe/image/sweep consistency): PASS — the master TP2 entry (conc 32-48, Assessed commit: |
…sglang-agentx-tp2 # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
Ingest report — ❌ Ingest into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 16, rawBenchmarkRows 16, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 2638, gpuMetricSamplesNew 317922, powerPublicationPoints 16, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ❌ 写入 上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Bring in #4524, #4648, #4650 and #4652. perf-changelog.yaml keeps main's MiniMax-M3.1-Flash B300 entry ahead of the #2916 and #4243 video entries; nvidia-master.yaml and MODELS.md merged cleanly. The sweep on 74bb681 (run 40873178356) stays the proof for this PR, so PR benchmark setup is skipped. 中文:合并 main(#4524、#4648、#4650、#4652)。perf-changelog.yaml 将 main 的 MiniMax-M3.1-Flash B300 条目放在 #2916 与 #4243 的视频条目之前; nvidia-master.yaml 与 MODELS.md 自动合并无冲突。74bb6816 上的 sweep (run 40873178356)仍是本 PR 的验证依据,因此跳过 PR 基准 setup。
- perf-changelog: resolve the conflict with main's new entries (#4524, #4599, #4647, #4627, #4596, #4526, #4664) using prepare_perf_changelog_merge resolve-conflict; main's bytes are kept and this PR's entry is re-appended unchanged. - amd-master.yaml: #4664 deleted dsv4-fp4-mi355x-sglang-agentic-mtp right after this PR's glm5.2-fp4-mi355x-atom-disagg-agentic-mtp block. Keep the GLM-5.2 block byte-for-byte and take main's deletion. - main's srt-slurm pin moves to v2.65.2 (#4601); 573 and 604-atom-tp-from-pp.patch still apply in order on 59920ea1. All six files this PR changes are byte-identical to the previous head 1a2707f. - perf-changelog:用 prepare_perf_changelog_merge resolve-conflict 解决与 main 新条目(#4524、#4599、#4647、#4627、#4596、#4526、#4664)的冲突; 保留 main 的原始字节,本 PR 的条目原样追加在末尾。 - amd-master.yaml:#4664 删除了紧跟在本 PR 的 glm5.2-fp4-mi355x-atom-disagg-agentic-mtp 之后的 dsv4-fp4-mi355x-sglang-agentic-mtp。GLM-5.2 条目逐字节保留,同时采纳 main 的删除。 - main 的 srt-slurm 固定版本升到 v2.65.2(#4601);573 和 604-atom-tp-from-pp.patch 仍能在 59920ea1 上按顺序打上。 本 PR 修改的六个文件与上一个 head 1a2707f 逐字节一致。 [skip-sweep] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add TP2 points with DRAM KV offload at conc 32 to 48 to the existing B300 SGLang AgentX curve, and store the DSpark draft KV cache in FP8 on every point. The whole curve is rerun.
agentic-tp2.yamlrecipe adds conc 32 to 48. The host cache is 370 GB per rank, inside the 749 GB allocation of a two-GPU point at DRAM utilization 1.0. Conc 44 to 48 use longest-prefix-match scheduling.agentic.yamlrecipe also gainsspeculative-draft-kv-cache-dtype: fp8_e4m3, so the existing TP4 points change too. FP8 replaces the bf16 default for the draft KV cache, which frees about 10 GB of HBM per rank. All TP4 points are rerun with this setting.中文
为现有的 B300 SGLang AgentX 曲线新增 conc 32 到 48 的 TP2 + DRAM KV offload 点,并在所有点上将 DSpark draft KV cache 存为 FP8。整条曲线会重新运行。
agentic-tp2.yamlrecipe 新增 conc 32 到 48。每个 rank 的 host cache 为 370 GB,在 DRAM utilization 1.0 下两卡点 749 GB 的分配额度以内。conc 44 到 48 使用 longest-prefix-match 调度。agentic.yamlrecipe 也加入了speculative-draft-kv-cache-dtype: fp8_e4m3,因此现有的 TP4 点也会变化。draft KV cache 由默认的 bf16 改为 FP8,每个 rank 可节省约 10 GB HBM。所有 TP4 点都会以此设置重新运行。AI model disclosure
Prepared with Claude Code using
claude-opus-5-5. Reviewed by the human author before submission.