Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
base:
schema: 2
name: minimaxm31flash-fp4-gb300-sglang-agentic
resources:
gpu_type: gb300
gpus_per_node: 4
frontend:
type: sglang
enable_multiple_frontends: false
observability:
enabled: false
tachometer:
enabled: false
engine: sglang
# Cold loads of the 250 GB checkpoint plus JIT MSA kernel builds and graph capture outlast the default.
health_check:
max_attempts: 1440
interval_seconds: 5
roles:
agg:
nodes: 1
workers: 1
gpus: 4
# Serve flags follow the cookbook's GB300 cells at
# https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3.1.
args:
served-model-name: MiniMaxAI/MiniMax-M3.1-Flash
trust-remote-code: true
reasoning-parser: minimax-m3
tool-call-parser: minimax-m3
# Seven-token DSpark, the bundled draft's dspark_block_size. Throughput runs
# switch to synthetic acceptance at the golden acceptance length.
speculative-algorithm: DSPARK
speculative-dspark-block-size: 7
speculative-use-block-verification: true
cuda-graph-backend-prefill: breakable
tokenizer-worker-num: 8
watchdog-timeout: 3600
enable-metrics: true
env:
PYTHONNOUSERSITE: '1'
PYTHONUNBUFFERED: '1'
SGLANG_TIMEOUT_KEEP_ALIVE: '1800'
FMHA_SM100_ALLOW_JIT: '1'
benchmark:
env:
# Warmup can leave bytes unacknowledged past AIPerf's 30 s default.
AIPERF_HTTP_TCP_USER_TIMEOUT: '900000'
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:'

# TP4 Balanced cell (TRT-LLM routed MoE). The scheduler batch is 2x CONC (AgentX
# subagent fan-out), capped at the cookbook's 128; the decode graph batch matches it.

override_tp4_c1:
roles:
agg:
args:
tensor-parallel-size: 4
moe-runner-backend: flashinfer_trtllm_routed
chunked-prefill-size: 32768
max-running-requests: 2
cuda-graph-max-bs-decode: 2
mem-fraction-static: 0.8
benchmark:
env:
CONC: '1'
KV_OFFLOADING: none

override_tp4_c4:
roles:
agg:
args:
tensor-parallel-size: 4
moe-runner-backend: flashinfer_trtllm_routed
chunked-prefill-size: 32768
max-running-requests: 8
cuda-graph-max-bs-decode: 8
mem-fraction-static: 0.8
benchmark:
env:
CONC: '4'
KV_OFFLOADING: none

override_tp4_c8:
roles:
agg:
args:
tensor-parallel-size: 4
moe-runner-backend: flashinfer_trtllm_routed
chunked-prefill-size: 32768
max-running-requests: 16
cuda-graph-max-bs-decode: 16
mem-fraction-static: 0.8
benchmark:
env:
CONC: '8'
KV_OFFLOADING: none

override_tp4_c16:
roles:
agg:
args:
tensor-parallel-size: 4
moe-runner-backend: flashinfer_trtllm_routed
chunked-prefill-size: 32768
max-running-requests: 32
cuda-graph-max-bs-decode: 32
mem-fraction-static: 0.8
benchmark:
env:
CONC: '16'
KV_OFFLOADING: none

override_tp4_c32:
roles:
agg:
args:
tensor-parallel-size: 4
moe-runner-backend: flashinfer_trtllm_routed
chunked-prefill-size: 32768
max-running-requests: 64
cuda-graph-max-bs-decode: 64
mem-fraction-static: 0.8
benchmark:
env:
CONC: '32'
KV_OFFLOADING: none

# High-Throughput cell: TP4 with EP4 (MegaMoE). The scheduler batch and decode graph
# batch are the cookbook's 128.

override_tp4_ep4_c64:
roles:
agg:
args:
tensor-parallel-size: 4
expert-parallel-size: 4
chunked-prefill-size: 32768
max-running-requests: 128
cuda-graph-max-bs-decode: 128
mem-fraction-static: 0.8
env:
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: '32768'
Comment on lines +135 to +143

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The conc 64 and 128 results will be published as High-Throughput MegaMoE numbers, but those runs never actually use MegaMoE. Neither EP4 override sets moe-a2a-backend: megamoe, so SGLang falls back to its default MoE path. That leaves SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK with no effect. Fix: add moe-a2a-backend: megamoe to the args of both EP4 overrides (and the MoE runner the cookbook's High-Throughput cell uses), which covers the 2 sites listed. Same pattern at 2 sites (agentic.yaml:135, agentic.yaml:152).

Why this was flagged

The matrix point { tp: 4, ep: 4, ... conc-list: [64, 128] } at nvidia-master.yaml binds to override_tp4_ep4_c64 and override_tp4_ep4_c128 (agentic.yaml:132, agentic.yaml:149). Their args set only tensor-parallel-size: 4 and expert-parallel-size: 4, with no moe-a2a-backend and no moe-runner-backend. Every other SGLang MegaMoE recipe in the repo sets moe-a2a-backend: megamoe next to this same env var, for example dsv41flash/sglang/gb200-fp4-mtp/agentic.yaml:189 with :196, and dsv4/sglang/b200-fp4-mtp/agentic.yaml:232 with :246. So the server starts with the default a2a backend, and the DeepGEMM MegaMoE token cap at agentic.yaml:143 and :160 is never read. Nothing in the binder adds that flag, so the conc 64/128 results land in the changelog labelled 'TP4 + EP4 (MegaMoE)' even though they come from a different MoE path.

Verification: The args of override_tp4_ep4_c64 (agentic.yaml:132-147) and override_tp4_ep4_c128 (:149-164) set no moe-a2a-backend and no moe-runner-backend, and neither does the base block. The only MegaMoE-related setting is the env var SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: '32768' (:143, :160). Every other SGLang recipe that sets this env var also turns MegaMoE on explicitly.

benchmark:
env:
CONC: '64'
KV_OFFLOADING: none

override_tp4_ep4_c128:
roles:
agg:
args:
tensor-parallel-size: 4
expert-parallel-size: 4
chunked-prefill-size: 32768
max-running-requests: 128
cuda-graph-max-bs-decode: 128
mem-fraction-static: 0.8
Comment on lines +149 to +158

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) The conc 128 point admits only 128 requests, 1x CONC. Subagent fan-out above that waits in the SGLang queue and is measured as TTFT. Every other point here, and the vLLM sibling at c128, admits 2x CONC. In override_tp4_ep4_c128, max-running-requests: 128 caps admission at CONC, while c64 with the same server admits 2x. Fix: admit 2x CONC (256) at c128 and keep cuda-graph-max-bs-decode: 128, as the dsv41flash GB300 SGLang DEP2 c128 variant does, or drop the c128 point. 256 x 8 verify tokens is far below the 32768-token MegaMoE per-rank budget.

Why this was flagged

The matrix point { tp: 4, ep: 4, ... conc-list: [64, 128] } (nvidia-master.yaml:8896) binds c128 to override_tp4_ep4_c128, which sets max-running-requests: 128 and CONC: '128' (agentic.yaml:149-163). AgentX replays sessions at --concurrency CONC (infx/bench/agentic/replay.py:142). Sessions with subagents put more requests in flight than CONC. That is why this recipe's own comment (agentic.yaml:51-52) and its siblings size admission at 2x CONC. At c128, any burst above 128 in-flight requests waits in the scheduler's waiting queue, and the wait shows up in TTFT and end-to-end latency. c64 runs the same server with 2x headroom, so c64 and c128 are not measured under the same admission rule. The vLLM GB300 sibling DEP4 c128 admits 64 per rank x 4 = 256 (vllm/gb300-fp4-mtp/agentic.yaml), so the published SGLang c128 row is biased against SGLang. That cap is not a technical limit here.

Verification: In agentic.yaml, override_tp4_ep4_c128 sets max-running-requests: 128 together with CONC: '128' (lines 149-163). That admits only 1x CONC. In the same file, override_tp4_ep4_c64 keeps max-running-requests: 128 at CONC: '64', which is 2x. Requests above 128 wait in SGLang's queue, and that wait is counted as TTFT.

env:
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: '32768'
benchmark:
env:
CONC: '128'
KV_OFFLOADING: none
23 changes: 23 additions & 0 deletions inferencex-e2e/configs/nvidia-master.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8880,6 +8880,29 @@ minimaxm31flash-fp4-gb300-vllm-agentic-dspark:
- { tp: 4, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 16, 32], srt-recipe: agentic.yaml }
- { tp: 4, ep: 4, dp-attn: true, spec-decoding: mtp, kv-offloading: none, conc-list: [64, 96, 128], router: { name: vllm-router, version: "0.1.14" }, srt-recipe: agentic.yaml }

# MiniMax-M3.1-Flash AgentX on GB300 with SGLang: day-zero image from the SGLang
# cookbook (https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3.1),
# manifest digest verified 2026-10-24. Serve flags follow the cookbook's
# GB300 Balanced (TP4) and High-Throughput (TP4, EP4).
# Seven-token DSpark (bundled draft); synthetic acceptance follows
# infx/golden_al_distribution/minimaxm31flash_dspark.yaml.
Comment on lines +8883 to +8888

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) This adds header and provenance comments that AGENTS.md's mandatory code-comment rules forbid, so later readers get history in place of technical notes. The class is section/header or cookbook-provenance comments on new lines. Fix: delete these comments and keep only one-line technical notes on specific values (the health_check comment is fine), which covers the 5 sites listed. Same pattern at 5 sites (nvidia-master.yaml:8875-8880, nvidia-master.yaml:8893, nvidia-master.yaml:8895, agentic.yaml:24-25, agentic.yaml:51-52 and 130-131).

Why this was flagged

AGENTS.md "Code comments (mandatory)" applies to recipes and configs. It says: no header comments above a recipe, override or section; no comments about "where it was derived from"; only short technical explanations. nvidia-master.yaml:8875-8880 is a block above the new key. It describes the config, cites the SGLang cookbook URL, and records "manifest digest verified 2026-10-24", which is provenance and a date after today's 2026-10-23. nvidia-master.yaml:8893 "# Balanced: TP4." and :8895 "# High-Throughput: TP4 + EP4 (MegaMoE)." are section labels that restate the row. The recipe has "Serve flags follow the cookbook's GB300 cells at " (agentic.yaml:24-25). It also has the section headers "# TP4 Balanced cell ..." (51-52) and "# High-Throughput cell ..." (130-131) above the override groups. The sibling vLLM GB300 recipe has none of these. Merging adds forbidden comment styles that the repo's own rules say belong in git history and perf-changelog.yaml.

Verification: AGENTS.md:60-66 "Code comments (mandatory)" says the rules apply to "every file, including recipes, configs". They forbid header comments "above a recipe, override, or section" and history comments about "where it was derived from". nvidia-master.yaml:8875-8880 is a header block above the new key. Nothing breaks at runtime.

minimaxm31flash-fp4-gb300-sglang-agentic-dspark:
image: lmsysorg/sglang:dev-minimax-m31-flash@sha256:614b1509bcb729c653528d448bff9850f8977fcad9d12ba56bd6dbe3d7985e87
model: MiniMaxAI/MiniMax-M3.1-Flash
model-prefix: minimaxm31flash
runner: cluster:gb300-nv
precision: fp4
framework: sglang
multinode: false
srt-recipe-dir: minimaxm31flash/sglang/gb300-fp4-mtp
scenarios:
agentic-coding:
- search-space:
# Balanced: TP4.
- { tp: 4, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 16, 32], srt-recipe: agentic.yaml }
# High-Throughput: TP4 + EP4 (MegaMoE).
- { tp: 4, ep: 4, spec-decoding: mtp, kv-offloading: none, conc-list: [64, 128], srt-recipe: agentic.yaml }

# H200 AgentX arm for DeepSeek-V4.1-Flash. Upstream marks h200 verified and says
# the GB200 NVL4 TP4 layout becomes TP8 on 8-GPU nodes, so this is TP8.
# `precision: fp4` labels the checkpoint's MXFP4 routed experts, as on the
Expand Down
10 changes: 10 additions & 0 deletions inferencex-e2e/perf-changelog.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11071,3 +11071,13 @@
- "Search space: Balanced TP4 at conc 1-32; TP4 + EP4 at conc 64-128."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4444
scenario-type: [agentic-coding]

- config-keys:
- minimaxm31flash-fp4-gb300-sglang-agentic-dspark
description:
- "Add day-zero MiniMax-M3.1-Flash (MiniMaxAI/MiniMax-M3.1-Flash, NVFP4) AgentX on GB300 with SGLang image lmsysorg/sglang:dev-minimax-m31-flash (https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3.1)."
- "Serve flags follow the cookbook's GB300 Balanced (TP4) and High-Throughput (TP4, EP4): DSpark with block verification, breakable prefill graphs, minimax-m3 reasoning and tool parsers, 8 tokenizer workers."
- "Seven-token DSpark (the bundled draft's dspark_block_size); throughput runs use synthetic acceptance from the minimaxm31flash_dspark golden curve (AL 4.04 at 7 draft tokens, #4425)."
- "Search space: Balanced TP4 at conc 1-32; TP4 + EP4 at conc 64-128."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4443
scenario-type: [agentic-coding]
Loading