Skip to content

[Klaud Cold] Add MiniMax-M3.1-Flash DSpark golden AL curve and SPEED-Bench collector / [Klaud Cold] 新增 MiniMax-M3.1-Flash DSpark 黄金 AL 曲线与 SPEED-Bench 采集脚本 - #4425

Merged
Oseltamivir merged 1 commit into
mainfrom
feat/minimaxm31flash-speedbench-al
Oct 23, 2026
Merged

Oseltamivir merged 1 commit into
mainfrom
feat/minimaxm31flash-speedbench-al

Conversation

@Oseltamivir

Copy link
Copy Markdown
Collaborator

Summary

Golden SPEED-Bench acceptance-length curve for MiniMax-M3.1-Flash DSpark (minimaxm31flash_dspark.yaml), plus the collector that produced it. AgentX DSpark recipes for this model need it for synthetic acceptance.

Measured in SpeedBench AL Collection run 39895397914 on B200 (b200-nscale), TP4, image vllm/vllm-openai:minimax-m31-flash@sha256:86f582da…. Each level ran all 80 prompts with 0 failures.

num_speculative_tokens 1 2 3 4 5 6 7
AL (thinking_on) 1.82 2.44 2.88 3.19 3.46 3.81 4.04

Collector (benchmarks/single_node/speedbench/minimaxm31flash_fp4_b300_vllm.sh)

  • Based on the Kimi-K3 DSpark collector. Serve flags follow the recipes.vllm.ai Blackwell TP4 recipe: block-size 128, NVFP4 KV and indexer cache, prefill query quantization, FULL_AND_PIECEWISE graphs, minimax_m3 parsers.
  • DSpark uses the drafter built into the checkpoint (dspark/, dspark_block_size 7), so there is no draft model path and the levels go 1–7. It uses FLASH_ATTN drafter attention, an FP8 draft KV cache, probabilistic drafting and block rejection.
  • SPEED-Bench coding category, temperature 1.0, top_p 0.95, output length 4096, concurrency 64.
  • thinking_on only. The M3.1 chat template hard-codes adaptive thinking and reads only reasoning_effort. Unlike M3 it has no thinking_mode kwarg, and the day-0 vLLM minimax_m3 reasoning parser and renderer add no off switch. Collected with thinking-kwargs={} so prompts match what the AgentX server sends; AgentX always looks up thinking_on.
  • The _b300_ in the filename comes from the speedbench-al.yml naming convention.

configs/runners.yaml

  • b200-nscale gets download-root: writable-models (/data/home/sa-shared/gharunners/models, owned by sa-shared, group-writable), mirroring b300-dsxe. Before this, script-driver runs on this cluster could only use staged checkpoints.

Dispatch used to produce the curve:

gh workflow run speedbench-al.yml --ref <branch> -f runner=b200-nscale \
  -f model=MiniMaxAI/MiniMax-M3.1-Flash -f model-prefix=minimaxm31flash -f framework=vllm \
  -f image='vllm/vllm-openai:minimax-m31-flash@sha256:86f582dae25ea92802353727164ae4eb03b37b18baf9119b805ffaec2a7ef07c' \
  -f mtp-list='1 2 3 4 5 6 7' -f thinking-modes=on -f 'thinking-kwargs={}'

The B200/B300 AgentX recipes (#4421) build on this.

SPEED-Bench coding AL for the built-in DSpark drafter, levels 1-7,
thinking_on (the M3.1 template is adaptive-only), measured on B200 TP4
with the vLLM day-0 image. Give b200-nscale a writable download root so
script runs can fetch unstaged checkpoints.
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@github-actions

Copy link
Copy Markdown
Contributor

inferencex-e2e/infx diff: +18 -0

Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package.

中文

大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。

@Oseltamivir
Oseltamivir merged commit 39b1aab into main Oct 23, 2026
6 checks passed
@Oseltamivir
Oseltamivir deleted the feat/minimaxm31flash-speedbench-al branch October 23, 2026 05:01

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Comment on lines +235 to +238
local bench_rc=$?
if [[ $bench_rc -ne 0 ]]; then
echo " -> benchmark client exited rc=$bench_rc (thinking=$mode dspark=$mtp); cell will be N/A"
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) If the benchmark client fails partway, the collector still records an AL from partial data and the job passes, even though the log says the cell will be N/A. After bench_rc is non-zero, run_cell still reads the counter deltas. It stores al whenever delta_drf > 0, so a crash after some drafts writes a real-looking value into the golden YAML. Fix: when bench_rc is non-zero, set AL_RESULT[${mode}_${mtp}] to N/A, call cleanup_server and return before computing deltas, so the log message and the matrix agree.

Why this was flagged

Trigger: the client python3 "$BENCH_DRIVER" exits non-zero midway, for example from request timeouts, OOM or a server crash after a few batches. The script prints "cell will be N/A" at line 237 but does not return. Lines 240-249 still compute AL from whatever drafts the server counted, and line 251 stores it. The MEASURED check at the end of the file passes because that cell is not N/A. The emitted OUT_YAML then holds an AL computed from a subset of the 80 SPEED-Bench prompts, with no marker that the data is partial. That value can get committed as a golden curve, which AgentX synthetic acceptance consumes. The Kimi collectors copy the same logic, so this new file brings the defect back rather than fixing it. Nothing guards against it: N/A is only assigned when the server fails to start or when delta_drf is 0.

Verification: Line 235 sets local bench_rc=$?, and lines 236-238 only echo " -> benchmark client exited rc=$bench_rc ...; cell will be N/A". Nothing after that returns or sets N/A. Line 246 checks only if [[ "$delta_drf" -gt 0 ]], and line 252 stores AL_RESULT["${mode}_${mtp}"]="$al". The final guard at lines 293-303 fails the job only when every cell is N/A.

Comment on lines +93 to +96
if [[ -n "${MODEL_PATH:-}" ]]; then
if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then
hf download "$MODEL" --local-dir "$MODEL_PATH"
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) If a download into the newly writable b200-nscale model root is interrupted, every later collection skips it and fails all cells until someone deletes the folder. Line 94 only downloads when MODEL_PATH is missing or empty. A partial hf download --local-dir leaves the folder non-empty, and the exit status is never checked. Fix: on every run, call hf download "$MODEL" --local-dir "$MODEL_PATH" (it resumes and skips finished files) and exit if it fails. Other alternative: write a completion marker after a successful download and check for that marker instead of a non-empty folder.

Why this was flagged

Before this change, b200-nscale had no models.download-root. resolve_model in infx/launch/drivers/script.py:47-51 raised LaunchError for unstaged checkpoints, so nothing was downloaded there. runners.yaml:452 now resolves MODEL_PATH to /models/MiniMax-M3.1-Flash on the shared /data/home/sa-shared/gharunners/models volume, and that folder outlives the job. If a run is cancelled, hits SALLOC_TIME_LIMIT, or hf download fails partway (quota, network, gated token), the folder already holds files such as .cache/huggingface/. The test at line 94, -z "$(ls -A "$MODEL_PATH")", is then false on every later run, so the download is never resumed. The script has no set -e (line 25) and ignores the download's exit status. It goes on to start seven vllm serve processes against incomplete weights, every cell is N/A, and line 299 exits 1. This repeats until someone cleans up the shared folder by hand. Other download paths in this repo are completeness-checked and flock-serialized (perf-changelog.yaml:8042). This one has neither, and nothing else on this path checks the weights.

Verification: Trigger: a collection on b200-nscale is cancelled or has hf download fail partway. The script only downloads when the folder is missing or empty (minimaxm31flash_fp4_b300_vllm.sh ~lines 94-96). Nothing checks the exit status of hf download. After an interruption ls -A therefore returns something, and every later run skips the download.

Comment on lines +38 to +40
# AL is concurrency-independent (per-token accept/reject; no spec-disable-by-batch is
# set), so batch the SPEED-Bench pass to stay under the CI wall-time limit; conc=1
# blew the 8h budget on an earlier Kimi model.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 nit (optional): the new script keeps comment types AGENTS.md bans, a file header and a history note. sweep:(earlier Kimi model|^# MiniMax-M3.1-Flash vLLM SPEED-Bench AL matrix collector). Lines 3-23 are a header describing the file. Lines 38-40 include "conc=1 blew the 8h budget on an earlier Kimi model", which AGENTS.md cites as a banned past-results comment. Fix: delete the header block and keep only the technical reason on lines 38-39 (AL is concurrency-independent).

Why this was flagged

AGENTS.md:64 says "No header comments" and forbids describing what the file is at its top. AGENTS.md:65 says "No history" and forbids comments about past results. Lines 3-23 of the new file describe the collector, the recipe it follows and the filename convention. Lines 39-40 cite a past Kimi run blowing the 8h budget. The base has no such file. These comments were copied from the Kimi collector and now land in a new file.

Verification: nit. Trigger: the file is new in this PR, so the comments come in with this change. Lines 3-23 of inferencex-e2e/benchmarks/single_node/speedbench/minimaxm31flash_fp4_b300_vllm.sh are a top-of-file header. Lines 39-40 say "conc=1 / blew the 8h budget on an earlier Kimi model.", which is a comment about a past result. AGENTS.md:64 and AGENTS.md:65 ban both.

("dsv4dsparkprob", {"method": "dspark"}, "dsv4-pro-0813-dspark"),
("dsv41flash", {"method": "dspark"}, "dsv41flash_dspark"),
("musespark", {"method": "dflash"}, "musespark_dflash"),
("minimaxm31flash", {"method": "dspark"}, "minimaxm31flash_dspark"),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 nit (optional): the new test row adds no coverage and breaks the repo's test rules. It hits the default f"{model}_{method}" branch of curve_name (curves.py:47), which the dsv41flash and musespark rows already cover. Its is_file() assertion only checks that a committed data file exists. Fix: drop the minimaxm31flash row. test_every_committed_curve_is_valid already loads and range-checks the new YAML.

Why this was flagged

curve_name in infx/golden_al_distribution/curves.py:41-62 has special cases only for dsv4*, minimaxm3 and kimik3. For minimaxm31flash + dspark it returns the plain f"{model}_{method}" from line 47. The existing rows ("dsv41flash", {"method": "dspark"}) and ("musespark", {"method": "dflash"}) already reach that branch. AGENTS.md "Test quality" item 9 forbids tests that duplicate a covered path. Item 4 forbids pinning checked-in data, and (GOLDEN_DIR / f"{expected}.yaml").is_file() does exactly that. The new curve is already validated through real code by test_every_committed_curve_is_valid, which globs every YAML in GOLDEN_DIR. So the row only adds a test that fails when someone renames a data file, not when behavior regresses.

Verification: nit: the new parametrize row (test_golden_al_distribution.py:42) adds no branch coverage. In curves.py curve_name, minimaxm31flash with dspark skips every special case and returns the plain curve = f"{model}_{method}" (line 47), which the dsv41flash and musespark rows already reach. test_every_committed_curve_is_valid already range-checks the new YAML.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant