Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,303 @@
#!/usr/bin/env bash

# MiniMax-M3.1-Flash vLLM SPEED-Bench AL matrix collector for DSpark speculative decoding.
#
# For each DSpark speculative-token count, measure the REAL acceptance length (AL) on
# one SPEED-Bench category and emit a YAML matrix in the golden_al_distribution shape.
# The DSpark drafter ships inside the checkpoint (dspark/, dspark_block_size 7), so
# --speculative-config takes no draft model path and levels stop at 7.
# The chat template has no thinking on/off switch (thinking is always adaptive), so
# the golden curve is collected for thinking_on only, the mode AgentX resolves.
#
# Serve flags follow the Blackwell TP4 recipe at
# https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3.1-Flash: NVFP4 KV and indexer cache,
# block-size 128 (MSA), and DSpark with probabilistic drafting and block rejection.
#
# The _b300_ in the filename is the speedbench-al.yml naming convention; the golden
# curve is collected on B200 (runner b200-nscale).
#
# Usage (inside the MiniMax-M3.1-Flash vLLM container, on a B200 or B300 node):
# export MODEL=MiniMaxAI/MiniMax-M3.1-Flash
# bash benchmarks/single_node/speedbench/minimaxm31flash_fp4_b300_vllm.sh
#
# Required collection settings come from speedbench-al.yml.

set -o pipefail
repo_root="$(cd "$(dirname "$0")/../../.." && pwd)"
source "$repo_root/benchmarks/check_env.sh"
check_env_vars \
CATEGORY CHAT_TEMPLATE_KWARGS_ON DP_ATTENTION EP_SIZE MODEL MODEL_PATH \
MTP_LIST OUT_YAML PORT SPEEDBENCH_OUTPUT_LEN THINKING_MODES TP

SERVE_MODEL="${MODEL_PATH}"
GPU_MEM_UTIL="0.90"
MAX_MODEL_LEN="16384"
MAX_NUM_SEQS="128"

MODEL_KEY="$(basename "$SERVE_MODEL" | tr '[:upper:]' '[:lower:]')"
# AL is concurrency-independent (per-token accept/reject; no spec-disable-by-batch is
# set), so batch the SPEED-Bench pass to stay under the CI wall-time limit; conc=1
# blew the 8h budget on an earlier Kimi model.
Comment on lines +38 to +40

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 nit (optional): the new script keeps comment types AGENTS.md bans, a file header and a history note. sweep:(earlier Kimi model|^# MiniMax-M3.1-Flash vLLM SPEED-Bench AL matrix collector). Lines 3-23 are a header describing the file. Lines 38-40 include "conc=1 blew the 8h budget on an earlier Kimi model", which AGENTS.md cites as a banned past-results comment. Fix: delete the header block and keep only the technical reason on lines 38-39 (AL is concurrency-independent).

Why this was flagged

AGENTS.md:64 says "No header comments" and forbids describing what the file is at its top. AGENTS.md:65 says "No history" and forbids comments about past results. Lines 3-23 of the new file describe the collector, the recipe it follows and the filename convention. Lines 39-40 cite a past Kimi run blowing the 8h budget. The base has no such file. These comments were copied from the Kimi collector and now land in a new file.

Verification: nit. Trigger: the file is new in this PR, so the comments come in with this change. Lines 3-23 of inferencex-e2e/benchmarks/single_node/speedbench/minimaxm31flash_fp4_b300_vllm.sh are a top-of-file header. Lines 39-40 say "conc=1 / blew the 8h budget on an earlier Kimi model.", which is a comment about a past result. AGENTS.md:64 and AGENTS.md:65 ban both.

CONCURRENCY="64"
# Model card sampling: temperature 1.0, top_p 0.95.
TOP_P="0.95"
# speedbench-al.yml exports its thinking-kwargs input as CHAT_TEMPLATE_KWARGS_ON. The
# template only reads reasoning_effort; dispatch with -f 'thinking-kwargs={}' to keep
# the template default the AgentX server sees.
if [[ " $THINKING_MODES " == *" off "* ]]; then
echo "CRITICAL: MiniMax-M3.1-Flash has no thinking-off mode; dispatch thinking-modes=on."
exit 1
fi

SPEEDBENCH_DIR="/workspace/speed_bench_data"
RESULTS_DIR="/workspace/speedbench_results"

export VLLM_FLOAT32_MATMUL_PRECISION="high"
export VLLM_ENGINE_READY_TIMEOUT_S=3600

# `vllm bench serve` os.execv's the CLIENT into the Rust vllm-rs binary whenever the
# dataset (speed_bench qualifies) and backend are supported, with no opt-out. The
# Rust flag surface lacks --speed-bench-output-len, --save-detailed and
# --chat-template-kwargs, so every cell would die at argument parsing. Call the
# Python entrypoint directly (vllm.benchmarks.serve.main), as the CLI's own Python
# fallback does. The SERVER keeps the Rust frontend; AL is read from /metrics.
BENCH_DRIVER="$RESULTS_DIR/bench_serve_python.py"
mkdir -p "$RESULTS_DIR"
cat > "$BENCH_DRIVER" <<'PYEOF'
# Python-only `vllm bench serve`: bypasses the Rust os.execv delegation in
# vllm/entrypoints/cli/benchmark/serve.py by importing the benchmark directly.
from vllm.benchmarks.serve import add_cli_args, main

try: # import path moved in newer vLLM
from vllm.utils.argparse_utils import FlexibleArgumentParser
except ImportError:
from vllm.utils import FlexibleArgumentParser

parser = FlexibleArgumentParser(
description="vllm bench serve (Python entrypoint, no Rust delegation)"
)
add_cli_args(parser)
main(parser.parse_args())
PYEOF
# No `set -e` here, so a failed redirect would otherwise only surface later as a
# confusing "No such file" from the preflight.
if [[ ! -s "$BENCH_DRIVER" ]]; then
echo "CRITICAL: could not write the benchmark driver to $BENCH_DRIVER — aborting."
exit 1
fi

nvidia-smi

# CI downloads unstaged weights into the runner's writable model root; this covers
# a standalone run as well.
if [[ -n "${MODEL_PATH:-}" ]]; then
if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then
hf download "$MODEL" --local-dir "$MODEL_PATH"
fi
Comment on lines +93 to +96

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) If a download into the newly writable b200-nscale model root is interrupted, every later collection skips it and fails all cells until someone deletes the folder. Line 94 only downloads when MODEL_PATH is missing or empty. A partial hf download --local-dir leaves the folder non-empty, and the exit status is never checked. Fix: on every run, call hf download "$MODEL" --local-dir "$MODEL_PATH" (it resumes and skips finished files) and exit if it fails. Other alternative: write a completion marker after a successful download and check for that marker instead of a non-empty folder.

Why this was flagged

Before this change, b200-nscale had no models.download-root. resolve_model in infx/launch/drivers/script.py:47-51 raised LaunchError for unstaged checkpoints, so nothing was downloaded there. runners.yaml:452 now resolves MODEL_PATH to /models/MiniMax-M3.1-Flash on the shared /data/home/sa-shared/gharunners/models volume, and that folder outlives the job. If a run is cancelled, hits SALLOC_TIME_LIMIT, or hf download fails partway (quota, network, gated token), the folder already holds files such as .cache/huggingface/. The test at line 94, -z "$(ls -A "$MODEL_PATH")", is then false on every later run, so the download is never resumed. The script has no set -e (line 25) and ignores the download's exit status. It goes on to start seven vllm serve processes against incomplete weights, every cell is N/A, and line 299 exits 1. This repeats until someone cleans up the shared folder by hand. Other download paths in this repo are completeness-checked and flock-serialized (perf-changelog.yaml:8042). This one has neither, and nothing else on this path checks the weights.

Verification: Trigger: a collection on b200-nscale is cancelled or has hf download fail partway. The script only downloads when the folder is missing or empty (minimaxm31flash_fp4_b300_vllm.sh ~lines 94-96). Nothing checks the exit status of hf download. After an interruption ls -A therefore returns something, and every later run skips the download.

else
if [[ "$SERVE_MODEL" != /* ]]; then hf download "$SERVE_MODEL"; fi
fi

echo "=== Downloading SPEED-Bench dataset ==="
pip install -q datasets tiktoken
curl -LsSf https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py \
| python3 - --config qualitative --output_dir "$SPEEDBENCH_DIR"

if [[ ! -f "$SPEEDBENCH_DIR/qualitative.jsonl" ]]; then
echo "CRITICAL: SPEED-Bench download failed — $SPEEDBENCH_DIR/qualitative.jsonl not found"
exit 1
fi

# Preflight the client flags every cell uses; otherwise a CLI mismatch only shows up
# as an all-N/A matrix after eight full server starts (~1h). Probe the driver, not
# `vllm bench serve` (its help exits before the Rust execv), and ask for --help=all
# since plain --help prints only a group summary.
BENCH_HELP="$(python3 "$BENCH_DRIVER" --help=all 2>&1)"
for flag in --speed-bench-category --speed-bench-output-len --chat-template-kwargs --save-detailed; do
if [[ "$BENCH_HELP" != *"$flag"* ]]; then
echo "CRITICAL: the Python benchmark entrypoint does not support $flag — aborting."
echo "--- python3 $BENCH_DRIVER --help=all ---"
echo "$BENCH_HELP"
exit 1
fi
done
echo "=== Benchmark client flag preflight OK ==="

# speedbench-al.yml allocates TP=8 GPUs; serve on the first four with the recipe's
# TP4 Blackwell layout. AL does not depend on the parallel layout.
export CUDA_VISIBLE_DEVICES="0,1,2,3"
PARALLEL_ARGS=(--tensor-parallel-size 4)

fetch_metric() {
local port="$1" name="$2"
curl -s "http://localhost:${port}/metrics" \
| grep -oP "${name}\\{[^}]*\\} \\K[0-9.]+" || echo "0"
}

SERVER_PID=""
_descendants() {
local pid="$1" child
for child in $(pgrep -P "$pid" 2>/dev/null || true); do
echo "$child"
_descendants "$child"
done
}
cleanup_server() {
if [[ -n "$SERVER_PID" ]]; then
local descendants
descendants=$(_descendants "$SERVER_PID")
kill "$SERVER_PID" 2>/dev/null || true
wait "$SERVER_PID" 2>/dev/null || true
local pid
for pid in $descendants; do
kill -9 "$pid" 2>/dev/null || true
done
local waited=0
while [[ $waited -lt 120 ]]; do
local used
used=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits 2>/dev/null | sort -rn | head -1)
if [[ -z "$used" || "$used" -lt 2000 ]]; then break; fi
sleep 3; waited=$((waited + 3))
done
SERVER_PID=""
fi
}
trap 'cleanup_server' EXIT

declare -A AL_RESULT

run_cell() {
local mode="$1" mtp="$2"
local think_args=()
local temperature=1.0
if [[ -n "$CHAT_TEMPLATE_KWARGS_ON" && "$CHAT_TEMPLATE_KWARGS_ON" != "{}" ]]; then
think_args=(--chat-template-kwargs "$CHAT_TEMPLATE_KWARGS_ON")
fi

echo ""
echo "=========================================="
echo " Cell: thinking=$mode DSPARK=$mtp category=$CATEGORY"
echo "=========================================="

local serve_args=(
--host 0.0.0.0 --port "$PORT"
"${PARALLEL_ARGS[@]}"
--trust-remote-code
--language-model-only
--enable-prefix-caching
--block-size 128
--kv-cache-dtype nvfp4
--attention-config '{"indexer_kv_dtype":"nvfp4","use_prefill_query_quantization":true}'
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}'
--disable-hybrid-kv-cache-manager
--reasoning-parser minimax_m3
--tool-call-parser minimax_m3
--enable-auto-tool-choice
--gpu-memory-utilization "$GPU_MEM_UTIL"
--max-num-seqs "$MAX_NUM_SEQS"
--max-model-len "$MAX_MODEL_LEN"
--speculative-config "{\"method\": \"dspark\", \"num_speculative_tokens\": $mtp, \"attention_backend\": \"FLASH_ATTN\", \"kv_cache_dtype\": \"fp8\", \"draft_sample_method\": \"probabilistic\", \"rejection_sample_method\": \"block\"}"
)

local server_log="$RESULTS_DIR/server_${mode}_mtp${mtp}.log"
vllm serve "$SERVE_MODEL" "${serve_args[@]}" > "$server_log" 2>&1 &
SERVER_PID=$!

if ! PYTHONSAFEPATH=1 PYTHONPATH="$repo_root${PYTHONPATH:+:$PYTHONPATH}" python3 -m infx.bench wait \
--url "http://0.0.0.0:${PORT}/health" --pid "$SERVER_PID" --log "$server_log"; then
echo " -> server failed to start (thinking=$mode dspark=$mtp), recording N/A"
AL_RESULT["${mode}_${mtp}"]="N/A"
cleanup_server
return
fi

local acc_before drf_before acc_after drf_after
acc_before=$(fetch_metric "$PORT" "vllm:spec_decode_num_accepted_tokens_total")
drf_before=$(fetch_metric "$PORT" "vllm:spec_decode_num_drafts_total")

python3 "$BENCH_DRIVER" \
--model "$SERVE_MODEL" \
--port "$PORT" \
--dataset-name speed_bench \
--dataset-path "$SPEEDBENCH_DIR" \
--speed-bench-category "$CATEGORY" \
--speed-bench-output-len "$SPEEDBENCH_OUTPUT_LEN" \
--num-prompts -1 \
--max-concurrency "$CONCURRENCY" \
--save-result \
--save-detailed \
--result-dir "$RESULTS_DIR" \
--result-filename "speedbench_${mode}_mtp${mtp}" \
--trust-remote-code \
--temperature "$temperature" \
--top-p "$TOP_P" \
"${think_args[@]}"
local bench_rc=$?
if [[ $bench_rc -ne 0 ]]; then
echo " -> benchmark client exited rc=$bench_rc (thinking=$mode dspark=$mtp); cell will be N/A"
fi
Comment on lines +235 to +238

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) If the benchmark client fails partway, the collector still records an AL from partial data and the job passes, even though the log says the cell will be N/A. After bench_rc is non-zero, run_cell still reads the counter deltas. It stores al whenever delta_drf > 0, so a crash after some drafts writes a real-looking value into the golden YAML. Fix: when bench_rc is non-zero, set AL_RESULT[${mode}_${mtp}] to N/A, call cleanup_server and return before computing deltas, so the log message and the matrix agree.

Why this was flagged

Trigger: the client python3 "$BENCH_DRIVER" exits non-zero midway, for example from request timeouts, OOM or a server crash after a few batches. The script prints "cell will be N/A" at line 237 but does not return. Lines 240-249 still compute AL from whatever drafts the server counted, and line 251 stores it. The MEASURED check at the end of the file passes because that cell is not N/A. The emitted OUT_YAML then holds an AL computed from a subset of the 80 SPEED-Bench prompts, with no marker that the data is partial. That value can get committed as a golden curve, which AgentX synthetic acceptance consumes. The Kimi collectors copy the same logic, so this new file brings the defect back rather than fixing it. Nothing guards against it: N/A is only assigned when the server fails to start or when delta_drf is 0.

Verification: Line 235 sets local bench_rc=$?, and lines 236-238 only echo " -> benchmark client exited rc=$bench_rc ...; cell will be N/A". Nothing after that returns or sets N/A. Line 246 checks only if [[ "$delta_drf" -gt 0 ]], and line 252 stores AL_RESULT["${mode}_${mtp}"]="$al". The final guard at lines 293-303 fails the job only when every cell is N/A.


acc_after=$(fetch_metric "$PORT" "vllm:spec_decode_num_accepted_tokens_total")
drf_after=$(fetch_metric "$PORT" "vllm:spec_decode_num_drafts_total")

local delta_acc delta_drf al
delta_acc=$(awk "BEGIN {printf \"%d\", $acc_after - $acc_before}")
delta_drf=$(awk "BEGIN {printf \"%d\", $drf_after - $drf_before}")
if [[ "$delta_drf" -gt 0 ]]; then
al=$(awk "BEGIN {printf \"%.2f\", 1 + ($delta_acc / $delta_drf)}")
else
al="N/A"
fi
echo " -> thinking=$mode DSPARK=$mtp AL=$al (accepted=$delta_acc drafts=$delta_drf)"
AL_RESULT["${mode}_${mtp}"]="$al"

cleanup_server
}

for mode in $THINKING_MODES; do
for mtp in $MTP_LIST; do
run_cell "$mode" "$mtp"
done
done

emit_mode_block() {
local mode="$1"
for mtp in $MTP_LIST; do
echo " $mtp: ${AL_RESULT[${mode}_${mtp}]:-N/A}"
done
}

{
echo "# Acceptance Length (AL) reference values measured with SPEED-Bench."
echo "# dataset: $CATEGORY | top_p: $TOP_P | output_len: $SPEEDBENCH_OUTPUT_LEN"
echo "# thinking_on (adaptive): temperature=1.0, chat_template_kwargs: $CHAT_TEMPLATE_KWARGS_ON"
echo "# Measured on $MODEL_KEY (TP4, vLLM DSpark built-in drafter, probabilistic drafting, block rejection), per num_speculative_tokens."
echo "# Auto-generated by benchmarks/single_node/speedbench/minimaxm31flash_fp4_b300_vllm.sh (speedbench-al.yml)."
echo "#"
echo "# key = num_speculative_tokens (DSpark level); value = golden AL"
echo "${MODEL_KEY}:"
if [[ " $THINKING_MODES " == *" on "* ]]; then
echo " thinking_on:"
emit_mode_block on
fi
} > "$OUT_YAML"

echo ""
echo "=========================================="
echo " SPEED-Bench AL matrix written to: $OUT_YAML"
echo "=========================================="
cat "$OUT_YAML"

# A matrix where every cell is N/A is a failed collection, not a result: fail the
# job so it is not mistaken for a curve worth reviewing.
MEASURED=0
for mode in $THINKING_MODES; do
for mtp in $MTP_LIST; do
[[ "${AL_RESULT[${mode}_${mtp}]:-N/A}" != "N/A" ]] && MEASURED=$((MEASURED + 1))
done
done
if [[ "$MEASURED" -eq 0 ]]; then
echo "CRITICAL: no cell produced an AL value — see the server logs and the"
echo "benchmark client output above."
exit 1
fi
2 changes: 2 additions & 0 deletions inferencex-e2e/configs/runners.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -449,6 +449,7 @@ clusters:
available-cpu-dram-mib: 2_047_536
arch: x86_64
models:
download-root: writable-models
entries:
DeepSeek-R1-0528: {root: scratch, dir: DeepSeek-R1-0528}
DeepSeek-R1-0528-NVFP4-v2: {root: scratch, dir: DeepSeek-R1-0528-NVFP4-v2}
Expand Down Expand Up @@ -479,6 +480,7 @@ clusters:
aiperf-cache: {path: /data/home/sa-shared/gharunners/aiperf-cache}
scratch: {path: /scratch/models, visibility: node-local}
data-models: {path: /data/models, visibility: shared}
writable-models: {path: /data/home/sa-shared/gharunners/models}
squash:
dir: /data/home/sa-shared/containers
visibility: shared
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Source GitHub Actions run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/39895397914
# Acceptance Length (AL) reference values measured with SPEED-Bench.
# dataset: coding | top_p: 0.95 | output_len: 4096
# thinking_on (adaptive): temperature=1.0, chat_template_kwargs: {}
# Measured on minimax-m3.1-flash (B200 TP4, vLLM DSpark built-in drafter, probabilistic drafting, block rejection), per num_speculative_tokens.
# Auto-generated by benchmarks/single_node/speedbench/minimaxm31flash_fp4_b300_vllm.sh (speedbench-al.yml).
#
# key = num_speculative_tokens (DSpark level); value = golden AL
minimax-m3.1-flash:
thinking_on:
1: 1.82
2: 2.44
3: 2.88
4: 3.19
5: 3.46
6: 3.81
7: 4.04
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ def test_every_committed_curve_is_valid() -> None:
("dsv4dsparkprob", {"method": "dspark"}, "dsv4-pro-0813-dspark"),
("dsv41flash", {"method": "dspark"}, "dsv41flash_dspark"),
("musespark", {"method": "dflash"}, "musespark_dflash"),
("minimaxm31flash", {"method": "dspark"}, "minimaxm31flash_dspark"),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 nit (optional): the new test row adds no coverage and breaks the repo's test rules. It hits the default f"{model}_{method}" branch of curve_name (curves.py:47), which the dsv41flash and musespark rows already cover. Its is_file() assertion only checks that a committed data file exists. Fix: drop the minimaxm31flash row. test_every_committed_curve_is_valid already loads and range-checks the new YAML.

Why this was flagged

curve_name in infx/golden_al_distribution/curves.py:41-62 has special cases only for dsv4*, minimaxm3 and kimik3. For minimaxm31flash + dspark it returns the plain f"{model}_{method}" from line 47. The existing rows ("dsv41flash", {"method": "dspark"}) and ("musespark", {"method": "dflash"}) already reach that branch. AGENTS.md "Test quality" item 9 forbids tests that duplicate a covered path. Item 4 forbids pinning checked-in data, and (GOLDEN_DIR / f"{expected}.yaml").is_file() does exactly that. The new curve is already validated through real code by test_every_committed_curve_is_valid, which globs every YAML in GOLDEN_DIR. So the row only adds a test that fails when someone renames a data file, not when behavior regresses.

Verification: nit: the new parametrize row (test_golden_al_distribution.py:42) adds no branch coverage. In curves.py curve_name, minimaxm31flash with dspark skips every special case and returns the plain curve = f"{model}_{method}" (line 47), which the dsv41flash and musespark rows already reach. test_every_committed_curve_is_valid already range-checks the new YAML.

("kimik3", {"method": "dspark", "draft_sample_method": "greedy"}, "kimik3_dspark"),
(
"kimik3",
Expand Down
Loading