Repository navigation
[AgentX] Retune Muse Spark Open B200 SGLang per concurrency / [AgentX] 按并发重新调优 Muse Spark Open B200 SGLang - #4707
Conversation
Move to lmsysorg/sglang:nightly-dev-cu13-20261030-302b78cc, retune TP8 concurrency 1, 4, 10 and 64 and TP4 concurrency 16 and 24, add TP4 concurrency 7 and 40, and drop TP8 concurrency 2, 6, 8 and 56 and TP4 concurrency 8 and 10. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献!
|
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. The NVIDIA master config and the b200 srt-slurm recipes have designated code owners, and the retuned values are benchmark-tuning choices, so a code owner should still take a look.
What was reviewed:
- Each conc-list entry in the master config (TP8 1/4/10, TP8 hicache 64, TP4 hicache 7/16/24/40) has a matching override_tp{N}_c{CONC}, and each override's benchmark.env.CONC matches its name. No orphan overrides are left.
- Several overrides now admit fewer requests than CONC (tp8_c10 admits 3, tp4_c16 admits 6, tp8_c64 admits 56). The PR description explains this as deliberate tuning, and cuda-graph-max-bs-decode matches max-running-requests in each case.
- The perf-changelog entry is appended only at the end of the file and names the config key, the scenario, the new image and the concurrencies added and dropped. The pull/XXX placeholder follows an existing pattern.
Extended reasoning...
The change switches the image for musespark-fp4-b200-sglang-agentic-dflash in inferencex-e2e/configs/nvidia-master.yaml, rewrites the per-concurrency overrides in the single-node b200-fp4-mtp/agentic.yaml recipe, and appends one perf-changelog entry. It touches no security-sensitive surface and contains no shell. The diff is consistent across all three files. I deferred instead of approving because .github/CODEOWNERS assigns both nvidia-master.yaml and the b200-* recipe directories to specific owners, and the tuning values can only be validated by a benchmark sweep.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41297696082 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41297696082 |
|
/use 41297696082 |
|
@nvpohanh staging run 41297696082. Existing staged runs will be preserved; a completion link will be posted here. |
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX config ( The commands mirror the launch measured in run 41297696082, pinned by this PR's |
|
[by Claude Code] sgl-project/sglang#47929 has been merged. This PR's AgentX config ( The commands mirror the launch measured in run 41297696082, pinned by this PR's |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have: Checklist items
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Sweep + evals on in-PR commit): PASS — PR head ✅ Check 2 (Evals accuracy): PASS — ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#47929 MERGED 2026-11-03T03:28:14Z; ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest checklist template): PASS — all 20 items of the current ✅ Check 6 (Upstream images / main-branch / engine-first): PASS — ✅ Check 7 (Deprecated models/scenarios): PASS — ✅ Check 8 (No architecture hacks): PASS — no ✅ Check 9 (Spec-decode chat templates): PASS — AgentX replays through AIPerf chat completions ( ✅ Check 10 (No engine patches): PASS — diff is YAML-only; the effective recipe has no setup script, patch, ✅ Check 11 (Golden AL): PASS — harness injects ➖ Check 12 (Append-only): N/A — the new changelog entry does not set ✅ Check 13 (Draft precision): PASS — draft ✅ Check 14 (Pareto coverage): PASS — one curve (Muse-Spark-Open · agentic-coding · b200_sglang · FP4, run 41297696082 ✅ Check 15 (PR description): PASS — per-point changes, added/removed points, image, and removal-dominance claims match the pinned recipe, master config, and measured results. ✅ Check 16 (NVIDIA overview): PASS — Muse-Spark-Open has a populated SGLang cell in the Framework Recipes table of the NVIDIA overview. ➖ Check 17 (Metrics publication): N/A — direct SGLang deployment; no Dynamo TRT-LLM serving path. ✅ Check 18 (Recipe/image/sweep consistency): PASS — all 8 points generate at head; master TP/EP/conc lists match recipe overrides (TP4 Assessed commit: |
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( The commands mirror the launch measured in run 41297696082, pinned by this PR's |
|
[by Claude Code] sgl-project/sglang#48392 has been merged. This PR's AgentX config ( The commands mirror the launch measured in run 41297696082, pinned by this PR's |
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( The commands mirror the launch measured in run 41297696082, pinned by this PR's |
|
[by Claude Code] sgl-project/sglang#48634 has been merged. This PR's AgentX config ( The commands mirror the launch measured in run 41297696082, pinned by this PR's |
# Conflicts: # inferencex-e2e/perf-changelog.yaml
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 8, rawBenchmarkRows 8, coveredByPerJobFiles 0, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 7914, gpuMetricSamplesNew 250948, powerPublicationPoints 8, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Summary
Move the Muse Spark Open B200 SGLang AgentX curve to
lmsysorg/sglang:nightly-dev-cu13-20261030-302b78ccand retune it per concurrency:TP8 concurrency 2, 6, 8 and 56 and TP4 concurrency 8 and 10 are removed. Each removed point is covered by the new curve on throughput per GPU and mean interactivity: TP8 concurrency 2 is beaten by TP8 concurrency 4, TP8 concurrency 6 and 8 by TP8 concurrency 10, TP8 concurrency 56 lies below the line between TP4 concurrency 40 and TP8 concurrency 64, and TP4 concurrency 8 and 10 lie below the line between TP4 concurrency 7 and 16.
Per-point changes:
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096,PYTORCH_CUDA_ALLOC_CONF=expandable_segments:TrueThe checkpoint and the DFlash2 draft settings are unchanged.
AI model disclosure
PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
摘要
将 Muse Spark Open B200 SGLang AgentX 曲线切换到
lmsysorg/sglang:nightly-dev-cu13-20261030-302b78cc,并按并发重新调优:移除 TP8 并发 2、6、8、56 与 TP4 并发 8、10。每个被移除的点在每 GPU 吞吐与平均交互性上都被新曲线覆盖:TP8 并发 2 被 TP8 并发 4 超越,TP8 并发 6、8 被 TP8 并发 10 超越,TP8 并发 56 位于 TP4 并发 40 与 TP8 并发 64 的连线下方,TP4 并发 8、10 位于 TP4 并发 7 与 16 的连线下方。
各点改动:
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096、PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Truecheckpoint 与 DFlash2 draft 设置不变。
🤖 Generated with Claude Code