Repository navigation
[AgentX] Retune Muse Spark Open GB200 SGLang per concurrency / [AgentX] 按并发重新调优 Muse Spark Open GB200 SGLang - #4670
Conversation
Retune concurrency 4, 10, 16 and 28, add concurrency 7, 14, 36 and 40, drop concurrency 2, 6, 8, 20, 24 and 32, and set SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=0 in the base environment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献!
|
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. The recipe and master config are both under NVIDIA code ownership, and the per-concurrency tuning values are benchmark choices a maintainer should sign off on, so a human look is still worthwhile.
What was reviewed:
- The
conc-listin nvidia-master.yaml (4, 7, 10, 14, 16, 28, 36, 40) matches theoverride_tp4_cNkeys in agentic.yaml. Each HiCache override now setsKV_OFFLOADING: dramand has a matchingCONC. - The new changelog entry matches the recipe: SPF everywhere except c7, admission of 2-6 at c4-16, and 16K chunked prefill at c28-40. It is appended at the tail with no historical bytes changed. The
pr-linkis still the usual XXX placeholder. - The c1 settings, the image and the DFlash2 draft settings are unchanged.
Extended reasoning...
The change retunes one benchmark config (musespark-fp4-gb200-sglang-agentic-dflash). It touches its single-node srt-slurm recipe, a single conc-list line in nvidia-master.yaml and an appended perf-changelog entry. It touches nothing security-sensitive. All three files fall under CODEOWNERS rules for NVIDIA recipes, the NVIDIA master config or the core team, so this shouldn't be auto-approved. The new per-point serving values are empirical tuning choices that only the benchmark results can confirm.
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX config ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#47464 has been merged. This PR's AgentX config ( This PR has no authorized |
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41096285400 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41096285400 |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
/use 41096285400 |
|
@nvpohanh staging run 41096285400. Existing staged runs will be preserved; a completion link will be posted here. |
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( The commands mirror the launch measured in run 41096285400, pinned by this PR's |
|
[by Claude Code] sgl-project/sglang#47929 has been merged. This PR's AgentX config ( The commands mirror the launch measured in run 41096285400, pinned by this PR's |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have: Checklist items
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Sweep on in-PR commit): PASS — Run Sweep 41096285400 ( ✅ Check 2 (Evals): PASS — ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#47929 is MERGED (2026-11-03T03:28:14Z) and the MuseSparkOpen cookbook page is live. Its ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Checklist template): PASS — all 20 items in the current ✅ Check 6 (Upstream image / provenance / ordering): PASS — the image is the unchanged upstream ✅ Check 7 (Deprecated models): PASS — the ✅ Check 8 (Architecture hacks): PASS — there are no ✅ Check 9 (Chat template): PASS — the AgentX replay drives the chat endpoint ( ✅ Check 10 (Engine patches): PASS — the diff is YAML only, and the effective recipe has no setup script, patch, ✅ Check 11 (Golden AL): PASS — every point's submission env has ➖ Check 12 (Append-only): N/A — the new changelog entry does not set ✅ Check 13 (Draft precision): PASS — the draft is ✅ Check 14 (Pareto coverage): PASS — there is one curve: musespark / agentic-coding / ✅ Check 15 (PR description): PASS — the per-point changes, removed points, unchanged image and checkpoint, and the base env addition all match the recipe at ✅ Check 16 (NVIDIA overview): PASS — Muse-Spark-Open has an SGLang checkmark in the Framework Recipes table of the NVIDIA overview. ➖ Check 17 (Metrics publication): N/A — this is a direct SGLang deployment with no Dynamo TRT-LLM serving path. ✅ Check 18 (Recipe/image/sweep consistency): PASS — no generation errors at head. The master entry (TP4/EP4, conc [1] plus HiCache [4, 7, 10, 14, 16, 28, 36, 40]) matches the recipe's overrides. No new images; no orchestration or dispatch changes. The changelog claims match the resolved recipe. Assessed commit: |
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( The commands mirror the launch measured in run 41096285400, pinned by this PR's |
|
[by Claude Code] sgl-project/sglang#48392 has been merged. This PR's AgentX config ( The commands mirror the launch measured in run 41096285400, pinned by this PR's |
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX config ( The commands mirror the launch measured in run 41096285400, pinned by this PR's |
|
[by Claude Code] sgl-project/sglang#48634 has been merged. This PR's AgentX config ( The commands mirror the launch measured in run 41096285400, pinned by this PR's |
# Conflicts: # inferencex-e2e/perf-changelog.yaml
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 10, rawBenchmarkRows 18, coveredByPerJobFiles 9, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 5276, gpuMetricSamplesNew 189824, powerPublicationPoints 9, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Summary
Retune the Muse Spark Open GB200 SGLang AgentX curve on
lmsysorg/sglang:nightly-dev-20261026-8252e3af(image unchanged) and setSGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=0in the base serving environment:HiCache concurrency 2, 6, 8, 20, 24 and 32 are removed. Every removed point that was on the previous GB200 frontier is beaten on both throughput per GPU and interactivity by a retained point (6 and 8 by concurrency 10, 20 and 24 by concurrency 28, 32 by concurrency 40); concurrency 2 was already dominated by concurrency 4.
Per-point changes:
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096,PYTORCH_CUDA_ALLOC_CONF=expandable_segments:TrueThe checkpoint, DFlash2 draft settings, the concurrency 1 server settings and the image are unchanged.
AI model disclosure
PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
在
lmsysorg/sglang:nightly-dev-20261026-8252e3af(镜像不变)上重新调优 Muse Spark Open GB200 SGLang AgentX 曲线,并在基础服务环境中设置SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=0:移除 HiCache 并发 2、6、8、20、24、32。原 GB200 前沿上被移除的点,都被保留的点在每 GPU 吞吐和交互性两个维度同时超过(6 和 8 被并发 10 超过,20 和 24 被并发 28 超过,32 被并发 40 超过);并发 2 原本已被并发 4 支配。
各点改动:并发 4 将 prefill-decode interval 由 128 提高到 256,并使用 shortest-prefill-first 与 stream interval 20;新增并发 7(请求准入 2、prefill-decode interval 64、stream interval 20);并发 10 与 16 将请求准入与 decode graph 批大小分别降到 3 和 6,prefill-decode interval 提高到 64,并使用 stream interval 20、4 个 tokenizer worker 和增量流式输出;新增并发 14(准入 5、interval 96);并发 28 使用显存占用比例 0.9、16384 token chunked prefill、shortest-prefill-first、interval 96、stream interval 20、
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096;新增并发 36(准入 72、interval 128);新增并发 40(准入 48、显存占用比例 0.93、HiCache 160 GB、interval 64)。模型检查点、DFlash2 草稿模型设置、并发 1 的服务端设置和镜像保持不变。AI 模型使用说明:模型为 claude-opus-5-5。claude-opus-5-5 负责指挥调优、分析实测结果、生成配方、主配置和性能变更记录的改动、在本地检查,以及起草本 PR 说明;ARIA 编排代理及其任务代理(gpt-6.1-sol,经 Codex)负责提交与收集性能和 GSM8K 测试。
🤖 Generated with Claude Code