Repository navigation
[NVIDIA][AgentX] GB300 Muse Spark Open Dynamo+SGLang: retuned aggregated points and new TP8 disaggregated points / [NVIDIA][AgentX] GB300 Muse Spark Open Dynamo+SGLang:重新调优聚合配置点并新增 TP8 分离式配置点 - #4757
Conversation
|
Thanks for the contribution!
中文感谢你的贡献!
|
| PYTHONUNBUFFERED: "1" | ||
| SGLANG_TIMEOUT_KEEP_ALIVE: "900" | ||
| SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS: "1" | ||
| SGLANG_TRTLLM_MHA_DECODE_SEQ_LEN_SPLITS: "1" | ||
| SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION: "0" | ||
| SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS: "16" | ||
| SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS_MIN_BS: "4" |
There was a problem hiding this comment.
🔴 Muse Spark agentic runs could get turns rejected with HTTP 529 once a worker fills up. The base branch's Dynamo 1.6 did not have this backend admission check. The image's Dynamo 1.7 admits only a limited number of requests per engine, and almost every new point runs above max-running-requests. Fix: set DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT, DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT and DYN_BACKEND_ADMISSION_REQUEST_QUEUE_LIMIT to "10000" on every engine role, which covers the 3 sites listed. Same pattern at 3 sites (agg-variants.yaml:54, disagg-variants.yaml:55, disagg-variants.yaml:113).
Why this was flagged
The trigger is load above what one engine has room for, under the Dynamo 1.7.0.dev20261102 that the image ships. The base branch installed the 1.6.0.dev20261016 wheel, but dynamo.install: false at agg-variants.yaml:9 and disagg-variants.yaml:9 now uses the image's 1.7 build. Every other dynamo-sglang 1.7 key in the repo raises these three limits to "10000", in dsv41flash agg-variants.yaml:69-73 and minimaxm31flash sglang agg-variants.yaml:63-65. Their comment says "Dynamo 1.7 backend admission rejects turns with 529 once the engine is full". The new points routinely run beyond that capacity. TP4 c46/c50/c56 cap max-running-requests at 32. Disagg decode capacity is 4x56=224 at c672. Rejected turns would fail the AgentX run or skew its throughput and interactivity results compared with the 1.6 runs on the base branch. Neither recipe sets any DYN_BACKEND_ADMISSION_* variable.
Verification: The diff changes the base's wheel: "1.6.0.dev20261016" to dynamo.install: false (agg-variants.yaml:9; also disagg-variants.yaml:9). Neither musespark recipe sets DYN_BACKEND_ADMISSION_*. Agg c46/c50/c56 run against max-running-requests: 32 on a single TP4 engine.
There was a problem hiding this comment.
Not applicable to this image. Dynamo 1.7.0.dev20261102 includes ai-dynamo/dynamo#17481 (merged 2026-10-31), which made backend admission opt-in: with no engine limit set, the gate is bypassed (no queueing, no 529). Every run of these recipes logs Backend admission configuration engine_request_limit=unlimited engine_request_waiting_limit=unlimited request_queue_limit=0, and no 529s occurred. The dsv41flash / minimaxm31flash recipes set 10000 because they targeted earlier 1.7 builds, where admission was on by default. Keeping the recipes as measured.
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42374203350 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42374203350 |
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs ( This PR has no authorized |
…ted points and new TP8 disaggregated points Move the GB300 Muse Spark Open Dynamo + SGLang AgentX lane to lmsysorg/sglang:nightly-dev-cu13-20261103-797af717 (digest pinned) with its bundled ai-dynamo 1.7.0.dev20261102. Rename the aggregated key to musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg with TP4 c46/c50/c56 (HiCache 140 GB per rank) and TP8 c1/c36, add musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg with nine TP8 prefill / TP8 decode points (c32 to c672), and remove musespark-fp4-gb300-dynamo-sglang-agentic-dflash. 将 GB300 Muse Spark Open Dynamo + SGLang AgentX 迁移到 lmsysorg/sglang:nightly-dev-cu13-20261103-797af717(固定 digest),使用镜像自带的 ai-dynamo 1.7.0.dev20261102。聚合 key 改名为 -agg(TP4 c46/c50/c56,每 rank HiCache 140 GB;TP8 c1/c36),新增 -disagg(九个 TP8 prefill / TP8 decode 点, c32 至 c672),并移除原 key。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…G_TRTLLM_MHA_FULL_ATTN_KV_SPLITS Drop SGLANG_TRTLLM_MHA_DECODE_SEQ_LEN_SPLITS and SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS_MIN_BS from the aggregated and disaggregated recipes; both fall back to the SGLang defaults (1 and 12). 聚合与分离式配方移除 SGLANG_TRTLLM_MHA_DECODE_SEQ_LEN_SPLITS 与 SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS_MIN_BS,改用 SGLang 默认值(1 与 12)。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b37254d to
704ebb2
Compare
|
[by Claude Code] sgl-project/sglang#48392 has been merged. This PR's AgentX configs ( This PR has no authorized |
…X client to load its trace dataset The recipe-level HF_HUB_OFFLINE=1 is propagated to the benchmark step, so AIPerf cannot load semianalysisai/cc-traces-weka-062126 when its dataset cache misses (OfflineModeIsEnabled). Set HF_HUB_OFFLINE=0 in benchmark.env for the aggregated and disaggregated recipes; workers stay offline. 基准测试步骤会继承配方顶层的 HF_HUB_OFFLINE=1,数据集缓存未命中时 AIPerf 无法加载 semianalysisai/cc-traces-weka-062126(OfflineModeIsEnabled)。 聚合与分离式配方在 benchmark.env 设置 HF_HUB_OFFLINE=0,推理 worker 仍保持离线。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX configs ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#48634 has been merged. This PR's AgentX configs ( This PR has no authorized |
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX configs ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#48857 has been merged. This PR's AgentX configs ( This PR has no authorized |
|
[by Claude Code] I've started babysitting the pipeline of this PR. I'll keep monitoring it until it passes. If a job fails, I may rerun it when the cause is infrastructure, or push a fix or a merge of the latest |
|
[by Claude Code] Hi @cquil11, could you help with reusing this PR's sweep? Run Sweep 41854172505 on the current head The run is still marked Since To keep this from happening on future runs, #4868 makes Thank you! |
|
[by Claude Code] cc @functionstackx as well: could you take a look at the reuse question in my comment above, in case Cam is not available? Thank you! |
…ynamo-sglang-agentic-agg-disagg-1103 Conflict in inferencex-e2e/perf-changelog.yaml resolved by keeping main's entries and appending this PR's entry at the end. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[by Claude Code] We're starting a fresh full sweep for this PR instead of reusing run 41854172505. Why:
What changed on the branch: I merged the latest @cquil11 @functionstackx: no action needed on my earlier reuse question. Sorry for the noise, and thank you! |
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX configs ( This PR has no authorized |
|
/use 42374203350 |
|
@nvpohanh staged run 42374203350. This run remains available across future |
|
As a PR reviewer and CODEOWNER, I have reviewed this and have: Checklist items
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (In-PR sweep + evals): PASS — Head ✅ Check 2 (Eval accuracy): PASS — 13 gsm8k rows from run 42374203350 on ✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#49260 is MERGED (2026-11-09). The published Muse Spark Open cookbook AgentX data ( ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Checklist template): PASS — All 20 items in the current ✅ Check 6 (Upstream image / provenance / ordering): PASS — Both keys use upstream ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (No architecture hacks): PASS — There are no ✅ Check 9 (Chat template): PASS — The c1 raw artifact's ✅ Check 10 (No engine patches): PASS — The recipes have no setup script, patch, pip install, or source edits. The only patches applied are the existing srt-slurm orchestrator patches; the engine image is untouched. ✅ Check 11 (Golden AL): PASS — DFlash with 8 draft tokens gives 7 speculative tokens, and ➖ Check 12 (Append-only): N/A — The new changelog entry does not set ✅ Check 13 (Draft precision): PASS — Draft ✅ Check 14 (Pareto coverage): PASS — One curve (Muse Spark Open / agentic-coding / GB300 dynamo-sglang / FP4 / run 42374203350). From 14 raw ✅ Check 15 (PR description): PASS — The body's per-point settings (HiCache sizes, schedule policies, running-request caps, intervals, draft backends), disagg topologies and node counts, shared env, image and Dynamo version, and removed-point dominance claims all match the pinned recipes and run evidence. ✅ Check 16 (NVIDIA overview): PASS — Muse-Spark-Open has a populated SGLang cell in the NVIDIA overview Framework Recipes table. ➖ Check 17 (Metrics publication): N/A — Only ✅ Check 18 (Recipe/image/sweep consistency): PASS — (a) Both recipes resolve and nothing fails to generate. (b) Master workers, TP/EP and node math match every override. (c) The image digest resolves. (d)/(e) No new orchestration pins or dispatch. (f) The changelog description is accurate ( Assessed commit: |
functionstackx
left a comment
There was a problem hiding this comment.
Approved as requested by Oren in the linked thread.
中文
根据 Oren 在所链接讨论中的明确要求,提交批准。
Created with Perplexity
…ynamo-sglang-agentic-agg-disagg-1103 [skip-sweep] Conflict in inferencex-e2e/perf-changelog.yaml only: kept main's entries and re-appended this PR's entry at the end, with its pr-link set to #4757. Benchmark results are reused from the pinned sweep run 42374203350. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 15, rawBenchmarkRows 29, coveredByPerJobFiles 15, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 17147, gpuMetricSamplesNew 2350788, powerPublicationPoints 14, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
Summary
Moves the GB300 Muse Spark Open Dynamo + SGLang AgentX lane to
lmsysorg/sglang:nightly-dev-cu13-20261103-797af717(multiarch digest pinned) and runs the ai-dynamo1.7.0.dev20261102that the image ships (dynamo.install: false). It replaces the aggregated points under a new-aggkey, adds TP8 disaggregated points under a new-disaggkey, and removesmusespark-fp4-gb300-dynamo-sglang-agentic-dflash(#4344).musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg(new)musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg(new)SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS=16).SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=0on every role.AI model disclosure
PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
将 GB300 Muse Spark Open Dynamo + SGLang AgentX 配置迁移到
lmsysorg/sglang:nightly-dev-cu13-20261103-797af717(固定 multiarch digest),并直接使用镜像自带的 ai-dynamo1.7.0.dev20261102(dynamo.install: false)。本 PR 在新的-aggconfig key 下替换聚合配置点,在新的-disaggconfig key 下新增 TP8 分离式配置点,并移除musespark-fp4-gb300-dynamo-sglang-agentic-dflash(#4344)。musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg(新增):musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg(新增),prefill 与 decode worker 均为 TP8:SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS=16)。SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=0。AI 模型使用说明:Claude Opus 5.5;负责调优与配置修改。
🤖 Generated with Claude Code