Skip to content

Commit 5e70328

Browse files
[AgentX] GB300 DSV4.1 Flash SGLang: fold DSpark sampling, retune c64/c128 / [AgentX] GB300 DSV4.1 Flash SGLang:折叠 DSpark 采样并重调 c64/c128 (#3782)
* perf(agentx): fold DSpark sampling and retune c64/c128 for GB300 DSV4.1 Flash SGLang Bump the image to nightly-dev-cu13-20260929-79cafec0 and set SGLANG_DSPARK_FOLDED_SAMPLING=2 on every point: the AUTO default keeps DSpark draft sampling eager for DeepSeek-V4.1 TP shards, which left low-concurrency decode launch-bound. Run TP4 c64 with 8K chunks and a 16-step prefill-decode interval, and let TP2 c128 admit all 128 sessions with a 4-step interval and mem-fraction-static 0.85. 中文:将镜像升级到 nightly-dev-cu13-20260929-79cafec0,并在所有点上设置 SGLANG_DSPARK_FOLDED_SAMPLING=2:AUTO 默认值对 DeepSeek-V4.1 的 TP 分片 保持 DSpark draft 采样为 eager 执行,导致低并发 decode 受 kernel 发射限制。 TP4 c64 改为 8K prefill chunk 与 16 步 prefill-decode interval;TP2 c128 接纳全部 128 个会话,使用 4 步 interval 与 mem-fraction-static 0.85。 * chore(changelog): link the GB300 DSV4.1 Flash SGLang entry to #3782 中文:将 GB300 DSV4.1 Flash SGLang 的 changelog 条目链接到 #3782。 * chore: refresh PR #3782 for sweep reuse [skip-sweep] --------- Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com> Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
1 parent b9cb358 commit 5e70328

3 files changed

Lines changed: 31 additions & 10 deletions

File tree

‎inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4-mtp/agentic.yaml‎

Lines changed: 19 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ base:
55
name: dsv41flash-fp4-gb300-sglang-agentic
66
model:
77
path: hf:deepseek-ai/DeepSeek-V4.1-Flash
8-
container: lmsysorg/sglang:dev-cu13-nightly-0924@sha256:d2c9929f8cc9889326203f6cbc46da48b69de42891b167592c9c76cc803d246e
8+
container: lmsysorg/sglang:nightly-dev-cu13-20260929-79cafec0@sha256:0f075735a6bf913a7cc1cd7ece4f526ae8af70fc5b88fdc02d9efe92a61cf3ae
99
precision: fp4
1010
resources:
1111
gpu_type: gb300
@@ -47,6 +47,10 @@ base:
4747
SGLANG_DEFAULT_THINKING: '1'
4848
SGLANG_DSV41_REASONING_EFFORT: high
4949
SGLANG_DSPARK_OPT_MARKOV_W2_BF16: 'True'
50+
# Fold DSpark draft sampling into the draft CUDA graph. AUTO leaves it eager
51+
# for V4.1 TP shards, which left c1 decode launch-bound (~6 ms GPU idle per
52+
# step in nsys at TP4 c1).
53+
SGLANG_DSPARK_FOLDED_SAMPLING: '2'
5054
# Cold weight loads from the shared HF cache plus graph capture outlast
5155
# srt-slurm's 1800 s default, as they did the legacy one-hour deadline.
5256
health_check:
@@ -63,11 +67,12 @@ base:
6367

6468
# One variant per point. Admission is 2x CONC, capped at the graph batch. SWA
6569
# prefix tails are 64x CONC up to 4096 from c2. TP2 keeps row-sharded Engram
66-
# tables in host DRAM and doubles the prefill chunk from c64. TP4 uses static
67-
# ragged verify; pure TP4 c1/c2 keep Engram in HBM, while TP4/EP4 from c4 keeps
68-
# it in host DRAM with 16K chunks, no prefill-decode interval, 4096 SWA tails
69-
# and a 128 graph batch, as matched one-hour c64 runs favored. Saturation
70-
# points get a longer warmup drain.
70+
# tables in host DRAM and doubles the prefill chunk from c64; TP2 c128 admits
71+
# 128 requests with a 4-step prefill-decode interval. TP4 uses static ragged
72+
# verify; pure TP4 c1/c2 keep Engram in HBM, while TP4/EP4 from c4 keeps it in
73+
# host DRAM with 4096 SWA tails and a 128 graph batch, 16K chunks and no
74+
# prefill-decode interval, except c64, which runs 8K chunks with a 16-step
75+
# interval. Saturation points get a longer warmup drain.
7176
override_tp2_c1:
7277
roles:
7378
agg:
@@ -202,7 +207,12 @@ override_tp2_c128:
202207
expert-parallel-size: 2
203208
chunked-prefill-size: 8192
204209
swa-prefix-tails: 4096
205-
max-running-requests: 64
210+
# Admitting 64 of 128 sessions with a 16-step interval collapsed this point
211+
# in the PR #3421 sweep; admitting all 128 with a 4-step interval holds it.
212+
prefill-decode-interval: 4
213+
mem-fraction-static: 0.85
214+
max-running-requests: 128
215+
cuda-graph-max-bs-decode: 128
206216
env:
207217
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE: '1'
208218
SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT: per_rank
@@ -337,8 +347,8 @@ override_tp4_c64:
337347
args:
338348
tensor-parallel-size: 4
339349
expert-parallel-size: 4
340-
chunked-prefill-size: 16384
341-
prefill-decode-interval: 0
350+
chunked-prefill-size: 8192
351+
prefill-decode-interval: 16
342352
swa-prefix-tails: 4096
343353
max-running-requests: 128
344354
cuda-graph-max-bs-decode: 128

‎inferencex-e2e/configs/nvidia-master.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9226,7 +9226,7 @@ dsv41flash-fp4-b300-sglang-agentic-dspark:
92269226

92279227
# Official SGLang nightly; TP4 C1/C2 use GPU Engram, other points use host DRAM.
92289228
dsv41flash-fp4-gb300-sglang-agentic-dspark:
9229-
image: lmsysorg/sglang:dev-cu13-nightly-0924@sha256:d2c9929f8cc9889326203f6cbc46da48b69de42891b167592c9c76cc803d246e
9229+
image: lmsysorg/sglang:nightly-dev-cu13-20260929-79cafec0@sha256:0f075735a6bf913a7cc1cd7ece4f526ae8af70fc5b88fdc02d9efe92a61cf3ae
92309230
model: deepseek-ai/DeepSeek-V4.1-Flash
92319231
model-prefix: dsv41flash
92329232
runner: cluster:gb300-nv

‎inferencex-e2e/perf-changelog.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9517,3 +9517,14 @@
95179517
- "For the TP arms, switch to EP1, use 16K prefill chunks, and set point-specific prefill-decode scheduling."
95189518
- "Add an aggregated DP4 C192 point with four internal TP4/EP4 data-parallel ranks behind sglang-router 0.3.2, using cache-aware routing and correlation-ID affinity."
95199519
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3744
9520+
9521+
- config-keys:
9522+
- dsv41flash-fp4-gb300-sglang-agentic-dspark
9523+
scenario-type:
9524+
- agentic-coding
9525+
description:
9526+
- "Bump the image from lmsysorg/sglang:dev-cu13-nightly-0924 to lmsysorg/sglang:nightly-dev-cu13-20260929-79cafec0 and update model.container to match. With the recipe otherwise unchanged, local GB300 runs on the new image reproduce the PR #3421 sweep (TP4 c1 439.8 vs 436 tok/s/user median, TP4 c32 249.5 vs 250)."
9527+
- "Set SGLANG_DSPARK_FOLDED_SAMPLING=2 on every point. The AUTO default keeps DSpark draft sampling eager for DeepSeek-V4.1 TP shards; nsys at TP4 c1 shows the decode step drop from 10.98 ms to 6.79 ms and GPU idle per step from 5.98 ms to 1.96 ms when it is folded into the draft graph. Local one-hour runs (synthetic acceptance 3.51): TP4 c1 5,845 -> 6,931 tok/s/GPU at 439.8 -> 649.8 tok/s/user median (386.8 -> 539.2 P90); TP4 c32 56,719 -> 57,704 at 249.5 -> 274.2 median; TP2 c64 147,452 -> 149,666 at 96.3 -> 99.4 median."
9528+
- "TP4 c64: 8K prefill chunks with a 16-step prefill-decode interval instead of 16K chunks with none. Local runs with the fold: 82,586 -> 85,093 tok/s/GPU at 86.8 -> 136.8 tok/s/user median (46.3 -> 72.1 P90); P90 TTFT 5.7 -> 12.5 s."
9529+
- "TP2 c128: admit all 128 sessions (max-running-requests and cuda-graph-max-bs-decode 64 -> 128) with a 4-step prefill-decode interval and mem-fraction-static 0.85. The PR #3421 sweep measured 18,631 tok/s/GPU at 130.9 tok/s/user median at this point; locally the new settings gave 147,763 tok/s/GPU at 37.9 tok/s/user median with a 51 s P90 TTFT (measured before the fold was added)."
9530+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3782

0 commit comments

Comments
 (0)