Repository navigation
Commit 36585df
[NVIDIA][AgentX] Update GB300 DSV4.1-Flash disagg topology, caching and routing / [NVIDIA][AgentX] 更新 GB300 DSV4.1-Flash 分离式拓扑、缓存与路由 (#4303)
* feat(agentx): GB300 DSV4.1-Flash Dynamo+SGLang disagg on the 20261019 nightly with the Mooncake linker
Move dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg to
lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089 (multiarch digest
pinned) and the ai-dynamo 1.6.0.dev20261018 it ships, with the ZMQ event
plane and no NATS service.
- c1228 keeps 4x DEP2 prefill + 1x DEP16 decode; c2884 moves from 6 to 5
DEP2 prefill workers (3 prefill nodes, 26 GPUs) and caps prefill batches
at 32768 tokens.
- Both roles use the flashmla_mega attention backend, local DP control
broadcast and the Mooncake external cache linker with a Mooncake master
service (high watermark 0.90, client TTL 60 s, 64 RPC threads). The
linker replaces prefill HiCache, so the master rows declare Mooncake
0.3.13.post1 as the KV offload backend.
- Decode adds late admission, stream interval 32, polling interval 8,
2048 SWA prefix tails and decoder SWA bounded replay; the NCCL DP-batch
sync env is dropped because the image exchanges MLP sync rows with gloo
gather + broadcast (sgl-project/sglang#44622).
- A setup script applies four SGLang patches the image lacks, with git
apply under a lock, idempotently, failing on any hunk that does not
apply: sgl-project/sglang#44623 (open), #44580 (merged as 834bfe4ef0),
#45062 (merged as 06342acad2) and decode late admission
(weireweire/sglang@407e912a1e).
- Revise this PR's changelog entry to the final state.
中文:将 dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg 迁移到
lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089(固定 multiarch
digest)及镜像自带的 ai-dynamo 1.6.0.dev20261018,使用 ZMQ event plane,
不再启动 NATS 服务。
- c1228 保持 4 个 DEP2 prefill + 1 个 DEP16 decode;c2884 的 DEP2 prefill
由 6 个减为 5 个(3 个 prefill 节点,共 26 个 GPU),并将 prefill 批次
上限设为 32768 token。
- 两个角色均使用 flashmla_mega attention 后端、本地 DP control broadcast
和 Mooncake external cache linker,并启动 Mooncake master 服务(高水位
0.90,client TTL 60 秒,64 个 RPC 线程)。linker 取代了 prefill
HiCache,因此 master 配置将 KV offload 后端声明为 Mooncake 0.3.13.post1。
- Decode 新增 late admission、stream interval 32、polling interval 8、
2048 SWA prefix tails 与 decoder SWA bounded replay;镜像已通过 gloo
gather + broadcast 交换 MLP 同步数据(sgl-project/sglang#44622),因此
删除 NCCL DP batch 同步环境变量。
- 新增 setup 脚本,在锁保护下用 git apply 幂等地应用镜像缺少的四个
SGLang 补丁,任一 hunk 无法应用即失败:sgl-project/sglang#44623(未合并)、
#44580(已合并为 834bfe4ef0)、#45062(已合并为 06342acad2)以及 decode
late admission(weireweire/sglang@407e912a1e)。
- 将本 PR 的 changelog 条目更新为最终状态。
* fix(agentx): drop decode late admission from the GB300 DSV4.1-Flash disagg points
A same-time A/B on the 20261019 nightly showed no effect from decode late
admission: P90 interactivity 230/232 without it and 230/228 with it at
c1228. The image already exchanges the scheduler's MLP sync rows with gloo
gather + broadcast (sgl-project/sglang#44622) and avoids blocking Engram
history copies during PD admission (#44787). Remove the patch, its setup
script entry and SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION.
中文:在 20261019 nightly 上的同时段 A/B 显示 decode late admission 没有
效果:c1228 下关闭时 P90 interactivity 为 230/232,开启时为 230/228。镜像
已通过 gloo gather + broadcast 交换调度器的 MLP 同步数据
(sgl-project/sglang#44622),并避免了 PD admission 期间阻塞的 Engram
历史拷贝(#44787)。因此删除该补丁、setup 脚本中的对应条目以及
SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION。
* feat(agentx): move the GB300 DSV4.1-Flash disagg points to the 20261020 nightly
The 20261020 image (ab580b2e, ai-dynamo 1.6.0.dev20261019) contains
sgl-project/sglang#44580 and #45062, so the setup script now applies only
the external linker pools patch (#44623). Add the #4303 changelog entry on
top of the merged #4180 entry.
* feat(agentx): run GB300 DSV4.1-Flash disagg c1228 at 3P1D and add c4096 at 7P1D
With the Mooncake linker, three DEP2 prefill workers keep up at c1228
(364.5k/362.1k vs 336.7k tok/s/GPU at 4P1D, same interactivity). Add
c4096 with seven DEP2 prefill workers and 64 requests per decode rank
(630.1k tok/s/GPU, P90 137 tok/s/user). Both decode roles use a 140gb
Mooncake segment.
* perf(agentx): update DSV4.1 disagg to patch-free October 21 nightly
Root cause: the previous image needed an external-linker patch and the
submitted recipe omitted supported routing and worker settings used locally.
Fix: pin the October 21 public image, remove the linker patch and setup script,
enable per-role routing, total-token balancing, worker fastokens and BF16
decode index logits, and use eight DEP2 prefill workers at c4096 within the
same four prefill nodes. Update only the disaggregated matrix and changelog.
Validation: the repository matrix generator resolves all three points with
3600-second measurements and evaluation enabled. The exact new image/config
combination has not yet completed a performance sweep or native evaluation.
原因:旧镜像需要外部缓存 linker 补丁,提交配置也未包含本地使用的部分原生配置项。
修复:固定 10 月 21 日公开镜像,删除补丁,补齐路由、负载均衡、tokenizer 和
decode index logits 配置;c4096 在原有四个 prefill 节点使用八个 DEP2 worker。
验证:三个配置点均通过矩阵生成,保留 3600 秒测量和 eval;新版完整组合尚未实跑。
* config: remove unused HiCache write policy from DSV4.1 linker recipe
* fix: restore decode metrics in DSV4.1 disagg recipes
Root cause: prefill-only Prometheus counters were treated as deployment totals, preventing record-based token fallback and omitting decode KV metrics.
Fix: enable SGLang metrics on decode for all three disaggregated variants.
Validation: Ruff check and format checks passed; perf-changelog matrix validation passed. Full GPU sweep remains pending.
原因:仅 prefill 导出指标会使汇总缺少 decode 数据。修复:为三个 disagg 配置恢复 decode metrics。校验:Ruff 和 changelog 矩阵检查通过,GPU sweep 尚待验证。
* fix: backport KV-event port range correction for disagg runs
Root cause: the launcher can assign port 5252 to a decode publisher, colliding with node monitoring listeners and aborting performance/eval startup.
Fix: backport NVIDIA/srt-slurm#617 to allocate KV-event blocks from 3000. The SGLang image and serving parameters are unchanged. Validation: patch applies cleanly to the pinned srt-slurm revision; Ruff checks pass.
修复 KV-event 端口与节点监控监听端口冲突,回移 srt-slurm #617;保持镜像及模型参数不变。
* docs: keep the launcher port backport minimal
Retain only the port-base change and concise public descriptions. 简化端口补丁及说明。
* config: disable unused decode KV events and drop port backport
Round-robin decode routing does not consume KV events. Disable the decode publisher while retaining prefill events and metrics on both roles, and remove the srt-slurm #617 backport from this submission.
Validation: all three bound variants retain prefill events, omit decode events and retain metrics; Ruff and changelog checks pass. 关闭轮询解码侧的 KV-event 发布,保留预填充事件和双方指标,并移除端口补丁。
* ci: Include unchanged GB300 DSV4.1 aggregated points in sweep
将现有 TP4 c1/c4/c16 和 TP2 c16 纳入本 PR 的 sweep 与 eval 选点,覆盖低延迟区间;保留原有 agg 配置及镜像。矩阵校验生成 7 个性能点和 5 个 eval,Ruff 检查通过。
---------
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: Cam Quilici <cjquilici@gmail.com>1 parent 8dc0df1 commit 36585df
3 files changed
Lines changed: 122 additions & 32 deletions
File tree
- inferencex-e2e
- benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx
- configs
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
9 | | - | |
| 8 | + | |
| 9 | + | |
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| 13 | + | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
| |||
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
38 | 41 | | |
39 | 42 | | |
40 | 43 | | |
41 | 44 | | |
42 | 45 | | |
43 | 46 | | |
44 | 47 | | |
45 | | - | |
| 48 | + | |
| 49 | + | |
46 | 50 | | |
47 | 51 | | |
48 | 52 | | |
| |||
77 | 81 | | |
78 | 82 | | |
79 | 83 | | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
80 | 95 | | |
81 | 96 | | |
82 | 97 | | |
83 | 98 | | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
84 | 102 | | |
85 | 103 | | |
86 | 104 | | |
87 | 105 | | |
| 106 | + | |
88 | 107 | | |
89 | | - | |
| 108 | + | |
90 | 109 | | |
91 | 110 | | |
92 | 111 | | |
93 | 112 | | |
94 | 113 | | |
95 | | - | |
96 | | - | |
97 | | - | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
98 | 118 | | |
99 | 119 | | |
100 | 120 | | |
101 | 121 | | |
| 122 | + | |
102 | 123 | | |
103 | 124 | | |
104 | 125 | | |
105 | 126 | | |
106 | 127 | | |
107 | 128 | | |
108 | 129 | | |
109 | | - | |
110 | 130 | | |
111 | 131 | | |
112 | 132 | | |
| |||
120 | 140 | | |
121 | 141 | | |
122 | 142 | | |
123 | | - | |
124 | | - | |
125 | 143 | | |
126 | 144 | | |
| 145 | + | |
127 | 146 | | |
128 | 147 | | |
129 | 148 | | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
130 | 156 | | |
131 | 157 | | |
132 | 158 | | |
133 | 159 | | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
134 | 163 | | |
135 | 164 | | |
136 | 165 | | |
137 | 166 | | |
| 167 | + | |
138 | 168 | | |
139 | | - | |
| 169 | + | |
140 | 170 | | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
141 | 177 | | |
142 | 178 | | |
143 | 179 | | |
| 180 | + | |
144 | 181 | | |
145 | 182 | | |
146 | 183 | | |
147 | 184 | | |
148 | 185 | | |
149 | 186 | | |
150 | 187 | | |
| 188 | + | |
151 | 189 | | |
152 | 190 | | |
153 | 191 | | |
| |||
167 | 205 | | |
168 | 206 | | |
169 | 207 | | |
170 | | - | |
171 | | - | |
| 208 | + | |
| 209 | + | |
172 | 210 | | |
173 | 211 | | |
| 212 | + | |
174 | 213 | | |
175 | | - | |
176 | | - | |
177 | | - | |
| 214 | + | |
178 | 215 | | |
| 216 | + | |
| 217 | + | |
179 | 218 | | |
180 | 219 | | |
181 | 220 | | |
182 | 221 | | |
183 | 222 | | |
184 | | - | |
185 | | - | |
| 223 | + | |
| 224 | + | |
186 | 225 | | |
187 | 226 | | |
| 227 | + | |
188 | 228 | | |
189 | | - | |
| 229 | + | |
190 | 230 | | |
191 | | - | |
| 231 | + | |
192 | 232 | | |
| 233 | + | |
| 234 | + | |
193 | 235 | | |
194 | 236 | | |
195 | 237 | | |
196 | 238 | | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9322 | 9322 | | |
9323 | 9323 | | |
9324 | 9324 | | |
9325 | | - | |
| 9325 | + | |
9326 | 9326 | | |
9327 | 9327 | | |
9328 | 9328 | | |
9329 | 9329 | | |
9330 | 9330 | | |
9331 | | - | |
| 9331 | + | |
9332 | 9332 | | |
9333 | 9333 | | |
9334 | 9334 | | |
| |||
9339 | 9339 | | |
9340 | 9340 | | |
9341 | 9341 | | |
9342 | | - | |
| 9342 | + | |
9343 | 9343 | | |
9344 | | - | |
| 9344 | + | |
9345 | 9345 | | |
9346 | | - | |
| 9346 | + | |
9347 | 9347 | | |
9348 | 9348 | | |
9349 | 9349 | | |
| |||
9354 | 9354 | | |
9355 | 9355 | | |
9356 | 9356 | | |
9357 | | - | |
| 9357 | + | |
9358 | 9358 | | |
9359 | | - | |
| 9359 | + | |
9360 | 9360 | | |
9361 | | - | |
| 9361 | + | |
| 9362 | + | |
| 9363 | + | |
| 9364 | + | |
| 9365 | + | |
| 9366 | + | |
| 9367 | + | |
| 9368 | + | |
| 9369 | + | |
| 9370 | + | |
| 9371 | + | |
| 9372 | + | |
| 9373 | + | |
| 9374 | + | |
| 9375 | + | |
| 9376 | + | |
9362 | 9377 | | |
9363 | 9378 | | |
9364 | 9379 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10806 | 10806 | | |
10807 | 10807 | | |
10808 | 10808 | | |
| 10809 | + | |
| 10810 | + | |
| 10811 | + | |
| 10812 | + | |
| 10813 | + | |
| 10814 | + | |
| 10815 | + | |
| 10816 | + | |
| 10817 | + | |
| 10818 | + | |
| 10819 | + | |
| 10820 | + | |
| 10821 | + | |
| 10822 | + | |
| 10823 | + | |
| 10824 | + | |
| 10825 | + | |
0 commit comments