Repository navigation
fix: avoid monitoring ports in KV-event allocation - #617
Merged
Merged
Conversation
Root cause: the 5200-based KV-event range reaches a loopback monitoring listener on port 5252 for a 5P DEP2 / DEP16 deployment, so SGLang cannot bind its wildcard publisher. Fix: move the base to 3000, retaining DP block allocation and staying below the 9000-based ephemeral range. Add socket-binding regression coverage for DEP16 and DEP32. Validation: Ruff checks, formatting and Linux-targeted type checking passed.
weireweire
added a commit
to SemiAnalysisAI/InferenceX
that referenced
this pull request
Oct 21, 2026
Root cause: the launcher can assign port 5252 to a decode publisher, colliding with node monitoring listeners and aborting performance/eval startup. Fix: backport NVIDIA/srt-slurm#617 to allocate KV-event blocks from 3000. The SGLang image and serving parameters are unchanged. Validation: patch applies cleanly to the pinned srt-slurm revision; Ruff checks pass. 修复 KV-event 端口与节点监控监听端口冲突,回移 srt-slurm #617;保持镜像及模型参数不变。
weireweire
marked this pull request as ready for review
October 21, 2026 09:55
weireweire
requested review from
alec-flowers,
csahithi,
ishandhanani and
nlevin-ui
as code owners
October 21, 2026 09:55
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #617 +/- ##
==========================================
+ Coverage 85.45% 85.47% +0.01%
==========================================
Files 165 165
Lines 24436 24457 +21
==========================================
+ Hits 20883 20905 +22
+ Misses 3553 3552 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
cquil11
added a commit
to SemiAnalysisAI/InferenceX
that referenced
this pull request
Oct 22, 2026
…nd routing / [NVIDIA][AgentX] 更新 GB300 DSV4.1-Flash 分离式拓扑、缓存与路由 (#4303) * feat(agentx): GB300 DSV4.1-Flash Dynamo+SGLang disagg on the 20261019 nightly with the Mooncake linker Move dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg to lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089 (multiarch digest pinned) and the ai-dynamo 1.6.0.dev20261018 it ships, with the ZMQ event plane and no NATS service. - c1228 keeps 4x DEP2 prefill + 1x DEP16 decode; c2884 moves from 6 to 5 DEP2 prefill workers (3 prefill nodes, 26 GPUs) and caps prefill batches at 32768 tokens. - Both roles use the flashmla_mega attention backend, local DP control broadcast and the Mooncake external cache linker with a Mooncake master service (high watermark 0.90, client TTL 60 s, 64 RPC threads). The linker replaces prefill HiCache, so the master rows declare Mooncake 0.3.13.post1 as the KV offload backend. - Decode adds late admission, stream interval 32, polling interval 8, 2048 SWA prefix tails and decoder SWA bounded replay; the NCCL DP-batch sync env is dropped because the image exchanges MLP sync rows with gloo gather + broadcast (sgl-project/sglang#44622). - A setup script applies four SGLang patches the image lacks, with git apply under a lock, idempotently, failing on any hunk that does not apply: sgl-project/sglang#44623 (open), #44580 (merged as 834bfe4ef0), #45062 (merged as 06342acad2) and decode late admission (weireweire/sglang@407e912a1e). - Revise this PR's changelog entry to the final state. 中文:将 dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg 迁移到 lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089(固定 multiarch digest)及镜像自带的 ai-dynamo 1.6.0.dev20261018,使用 ZMQ event plane, 不再启动 NATS 服务。 - c1228 保持 4 个 DEP2 prefill + 1 个 DEP16 decode;c2884 的 DEP2 prefill 由 6 个减为 5 个(3 个 prefill 节点,共 26 个 GPU),并将 prefill 批次 上限设为 32768 token。 - 两个角色均使用 flashmla_mega attention 后端、本地 DP control broadcast 和 Mooncake external cache linker,并启动 Mooncake master 服务(高水位 0.90,client TTL 60 秒,64 个 RPC 线程)。linker 取代了 prefill HiCache,因此 master 配置将 KV offload 后端声明为 Mooncake 0.3.13.post1。 - Decode 新增 late admission、stream interval 32、polling interval 8、 2048 SWA prefix tails 与 decoder SWA bounded replay;镜像已通过 gloo gather + broadcast 交换 MLP 同步数据(sgl-project/sglang#44622),因此 删除 NCCL DP batch 同步环境变量。 - 新增 setup 脚本,在锁保护下用 git apply 幂等地应用镜像缺少的四个 SGLang 补丁,任一 hunk 无法应用即失败:sgl-project/sglang#44623(未合并)、 #44580(已合并为 834bfe4ef0)、#45062(已合并为 06342acad2)以及 decode late admission(weireweire/sglang@407e912a1e)。 - 将本 PR 的 changelog 条目更新为最终状态。 * fix(agentx): drop decode late admission from the GB300 DSV4.1-Flash disagg points A same-time A/B on the 20261019 nightly showed no effect from decode late admission: P90 interactivity 230/232 without it and 230/228 with it at c1228. The image already exchanges the scheduler's MLP sync rows with gloo gather + broadcast (sgl-project/sglang#44622) and avoids blocking Engram history copies during PD admission (#44787). Remove the patch, its setup script entry and SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION. 中文:在 20261019 nightly 上的同时段 A/B 显示 decode late admission 没有 效果:c1228 下关闭时 P90 interactivity 为 230/232,开启时为 230/228。镜像 已通过 gloo gather + broadcast 交换调度器的 MLP 同步数据 (sgl-project/sglang#44622),并避免了 PD admission 期间阻塞的 Engram 历史拷贝(#44787)。因此删除该补丁、setup 脚本中的对应条目以及 SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION。 * feat(agentx): move the GB300 DSV4.1-Flash disagg points to the 20261020 nightly The 20261020 image (ab580b2e, ai-dynamo 1.6.0.dev20261019) contains sgl-project/sglang#44580 and #45062, so the setup script now applies only the external linker pools patch (#44623). Add the #4303 changelog entry on top of the merged #4180 entry. * feat(agentx): run GB300 DSV4.1-Flash disagg c1228 at 3P1D and add c4096 at 7P1D With the Mooncake linker, three DEP2 prefill workers keep up at c1228 (364.5k/362.1k vs 336.7k tok/s/GPU at 4P1D, same interactivity). Add c4096 with seven DEP2 prefill workers and 64 requests per decode rank (630.1k tok/s/GPU, P90 137 tok/s/user). Both decode roles use a 140gb Mooncake segment. * perf(agentx): update DSV4.1 disagg to patch-free October 21 nightly Root cause: the previous image needed an external-linker patch and the submitted recipe omitted supported routing and worker settings used locally. Fix: pin the October 21 public image, remove the linker patch and setup script, enable per-role routing, total-token balancing, worker fastokens and BF16 decode index logits, and use eight DEP2 prefill workers at c4096 within the same four prefill nodes. Update only the disaggregated matrix and changelog. Validation: the repository matrix generator resolves all three points with 3600-second measurements and evaluation enabled. The exact new image/config combination has not yet completed a performance sweep or native evaluation. 原因:旧镜像需要外部缓存 linker 补丁,提交配置也未包含本地使用的部分原生配置项。 修复:固定 10 月 21 日公开镜像,删除补丁,补齐路由、负载均衡、tokenizer 和 decode index logits 配置;c4096 在原有四个 prefill 节点使用八个 DEP2 worker。 验证:三个配置点均通过矩阵生成,保留 3600 秒测量和 eval;新版完整组合尚未实跑。 * config: remove unused HiCache write policy from DSV4.1 linker recipe * fix: restore decode metrics in DSV4.1 disagg recipes Root cause: prefill-only Prometheus counters were treated as deployment totals, preventing record-based token fallback and omitting decode KV metrics. Fix: enable SGLang metrics on decode for all three disaggregated variants. Validation: Ruff check and format checks passed; perf-changelog matrix validation passed. Full GPU sweep remains pending. 原因:仅 prefill 导出指标会使汇总缺少 decode 数据。修复:为三个 disagg 配置恢复 decode metrics。校验:Ruff 和 changelog 矩阵检查通过,GPU sweep 尚待验证。 * fix: backport KV-event port range correction for disagg runs Root cause: the launcher can assign port 5252 to a decode publisher, colliding with node monitoring listeners and aborting performance/eval startup. Fix: backport NVIDIA/srt-slurm#617 to allocate KV-event blocks from 3000. The SGLang image and serving parameters are unchanged. Validation: patch applies cleanly to the pinned srt-slurm revision; Ruff checks pass. 修复 KV-event 端口与节点监控监听端口冲突,回移 srt-slurm #617;保持镜像及模型参数不变。 * docs: keep the launcher port backport minimal Retain only the port-base change and concise public descriptions. 简化端口补丁及说明。 * config: disable unused decode KV events and drop port backport Round-robin decode routing does not consume KV events. Disable the decode publisher while retaining prefill events and metrics on both roles, and remove the srt-slurm #617 backport from this submission. Validation: all three bound variants retain prefill events, omit decode events and retain metrics; Ruff and changelog checks pass. 关闭轮询解码侧的 KV-event 发布,保留预填充事件和双方指标,并移除端口补丁。 * ci: Include unchanged GB300 DSV4.1 aggregated points in sweep 将现有 TP4 c1/c4/c16 和 TP2 c16 纳入本 PR 的 sweep 与 eval 选点,覆盖低延迟区间;保留原有 agg 配置及镜像。矩阵校验生成 7 个性能点和 5 个 eval,Ruff 检查通过。 --------- Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com> Co-authored-by: Cam Quilici <cjquilici@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A KV-event range starting at 5200 assigns port 5252 to a decode rank in a five-DEP2-prefill / DEP16-decode deployment. A node monitoring agent listening on loopback port 5252 then prevents SGLang's wildcard publisher from binding, aborting startup.
Move the KV-event base to 3000 while preserving per-DP-rank block allocation. Update the existing launch snapshots for the new ports.