Skip to content

fix: avoid monitoring ports in KV-event allocation - #617

Merged
weireweire merged 3 commits into
NVIDIA:mainfrom
weireweire:fix/kv-event-port-range
Oct 21, 2026
Merged

weireweire merged 3 commits into
NVIDIA:mainfrom
weireweire:fix/kv-event-port-range

Conversation

@weireweire

@weireweire weireweire commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

A KV-event range starting at 5200 assigns port 5252 to a decode rank in a five-DEP2-prefill / DEP16-decode deployment. A node monitoring agent listening on loopback port 5252 then prevents SGLang's wildcard publisher from binding, aborting startup.

Move the KV-event base to 3000 while preserving per-DP-rank block allocation. Update the existing launch snapshots for the new ports.

Root cause: the 5200-based KV-event range reaches a loopback monitoring listener on port 5252 for a 5P DEP2 / DEP16 deployment, so SGLang cannot bind its wildcard publisher.

Fix: move the base to 3000, retaining DP block allocation and staying below the 9000-based ephemeral range. Add socket-binding regression coverage for DEP16 and DEP32.

Validation: Ruff checks, formatting and Linux-targeted type checking passed.
weireweire added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 21, 2026
Root cause: the launcher can assign port 5252 to a decode publisher, colliding with node monitoring listeners and aborting performance/eval startup.

Fix: backport NVIDIA/srt-slurm#617 to allocate KV-event blocks from 3000. The SGLang image and serving parameters are unchanged. Validation: patch applies cleanly to the pinned srt-slurm revision; Ruff checks pass.

修复 KV-event 端口与节点监控监听端口冲突,回移 srt-slurm #617;保持镜像及模型参数不变。
@weireweire
weireweire marked this pull request as ready for review October 21, 2026 09:55
@github-actions

Copy link
Copy Markdown

src/srtctl diff: +1 -1

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.47%. Comparing base (14c3630) to head (6f5e668).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #617      +/-   ##
==========================================
+ Coverage   85.45%   85.47%   +0.01%     
==========================================
  Files         165      165              
  Lines       24436    24457      +21     
==========================================
+ Hits        20883    20905      +22     
+ Misses       3553     3552       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@weireweire
weireweire merged commit 145bf2f into NVIDIA:main Oct 21, 2026
13 checks passed
cquil11 added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 22, 2026
…nd routing / [NVIDIA][AgentX] 更新 GB300 DSV4.1-Flash 分离式拓扑、缓存与路由 (#4303)

* feat(agentx): GB300 DSV4.1-Flash Dynamo+SGLang disagg on the 20261019 nightly with the Mooncake linker

Move dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg to
lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089 (multiarch digest
pinned) and the ai-dynamo 1.6.0.dev20261018 it ships, with the ZMQ event
plane and no NATS service.

- c1228 keeps 4x DEP2 prefill + 1x DEP16 decode; c2884 moves from 6 to 5
  DEP2 prefill workers (3 prefill nodes, 26 GPUs) and caps prefill batches
  at 32768 tokens.
- Both roles use the flashmla_mega attention backend, local DP control
  broadcast and the Mooncake external cache linker with a Mooncake master
  service (high watermark 0.90, client TTL 60 s, 64 RPC threads). The
  linker replaces prefill HiCache, so the master rows declare Mooncake
  0.3.13.post1 as the KV offload backend.
- Decode adds late admission, stream interval 32, polling interval 8,
  2048 SWA prefix tails and decoder SWA bounded replay; the NCCL DP-batch
  sync env is dropped because the image exchanges MLP sync rows with gloo
  gather + broadcast (sgl-project/sglang#44622).
- A setup script applies four SGLang patches the image lacks, with git
  apply under a lock, idempotently, failing on any hunk that does not
  apply: sgl-project/sglang#44623 (open), #44580 (merged as 834bfe4ef0),
  #45062 (merged as 06342acad2) and decode late admission
  (weireweire/sglang@407e912a1e).
- Revise this PR's changelog entry to the final state.

中文:将 dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg 迁移到
lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089(固定 multiarch
digest)及镜像自带的 ai-dynamo 1.6.0.dev20261018,使用 ZMQ event plane,
不再启动 NATS 服务。

- c1228 保持 4 个 DEP2 prefill + 1 个 DEP16 decode;c2884 的 DEP2 prefill
  由 6 个减为 5 个(3 个 prefill 节点,共 26 个 GPU),并将 prefill 批次
  上限设为 32768 token。
- 两个角色均使用 flashmla_mega attention 后端、本地 DP control broadcast
  和 Mooncake external cache linker,并启动 Mooncake master 服务(高水位
  0.90,client TTL 60 秒,64 个 RPC 线程)。linker 取代了 prefill
  HiCache,因此 master 配置将 KV offload 后端声明为 Mooncake 0.3.13.post1。
- Decode 新增 late admission、stream interval 32、polling interval 8、
  2048 SWA prefix tails 与 decoder SWA bounded replay;镜像已通过 gloo
  gather + broadcast 交换 MLP 同步数据(sgl-project/sglang#44622),因此
  删除 NCCL DP batch 同步环境变量。
- 新增 setup 脚本,在锁保护下用 git apply 幂等地应用镜像缺少的四个
  SGLang 补丁,任一 hunk 无法应用即失败:sgl-project/sglang#44623(未合并)、
  #44580(已合并为 834bfe4ef0)、#45062(已合并为 06342acad2)以及 decode
  late admission(weireweire/sglang@407e912a1e)。
- 将本 PR 的 changelog 条目更新为最终状态。

* fix(agentx): drop decode late admission from the GB300 DSV4.1-Flash disagg points

A same-time A/B on the 20261019 nightly showed no effect from decode late
admission: P90 interactivity 230/232 without it and 230/228 with it at
c1228. The image already exchanges the scheduler's MLP sync rows with gloo
gather + broadcast (sgl-project/sglang#44622) and avoids blocking Engram
history copies during PD admission (#44787). Remove the patch, its setup
script entry and SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION.

中文:在 20261019 nightly 上的同时段 A/B 显示 decode late admission 没有
效果:c1228 下关闭时 P90 interactivity 为 230/232,开启时为 230/228。镜像
已通过 gloo gather + broadcast 交换调度器的 MLP 同步数据
(sgl-project/sglang#44622),并避免了 PD admission 期间阻塞的 Engram
历史拷贝(#44787)。因此删除该补丁、setup 脚本中的对应条目以及
SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION。

* feat(agentx): move the GB300 DSV4.1-Flash disagg points to the 20261020 nightly

The 20261020 image (ab580b2e, ai-dynamo 1.6.0.dev20261019) contains
sgl-project/sglang#44580 and #45062, so the setup script now applies only
the external linker pools patch (#44623). Add the #4303 changelog entry on
top of the merged #4180 entry.

* feat(agentx): run GB300 DSV4.1-Flash disagg c1228 at 3P1D and add c4096 at 7P1D

With the Mooncake linker, three DEP2 prefill workers keep up at c1228
(364.5k/362.1k vs 336.7k tok/s/GPU at 4P1D, same interactivity). Add
c4096 with seven DEP2 prefill workers and 64 requests per decode rank
(630.1k tok/s/GPU, P90 137 tok/s/user). Both decode roles use a 140gb
Mooncake segment.

* perf(agentx): update DSV4.1 disagg to patch-free October 21 nightly

Root cause: the previous image needed an external-linker patch and the
submitted recipe omitted supported routing and worker settings used locally.

Fix: pin the October 21 public image, remove the linker patch and setup script,
enable per-role routing, total-token balancing, worker fastokens and BF16
decode index logits, and use eight DEP2 prefill workers at c4096 within the
same four prefill nodes. Update only the disaggregated matrix and changelog.

Validation: the repository matrix generator resolves all three points with
3600-second measurements and evaluation enabled. The exact new image/config
combination has not yet completed a performance sweep or native evaluation.

原因:旧镜像需要外部缓存 linker 补丁,提交配置也未包含本地使用的部分原生配置项。
修复:固定 10 月 21 日公开镜像,删除补丁,补齐路由、负载均衡、tokenizer 和
decode index logits 配置;c4096 在原有四个 prefill 节点使用八个 DEP2 worker。
验证:三个配置点均通过矩阵生成,保留 3600 秒测量和 eval;新版完整组合尚未实跑。

* config: remove unused HiCache write policy from DSV4.1 linker recipe

* fix: restore decode metrics in DSV4.1 disagg recipes

Root cause: prefill-only Prometheus counters were treated as deployment totals, preventing record-based token fallback and omitting decode KV metrics.

Fix: enable SGLang metrics on decode for all three disaggregated variants.

Validation: Ruff check and format checks passed; perf-changelog matrix validation passed. Full GPU sweep remains pending.

原因:仅 prefill 导出指标会使汇总缺少 decode 数据。修复:为三个 disagg 配置恢复 decode metrics。校验:Ruff 和 changelog 矩阵检查通过,GPU sweep 尚待验证。

* fix: backport KV-event port range correction for disagg runs

Root cause: the launcher can assign port 5252 to a decode publisher, colliding with node monitoring listeners and aborting performance/eval startup.

Fix: backport NVIDIA/srt-slurm#617 to allocate KV-event blocks from 3000. The SGLang image and serving parameters are unchanged. Validation: patch applies cleanly to the pinned srt-slurm revision; Ruff checks pass.

修复 KV-event 端口与节点监控监听端口冲突,回移 srt-slurm #617;保持镜像及模型参数不变。

* docs: keep the launcher port backport minimal

Retain only the port-base change and concise public descriptions. 简化端口补丁及说明。

* config: disable unused decode KV events and drop port backport

Round-robin decode routing does not consume KV events. Disable the decode publisher while retaining prefill events and metrics on both roles, and remove the srt-slurm #617 backport from this submission.

Validation: all three bound variants retain prefill events, omit decode events and retain metrics; Ruff and changelog checks pass. 关闭轮询解码侧的 KV-event 发布,保留预填充事件和双方指标,并移除端口补丁。

* ci: Include unchanged GB300 DSV4.1 aggregated points in sweep

将现有 TP4 c1/c4/c16 和 TP2 c16 纳入本 PR 的 sweep 与 eval 选点,覆盖低延迟区间;保留原有 agg 配置及镜像。矩阵校验生成 7 个性能点和 5 个 eval,Ruff 检查通过。

---------

Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: Cam Quilici <cjquilici@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants