Skip to content

[NV] DSR1 FP8 GB200 Dynamo TRT - #617

Merged
jthomson04 merged 9 commits into
mainfrom
nv/dsr1-fp8-gb200-dynamo-trt
Feb 3, 2026
Merged

jthomson04 merged 9 commits into
mainfrom
nv/dsr1-fp8-gb200-dynamo-trt

Conversation

@jthomson04

@jthomson04 jthomson04 commented Feb 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add DeepSeek R1 FP8 GB200 Dynamo TRT-LLM disaggregated multinode configurations using srt-slurm recipe-based workflow.

Changes

  • Add dsr1-fp8-gb200-dynamo-trt config to nvidia-master.yaml
  • Add GB200 runner support for dsr1 model prefix in launch_gb200-nv.sh
  • Add perf-changelog entry

Config Details

Image: nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.8.1.post2
Runner: GB200 NVL72 (multinode, disaggregated)
Model: DeepSeek-R1-0528 (FP8)

ISL/OSL Mode Scenarios Workers Concurrency Range
1k1k MTP (spec-decoding) 7 1P1D to 1P3D 9 - 4301
1k1k STP (no spec-decoding) 7 1P1D to 1P3D 3 - 6144
1k8k MTP 5 1P1D to 1P4D 4 - 8192
1k8k STP 5 1P1D to 1P4D 4 - 8192
8k1k MTP 7 1P1D to 5P1D 6 - 666
8k1k STP 7 1P1D to 5P1D 6 - 1229

Total: 38 scenarios

Architecture Variants

  • DP Attention mode: Disaggregated prefill with EP=TP (8/16/32)
  • TEP mode: Tensor Expert Parallelism with multiple decode workers (1P3D, 1P4D configurations)
  • MTP: Multi-Token Prediction (spec-decoding enabled)
  • STP: Single-Token Prediction (standard decoding)

Signed-off-by: jthomson04 <jothomson@nvidia.com>
Signed-off-by: jthomson04 <jothomson@nvidia.com>
@jthomson04
jthomson04 marked this pull request as ready for review February 2, 2026 20:09
@jthomson04
jthomson04 requested a review from a team as a code owner February 2, 2026 20:09
@functionstackx

Copy link
Copy Markdown
Collaborator

@claude please fix this, seems like typo in changelog.yaml

Traceback (most recent call last):
  File "/home/runner/work/InferenceMAX/InferenceMAX/utils/process_changelog.py", line 144, in <module>
    main()
  File "/home/runner/work/InferenceMAX/InferenceMAX/utils/process_changelog.py", line 73, in main
    changelog_data = yaml.safe_load(added_yaml)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 125, in safe_load
    return load(stream, SafeLoader)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/yaml/composer.py", line 110, in compose_sequence_node
    while not self.check_event(SequenceEndEvent):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
                         ^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/yaml/parser.py", line 392, in parse_block_sequence_entry
    raise ParserError("while parsing a block collection", self.marks[-1],
yaml.parser.ParserError: while parsing a block collection
  in "<unicode string>", line 3, column 3:
      - dsr1-fp8-gb200-dynamo-trt
      ^
expected <block end>, but found '?'
  in "<unicode string>", line 4, column 3:
      description:
      ^
Error: Process completed with exit code 1.

@Klaud-Cold

Klaud-Cold commented Feb 2, 2026 •

Copy link
Copy Markdown
Collaborator

Claude finished @functionstackx's task in 52s —— View job


Fixed YAML Syntax Error in perf-changelog.yaml ✓

  • Read perf-changelog.yaml to identify the YAML syntax error
  • Fix the YAML formatting issue
  • Commit and push the fix

Issue: Missing space after dash at line 329

- -config-keys:
-   - dsr1-fp8-gb200-dynamo-trt
+ - config-keys:
+     - dsr1-fp8-gb200-dynamo-trt

Fix pushed: 9cb5200

--- | Branch

Add missing space after dash in config-keys entry for dsr1-fp8-gb200-dynamo-trt.
'-config-keys:' should be '- config-keys:'

Co-authored-by: functionstackx <functionstackx@users.noreply.github.com>
@functionstackx

Copy link
Copy Markdown
Collaborator

@claude please fix this too in the nvidia-masteer.yaml. seems like just an simple typo of model prefix. change to just dsr1 instead of dsr1-fp8

error log

+ export CONFIG_FILE=recipes/trtllm/gb200-fp8/1k1k/mtp/ctx1_gen1_dep32_batch16_eplb0_mtp3_615.yaml
+ CONFIG_FILE=recipes/trtllm/gb200-fp8/1k1k/mtp/ctx1_gen1_dep32_batch16_eplb0_mtp3_615.yaml
+ export IS_MULTINODE=true
+ IS_MULTINODE=true
+ bash ./runners/launch_gb200-nv.sh
+ [[ dynamo-trt == \d\y\n\a\m\o\-\s\g\l\a\n\g ]]
Unsupported model prefix: dsr1-fp8. Supported prefixes are: gptoss or dsr1
+ [[ dynamo-trt == \d\y\n\a\m\o\-\t\r\t ]]
+ [[ dsr1-fp8 == \g\p\t\o\s\s ]]
+ [[ dsr1-fp8 == \d\s\r\1 ]]
+ echo 'Unsupported model prefix: dsr1-fp8. Supported prefixes are: gptoss or dsr1'
+ exit 1
Error: Process completed with exit code 1.

@Klaud-Cold

Klaud-Cold commented Feb 2, 2026 •

Copy link
Copy Markdown
Collaborator

Claude finished @functionstackx's task in 47s —— View job


Fixed Model Prefix Typo in nvidia-master.yaml

  • Read nvidia-master.yaml to find the incorrect model prefix
  • Fix dsr1-fp8 to dsr1 in the config
  • Commit and push the fix

Issue: Invalid model-prefix at line 2297

- model-prefix: dsr1-fp8
+ model-prefix: dsr1

The runner script only supports gptoss or dsr1 as model prefixes, not dsr1-fp8.

Fix pushed: e5efea3

--- | | Branch

The model-prefix 'dsr1-fp8' is not a supported prefix. The supported
prefixes are 'gptoss' and 'dsr1'. Changed to 'dsr1' to fix the
launch_gb200-nv.sh runner error.

Co-authored-by: functionstackx <functionstackx@users.noreply.github.com>
@functionstackx

Copy link
Copy Markdown
Collaborator

@jthomson04 can u take a look at this error related to the path of the model?

Traceback (most recent call last):
  File "/mnt/lustre01/users/sa-shared/gharunner0/actions-runner/_work/InferenceMAX/InferenceMAX/srt-slurm/outputs/8517/.srtctl_install/srtctl/cli/do_sweep.py", line 276, in main
    runtime = RuntimeContext.from_config(config, job_id)
  File "/mnt/lustre01/users/sa-shared/gharunner0/actions-runner/_work/InferenceMAX/InferenceMAX/srt-slurm/outputs/8517/.srtctl_install/srtctl/core/runtime.py", line 163, in from_config
    raise FileNotFoundError(f"Model path does not exist: {model_path}")
FileNotFoundError: Model path does not exist: /mnt/lustre01/users/sa-shared/gharunner0/actions-runner/_work/InferenceMAX/InferenceMAX/srt-slurm/dsr1-fp8

https://github.com/InferenceMAX/InferenceMAX/actions/runs/21609573902/job/62274835308?pr=617

@jthomson04

jthomson04 commented Feb 2, 2026 •

Copy link
Copy Markdown
Contributor Author

@jthomson04 can u take a look at this error related to the path of the model?

Traceback (most recent call last):
  File "/mnt/lustre01/users/sa-shared/gharunner0/actions-runner/_work/InferenceMAX/InferenceMAX/srt-slurm/outputs/8517/.srtctl_install/srtctl/cli/do_sweep.py", line 276, in main
    runtime = RuntimeContext.from_config(config, job_id)
  File "/mnt/lustre01/users/sa-shared/gharunner0/actions-runner/_work/InferenceMAX/InferenceMAX/srt-slurm/outputs/8517/.srtctl_install/srtctl/core/runtime.py", line 163, in from_config
    raise FileNotFoundError(f"Model path does not exist: {model_path}")
FileNotFoundError: Model path does not exist: /mnt/lustre01/users/sa-shared/gharunner0/actions-runner/_work/InferenceMAX/InferenceMAX/srt-slurm/dsr1-fp8

https://github.com/InferenceMAX/InferenceMAX/actions/runs/21609573902/job/62274835308?pr=617

Claude broke it. I have a fix locally, will push soon.

Signed-off-by: jthomson04 <jothomson@nvidia.com>
Signed-off-by: jthomson04 <jothomson@nvidia.com>
@functionstackx

Copy link
Copy Markdown
Collaborator

Claude broke it. I have a fix locally, will push soon.

mb lol

@cquil11

cquil11 commented Feb 3, 2026

Copy link
Copy Markdown
Collaborator

@jthomson04 claude not AGI confirmed

@cquil11

cquil11 commented Feb 3, 2026

Copy link
Copy Markdown
Collaborator

@claude please make the perf changelog description and PR description slightly more detailed. don't list every single thing per se, but give a high level of the scenarios added. be more detailed in the PR description (make a table of scenarios, something like https://github.com/InferenceMAX/InferenceMAX/pull/582 description)

@cquil11

cquil11 commented Feb 3, 2026

Copy link
Copy Markdown
Collaborator

@claude in .github/configs/nvidia-master.yaml, there are scenarios with corresponding links to external configs that contain more detailed information. can you please get the configs from the links and double check that the fields (such as prefill/decode ep/tp, num workers, etc) that are also present in this repo's config match?

@Klaud-Cold

Klaud-Cold commented Feb 3, 2026 •

Copy link
Copy Markdown
Collaborator

Claude finished @cquil11's task in 1m 29s —— View job


Updated PR and Changelog Descriptions ✓

  • Read PR [nv] add H200 SGLang disagg configs from srtslurm #582 to understand the expected description format
  • Read current perf-changelog.yaml to see the existing entry
  • Read nvidia-master.yaml to understand scenarios added
  • Update perf-changelog.yaml with more detailed description
  • Update PR description with detailed scenario table
  • Commit and push changes

Changes made:

  1. Updated perf-changelog.yaml with detailed breakdown:

    • Image version and source
    • Scenario counts per ISL/OSL (1k1k, 1k8k, 8k1k)
    • MTP vs STP breakdown
    • Worker configuration ranges (1P-5P, 1D-4D)
  2. Updated PR description with:

    • Summary and changes section
    • Scenario table showing ISL/OSL, mode, scenario counts, workers, and concurrency ranges
    • Architecture variants explanation (DP attention, TEP, MTP, STP)

Commit: 05cde10

--- | Branch

@Klaud-Cold

Klaud-Cold commented Feb 3, 2026 •

Copy link
Copy Markdown
Collaborator

Claude finished @cquil11's task in 2m 34s —— View job


Verified External Config Links ✓

  • Read nvidia-master.yaml to find scenarios with external config links
  • Fetch external configs and extract relevant fields (38 configs found)
  • Compare fields (prefill/decode ep/tp, num workers, etc.) with local config
  • Report any mismatches found

Result: All 38 external configs match the local values.

I fetched each external config from ishandhanani/srt-slurm@sa-submission-q1-2026 and verified the following fields:

  • num-worker (prefill/decode) ↔ prefill_workers / decode_workers
  • tp ↔ tensor_parallel_size
  • ep ↔ moe_expert_parallel_size
  • dp-attn ↔ enable_attention_dp
Verified configs (click to expand)

1k1k MTP configs (7):

  • ✓ ctx1_gen1_dep8_batch512_eplb0_mtp1_4301.yaml
  • ✓ ctx1_gen1_dep8_batch256_eplb0_mtp1_2151.yaml
  • ✓ ctx1_gen1_dep16_batch64_eplb0_mtp1_1229.yaml
  • ✓ ctx1_gen1_dep32_batch16_eplb0_mtp3_615.yaml
  • ✓ ctx1_gen3_tep8_batch8_eplb0_mtp3_36.yaml
  • ✓ ctx1_gen3_tep8_batch4_eplb0_mtp3_18.yaml
  • ✓ ctx1_gen3_tep8_batch2_eplb0_mtp3_9.yaml

1k1k STP configs (7):

  • ✓ ctx1_gen1_dep8_batch768_eplb0_mtp0_6144.yaml
  • ✓ ctx1_gen1_dep8_batch512_eplb0_mtp0_4301.yaml
  • ✓ ctx1_gen1_dep16_batch128_eplb0_mtp0_2151.yaml
  • ✓ ctx1_gen1_dep32_batch32_eplb0_mtp0_1127.yaml
  • ✓ ctx1_gen1_dep32_batch8_eplb0_mtp0_256.yaml
  • ✓ ctx1_gen3_tep8_batch8_eplb0_mtp0_27.yaml
  • ✓ ctx1_gen3_tep8_batch1_eplb0_mtp0_3.yaml

1k8k MTP configs (5):

  • ✓ ctx1_gen2_dep16_batch256_eplb0_mtp1_8192.yaml
  • ✓ ctx1_gen2_dep16_batch64_eplb0_mtp1_2152.yaml
  • ✓ ctx1_gen1_dep32_batch16_eplb0_mtp3_564.yaml
  • ✓ ctx1_gen1_dep32_batch2_eplb0_mtp3_72.yaml
  • ✓ ctx1_gen4_tep8_batch2_eplb0_mtp3_8.yaml

1k8k STP configs (5):

  • ✓ ctx1_gen2_dep16_batch256_eplb0_mtp0_8192.yaml
  • ✓ ctx1_gen1_dep32_batch64_eplb0_mtp0_2048.yaml
  • ✓ ctx1_gen1_dep32_batch16_eplb0_mtp0_564.yaml
  • ✓ ctx1_gen4_tep8_batch8_eplb0_mtp0_36.yaml
  • ✓ ctx1_gen4_tep8_batch1_eplb0_mtp0_4.yaml

8k1k MTP configs (7):

  • ✓ ctx3_gen1_dep8_batch64_eplb0_mtp3_666.yaml
  • ✓ ctx5_gen1_dep16_batch32_eplb0_mtp3_666.yaml
  • ✓ ctx3_gen1_dep16_batch16_eplb0_mtp3_333.yaml
  • ✓ ctx4_gen1_dep32_batch8_eplb0_mtp3_333.yaml
  • ✓ ctx2_gen1_dep32_batch2_eplb0_mtp3_90.yaml
  • ✓ ctx1_gen3_tep8_batch4_eplb0_mtp3_15.yaml
  • ✓ ctx1_gen3_tep8_batch2_eplb0_mtp3_6.yaml

8k1k STP configs (7):

  • ✓ ctx5_gen1_dep16_batch64_eplb0_mtp0_1229.yaml
  • ✓ ctx4_gen1_dep32_batch16_eplb0_mtp0_666.yaml
  • ✓ ctx3_gen1_dep16_batch32_eplb0_mtp0_615.yaml
  • ✓ ctx2_gen1_dep32_batch8_eplb0_mtp0_333.yaml
  • ✓ ctx1_gen3_tep8_batch16_eplb0_mtp0_63.yaml
  • ✓ ctx1_gen3_tep8_batch4_eplb0_mtp0_18.yaml
  • ✓ ctx1_gen3_tep8_batch1_eplb0_mtp0_6.yaml

--- | Branch

Co-authored-by: Cameron Quilici <cquil11@users.noreply.github.com>

@cquil11 cquil11 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Signed-off-by: jthomson04 <jothomson@nvidia.com>
@kedarpotdar-nv

Copy link
Copy Markdown
Collaborator

looking good to merge

@jthomson04
jthomson04 merged commit daa8239 into main Feb 3, 2026
13 checks passed
@jthomson04
jthomson04 deleted the nv/dsr1-fp8-gb200-dynamo-trt branch February 3, 2026 20:27
@cquil11 cquil11 added the NVIDIA label Apr 8, 2026
weireweire added a commit that referenced this pull request Oct 21, 2026
Root cause: the launcher can assign port 5252 to a decode publisher, colliding with node monitoring listeners and aborting performance/eval startup.

Fix: backport NVIDIA/srt-slurm#617 to allocate KV-event blocks from 3000. The SGLang image and serving parameters are unchanged. Validation: patch applies cleanly to the pinned srt-slurm revision; Ruff checks pass.

修复 KV-event 端口与节点监控监听端口冲突,回移 srt-slurm #617;保持镜像及模型参数不变。
weireweire added a commit that referenced this pull request Oct 21, 2026
Round-robin decode routing does not consume KV events. Disable the decode publisher while retaining prefill events and metrics on both roles, and remove the srt-slurm #617 backport from this submission.

Validation: all three bound variants retain prefill events, omit decode events and retain metrics; Ruff and changelog checks pass. 关闭轮询解码侧的 KV-event 发布,保留预填充事件和双方指标,并移除端口补丁。
cquil11 added a commit that referenced this pull request Oct 22, 2026
…nd routing / [NVIDIA][AgentX] 更新 GB300 DSV4.1-Flash 分离式拓扑、缓存与路由 (#4303)

* feat(agentx): GB300 DSV4.1-Flash Dynamo+SGLang disagg on the 20261019 nightly with the Mooncake linker

Move dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg to
lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089 (multiarch digest
pinned) and the ai-dynamo 1.6.0.dev20261018 it ships, with the ZMQ event
plane and no NATS service.

- c1228 keeps 4x DEP2 prefill + 1x DEP16 decode; c2884 moves from 6 to 5
  DEP2 prefill workers (3 prefill nodes, 26 GPUs) and caps prefill batches
  at 32768 tokens.
- Both roles use the flashmla_mega attention backend, local DP control
  broadcast and the Mooncake external cache linker with a Mooncake master
  service (high watermark 0.90, client TTL 60 s, 64 RPC threads). The
  linker replaces prefill HiCache, so the master rows declare Mooncake
  0.3.13.post1 as the KV offload backend.
- Decode adds late admission, stream interval 32, polling interval 8,
  2048 SWA prefix tails and decoder SWA bounded replay; the NCCL DP-batch
  sync env is dropped because the image exchanges MLP sync rows with gloo
  gather + broadcast (sgl-project/sglang#44622).
- A setup script applies four SGLang patches the image lacks, with git
  apply under a lock, idempotently, failing on any hunk that does not
  apply: sgl-project/sglang#44623 (open), #44580 (merged as 834bfe4ef0),
  #45062 (merged as 06342acad2) and decode late admission
  (weireweire/sglang@407e912a1e).
- Revise this PR's changelog entry to the final state.

中文:将 dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg 迁移到
lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089(固定 multiarch
digest)及镜像自带的 ai-dynamo 1.6.0.dev20261018,使用 ZMQ event plane,
不再启动 NATS 服务。

- c1228 保持 4 个 DEP2 prefill + 1 个 DEP16 decode;c2884 的 DEP2 prefill
  由 6 个减为 5 个(3 个 prefill 节点,共 26 个 GPU),并将 prefill 批次
  上限设为 32768 token。
- 两个角色均使用 flashmla_mega attention 后端、本地 DP control broadcast
  和 Mooncake external cache linker,并启动 Mooncake master 服务(高水位
  0.90,client TTL 60 秒,64 个 RPC 线程)。linker 取代了 prefill
  HiCache,因此 master 配置将 KV offload 后端声明为 Mooncake 0.3.13.post1。
- Decode 新增 late admission、stream interval 32、polling interval 8、
  2048 SWA prefix tails 与 decoder SWA bounded replay;镜像已通过 gloo
  gather + broadcast 交换 MLP 同步数据(sgl-project/sglang#44622),因此
  删除 NCCL DP batch 同步环境变量。
- 新增 setup 脚本,在锁保护下用 git apply 幂等地应用镜像缺少的四个
  SGLang 补丁,任一 hunk 无法应用即失败:sgl-project/sglang#44623(未合并)、
  #44580(已合并为 834bfe4ef0)、#45062(已合并为 06342acad2)以及 decode
  late admission(weireweire/sglang@407e912a1e)。
- 将本 PR 的 changelog 条目更新为最终状态。

* fix(agentx): drop decode late admission from the GB300 DSV4.1-Flash disagg points

A same-time A/B on the 20261019 nightly showed no effect from decode late
admission: P90 interactivity 230/232 without it and 230/228 with it at
c1228. The image already exchanges the scheduler's MLP sync rows with gloo
gather + broadcast (sgl-project/sglang#44622) and avoids blocking Engram
history copies during PD admission (#44787). Remove the patch, its setup
script entry and SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION.

中文:在 20261019 nightly 上的同时段 A/B 显示 decode late admission 没有
效果:c1228 下关闭时 P90 interactivity 为 230/232,开启时为 230/228。镜像
已通过 gloo gather + broadcast 交换调度器的 MLP 同步数据
(sgl-project/sglang#44622),并避免了 PD admission 期间阻塞的 Engram
历史拷贝(#44787)。因此删除该补丁、setup 脚本中的对应条目以及
SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION。

* feat(agentx): move the GB300 DSV4.1-Flash disagg points to the 20261020 nightly

The 20261020 image (ab580b2e, ai-dynamo 1.6.0.dev20261019) contains
sgl-project/sglang#44580 and #45062, so the setup script now applies only
the external linker pools patch (#44623). Add the #4303 changelog entry on
top of the merged #4180 entry.

* feat(agentx): run GB300 DSV4.1-Flash disagg c1228 at 3P1D and add c4096 at 7P1D

With the Mooncake linker, three DEP2 prefill workers keep up at c1228
(364.5k/362.1k vs 336.7k tok/s/GPU at 4P1D, same interactivity). Add
c4096 with seven DEP2 prefill workers and 64 requests per decode rank
(630.1k tok/s/GPU, P90 137 tok/s/user). Both decode roles use a 140gb
Mooncake segment.

* perf(agentx): update DSV4.1 disagg to patch-free October 21 nightly

Root cause: the previous image needed an external-linker patch and the
submitted recipe omitted supported routing and worker settings used locally.

Fix: pin the October 21 public image, remove the linker patch and setup script,
enable per-role routing, total-token balancing, worker fastokens and BF16
decode index logits, and use eight DEP2 prefill workers at c4096 within the
same four prefill nodes. Update only the disaggregated matrix and changelog.

Validation: the repository matrix generator resolves all three points with
3600-second measurements and evaluation enabled. The exact new image/config
combination has not yet completed a performance sweep or native evaluation.

原因:旧镜像需要外部缓存 linker 补丁,提交配置也未包含本地使用的部分原生配置项。
修复:固定 10 月 21 日公开镜像,删除补丁,补齐路由、负载均衡、tokenizer 和
decode index logits 配置;c4096 在原有四个 prefill 节点使用八个 DEP2 worker。
验证:三个配置点均通过矩阵生成,保留 3600 秒测量和 eval;新版完整组合尚未实跑。

* config: remove unused HiCache write policy from DSV4.1 linker recipe

* fix: restore decode metrics in DSV4.1 disagg recipes

Root cause: prefill-only Prometheus counters were treated as deployment totals, preventing record-based token fallback and omitting decode KV metrics.

Fix: enable SGLang metrics on decode for all three disaggregated variants.

Validation: Ruff check and format checks passed; perf-changelog matrix validation passed. Full GPU sweep remains pending.

原因:仅 prefill 导出指标会使汇总缺少 decode 数据。修复:为三个 disagg 配置恢复 decode metrics。校验:Ruff 和 changelog 矩阵检查通过,GPU sweep 尚待验证。

* fix: backport KV-event port range correction for disagg runs

Root cause: the launcher can assign port 5252 to a decode publisher, colliding with node monitoring listeners and aborting performance/eval startup.

Fix: backport NVIDIA/srt-slurm#617 to allocate KV-event blocks from 3000. The SGLang image and serving parameters are unchanged. Validation: patch applies cleanly to the pinned srt-slurm revision; Ruff checks pass.

修复 KV-event 端口与节点监控监听端口冲突,回移 srt-slurm #617;保持镜像及模型参数不变。

* docs: keep the launcher port backport minimal

Retain only the port-base change and concise public descriptions. 简化端口补丁及说明。

* config: disable unused decode KV events and drop port backport

Round-robin decode routing does not consume KV events. Disable the decode publisher while retaining prefill events and metrics on both roles, and remove the srt-slurm #617 backport from this submission.

Validation: all three bound variants retain prefill events, omit decode events and retain metrics; Ruff and changelog checks pass. 关闭轮询解码侧的 KV-event 发布,保留预填充事件和双方指标,并移除端口补丁。

* ci: Include unchanged GB300 DSV4.1 aggregated points in sweep

将现有 TP4 c1/c4/c16 和 TP2 c16 纳入本 PR 的 sweep 与 eval 选点,覆盖低延迟区间;保留原有 agg 配置及镜像。矩阵校验生成 7 个性能点和 5 个 eval,Ruff 检查通过。

---------

Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: Cam Quilici <cjquilici@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants