Skip to content

Commit 36585df

Browse files
weireweirecquil11
andauthored
[NVIDIA][AgentX] Update GB300 DSV4.1-Flash disagg topology, caching and routing / [NVIDIA][AgentX] 更新 GB300 DSV4.1-Flash 分离式拓扑、缓存与路由 (#4303)
* feat(agentx): GB300 DSV4.1-Flash Dynamo+SGLang disagg on the 20261019 nightly with the Mooncake linker Move dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg to lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089 (multiarch digest pinned) and the ai-dynamo 1.6.0.dev20261018 it ships, with the ZMQ event plane and no NATS service. - c1228 keeps 4x DEP2 prefill + 1x DEP16 decode; c2884 moves from 6 to 5 DEP2 prefill workers (3 prefill nodes, 26 GPUs) and caps prefill batches at 32768 tokens. - Both roles use the flashmla_mega attention backend, local DP control broadcast and the Mooncake external cache linker with a Mooncake master service (high watermark 0.90, client TTL 60 s, 64 RPC threads). The linker replaces prefill HiCache, so the master rows declare Mooncake 0.3.13.post1 as the KV offload backend. - Decode adds late admission, stream interval 32, polling interval 8, 2048 SWA prefix tails and decoder SWA bounded replay; the NCCL DP-batch sync env is dropped because the image exchanges MLP sync rows with gloo gather + broadcast (sgl-project/sglang#44622). - A setup script applies four SGLang patches the image lacks, with git apply under a lock, idempotently, failing on any hunk that does not apply: sgl-project/sglang#44623 (open), #44580 (merged as 834bfe4ef0), #45062 (merged as 06342acad2) and decode late admission (weireweire/sglang@407e912a1e). - Revise this PR's changelog entry to the final state. 中文:将 dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg 迁移到 lmsysorg/sglang:nightly-dev-cu13-20261019-3c560089(固定 multiarch digest)及镜像自带的 ai-dynamo 1.6.0.dev20261018,使用 ZMQ event plane, 不再启动 NATS 服务。 - c1228 保持 4 个 DEP2 prefill + 1 个 DEP16 decode;c2884 的 DEP2 prefill 由 6 个减为 5 个(3 个 prefill 节点,共 26 个 GPU),并将 prefill 批次 上限设为 32768 token。 - 两个角色均使用 flashmla_mega attention 后端、本地 DP control broadcast 和 Mooncake external cache linker,并启动 Mooncake master 服务(高水位 0.90,client TTL 60 秒,64 个 RPC 线程)。linker 取代了 prefill HiCache,因此 master 配置将 KV offload 后端声明为 Mooncake 0.3.13.post1。 - Decode 新增 late admission、stream interval 32、polling interval 8、 2048 SWA prefix tails 与 decoder SWA bounded replay;镜像已通过 gloo gather + broadcast 交换 MLP 同步数据(sgl-project/sglang#44622),因此 删除 NCCL DP batch 同步环境变量。 - 新增 setup 脚本,在锁保护下用 git apply 幂等地应用镜像缺少的四个 SGLang 补丁,任一 hunk 无法应用即失败:sgl-project/sglang#44623(未合并)、 #44580(已合并为 834bfe4ef0)、#45062(已合并为 06342acad2)以及 decode late admission(weireweire/sglang@407e912a1e)。 - 将本 PR 的 changelog 条目更新为最终状态。 * fix(agentx): drop decode late admission from the GB300 DSV4.1-Flash disagg points A same-time A/B on the 20261019 nightly showed no effect from decode late admission: P90 interactivity 230/232 without it and 230/228 with it at c1228. The image already exchanges the scheduler's MLP sync rows with gloo gather + broadcast (sgl-project/sglang#44622) and avoids blocking Engram history copies during PD admission (#44787). Remove the patch, its setup script entry and SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION. 中文:在 20261019 nightly 上的同时段 A/B 显示 decode late admission 没有 效果:c1228 下关闭时 P90 interactivity 为 230/232,开启时为 230/228。镜像 已通过 gloo gather + broadcast 交换调度器的 MLP 同步数据 (sgl-project/sglang#44622),并避免了 PD admission 期间阻塞的 Engram 历史拷贝(#44787)。因此删除该补丁、setup 脚本中的对应条目以及 SGLANG_ENABLE_DISAGG_DECODE_LATE_ADMISSION。 * feat(agentx): move the GB300 DSV4.1-Flash disagg points to the 20261020 nightly The 20261020 image (ab580b2e, ai-dynamo 1.6.0.dev20261019) contains sgl-project/sglang#44580 and #45062, so the setup script now applies only the external linker pools patch (#44623). Add the #4303 changelog entry on top of the merged #4180 entry. * feat(agentx): run GB300 DSV4.1-Flash disagg c1228 at 3P1D and add c4096 at 7P1D With the Mooncake linker, three DEP2 prefill workers keep up at c1228 (364.5k/362.1k vs 336.7k tok/s/GPU at 4P1D, same interactivity). Add c4096 with seven DEP2 prefill workers and 64 requests per decode rank (630.1k tok/s/GPU, P90 137 tok/s/user). Both decode roles use a 140gb Mooncake segment. * perf(agentx): update DSV4.1 disagg to patch-free October 21 nightly Root cause: the previous image needed an external-linker patch and the submitted recipe omitted supported routing and worker settings used locally. Fix: pin the October 21 public image, remove the linker patch and setup script, enable per-role routing, total-token balancing, worker fastokens and BF16 decode index logits, and use eight DEP2 prefill workers at c4096 within the same four prefill nodes. Update only the disaggregated matrix and changelog. Validation: the repository matrix generator resolves all three points with 3600-second measurements and evaluation enabled. The exact new image/config combination has not yet completed a performance sweep or native evaluation. 原因:旧镜像需要外部缓存 linker 补丁,提交配置也未包含本地使用的部分原生配置项。 修复:固定 10 月 21 日公开镜像,删除补丁,补齐路由、负载均衡、tokenizer 和 decode index logits 配置;c4096 在原有四个 prefill 节点使用八个 DEP2 worker。 验证:三个配置点均通过矩阵生成,保留 3600 秒测量和 eval;新版完整组合尚未实跑。 * config: remove unused HiCache write policy from DSV4.1 linker recipe * fix: restore decode metrics in DSV4.1 disagg recipes Root cause: prefill-only Prometheus counters were treated as deployment totals, preventing record-based token fallback and omitting decode KV metrics. Fix: enable SGLang metrics on decode for all three disaggregated variants. Validation: Ruff check and format checks passed; perf-changelog matrix validation passed. Full GPU sweep remains pending. 原因:仅 prefill 导出指标会使汇总缺少 decode 数据。修复:为三个 disagg 配置恢复 decode metrics。校验:Ruff 和 changelog 矩阵检查通过,GPU sweep 尚待验证。 * fix: backport KV-event port range correction for disagg runs Root cause: the launcher can assign port 5252 to a decode publisher, colliding with node monitoring listeners and aborting performance/eval startup. Fix: backport NVIDIA/srt-slurm#617 to allocate KV-event blocks from 3000. The SGLang image and serving parameters are unchanged. Validation: patch applies cleanly to the pinned srt-slurm revision; Ruff checks pass. 修复 KV-event 端口与节点监控监听端口冲突,回移 srt-slurm #617;保持镜像及模型参数不变。 * docs: keep the launcher port backport minimal Retain only the port-base change and concise public descriptions. 简化端口补丁及说明。 * config: disable unused decode KV events and drop port backport Round-robin decode routing does not consume KV events. Disable the decode publisher while retaining prefill events and metrics on both roles, and remove the srt-slurm #617 backport from this submission. Validation: all three bound variants retain prefill events, omit decode events and retain metrics; Ruff and changelog checks pass. 关闭轮询解码侧的 KV-event 发布,保留预填充事件和双方指标,并移除端口补丁。 * ci: Include unchanged GB300 DSV4.1 aggregated points in sweep 将现有 TP4 c1/c4/c16 和 TP2 c16 纳入本 PR 的 sweep 与 eval 选点,覆盖低延迟区间;保留原有 agg 配置及镜像。矩阵校验生成 7 个性能点和 5 个 eval,Ruff 检查通过。 --------- Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com> Co-authored-by: Cam Quilici <cjquilici@gmail.com>
1 parent 8dc0df1 commit 36585df

3 files changed

Lines changed: 122 additions & 32 deletions

File tree

‎inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv41flash/sglang/gb300-fp4/agentx/disagg-variants.yaml‎

Lines changed: 82 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,12 @@ base:
55
identity:
66
model: {}
77
frameworks:
8-
dynamo: "1.6.0.dev20261014"
9-
sglang: "0.0.0.dev1+ga6a8c5601"
8+
dynamo: "1.6.0.dev20261019"
9+
sglang: "0.0.0.dev1+gc1c287a42"
1010
dynamo:
1111
install: false
1212
request_plane: tcp
13+
event_plane: zmq
1314
# gb300-nv serves this model from the Hub; read it from the shared mounted cache.
1415
environment:
1516
HF_HUB_CACHE: /hf_hub_cache
@@ -31,18 +32,21 @@ base:
3132
type: etcd
3233
placement:
3334
node: infra
34-
- name: nats
35-
type: nats
36-
placement:
37-
node: infra
35+
- name: mooncake-master
36+
type: mooncake-master
37+
args:
38+
- --eviction_high_watermark_ratio=0.90
39+
- --client_ttl=60
40+
- --rpc_thread_num=64
3841
# Disables Dynamo's per-request lifecycle span export; InferenceX does not read it.
3942
- name: otel-collector
4043
type: otel-collector
4144
enabled: false
4245
frontend:
4346
type: dynamo
4447
enable_multiple_frontends: false
45-
# One CPU node hosts the frontend, etcd, NATS and the AgentX client.
48+
numa_bind: true
49+
# One CPU node hosts the frontend, etcd, the Mooncake master and the AgentX client.
4650
placement:
4751
node: dedicated
4852
env:
@@ -77,36 +81,52 @@ base:
7781
NCCL_MNNVL_ENABLE: "1"
7882
NCCL_CUMEM_ENABLE: "1"
7983
MC_FORCE_MNNVL: "1"
84+
MOONCAKE_PROTOCOL: rdma
85+
# The store registers host buffers only over RDMA; without a device list it
86+
# picks NVLink for same-domain peers.
87+
MOONCAKE_DEVICE: "@fabric.rdma-devices"
88+
MOONCAKE_GLOBAL_SEGMENT_SIZE: 80gb
89+
MOONCAKE_STANDALONE_STORAGE: "0"
90+
MC_STORE_CLIENT_METRIC_INTERVAL: "5"
91+
MC_ENABLE_DEST_DEVICE_AFFINITY: "1"
92+
# Some GB300 nodes list a BMC interface with the same address on every node first.
93+
MC_TCP_BIND_ADDRESS: "{node}"
94+
WITH_NVIDIA_PEERMEM: "0"
8095
args:
8196
served-model-name: deepseek-ai/DeepSeek-V4.1-Flash
8297
trust-remote-code: true
8398
tensor-parallel-size: 2
99+
tokenizer-backend: fastokens
100+
router-mode: kv
101+
router-session-affinity-ttl-secs: "14400"
84102
data-parallel-size: 2
85103
expert-parallel-size: 2
86104
enable-dp-attention: true
87105
enable-dp-lm-head: true
106+
enable-dp-attention-local-control-broadcast: true
88107
moe-a2a-backend: megamoe
89-
dsv4-attn-backend: flashmla
108+
dsv4-attn-backend: flashmla_mega
90109
disaggregation-transfer-backend: mooncake
91110
chunked-prefill-size: 65536
92111
max-running-requests: 128
93112
swa-prefix-tails: 2048
94113
enable-decoder-swa-bounded-replay: true
95-
enable-hierarchical-cache: true
96-
hicache-write-policy: write_back
97-
hicache-io-backend: direct
114+
enable-unified-cache-external-linker: true
115+
unified-cache-external-linker-backend: mooncake
116+
hicache-storage-backend-extra-config: '{"enable_group_semantics":true}'
117+
hicache-ratio: 1.02
98118
hicache-mem-layout: page_first_direct
99119
mem-fraction-static: 0.85
100120
speculative-algorithm: DSPARK
101121
speculative-dspark-block-size: 5
122+
speculative-num-steps: 5
102123
reasoning-parser: auto
103124
tool-call-parser: auto
104125
watchdog-timeout: 3600
105126
enable-metrics: true
106127
weight-loader-prefetch-checkpoints: true
107128
weight-loader-drop-cache-after-load: false
108129
model-loader-extra-config: '{"enable_multithread_load":true}'
109-
# Only prefill publishes KV events to the Dynamo KV router.
110130
kv_events: true
111131
decode:
112132
nodes: 4
@@ -120,34 +140,52 @@ base:
120140
DYN_SGL_GATEWAY_WORKERS: "8"
121141
SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT: "3600"
122142
SGLANG_DISAGGREGATION_WAITING_TIMEOUT: "3600"
123-
# Sync the per-step DP batch over NCCL instead of gloo.
124-
SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH: "1"
125143
SGLANG_MOONCAKE_CUSTOM_MEM_POOL: "True"
126144
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK: "16384"
145+
SGLANG_OPT_DSV41_BF16_INDEX_LOGITS: "1"
127146
NCCL_MNNVL_ENABLE: "1"
128147
NCCL_CUMEM_ENABLE: "1"
129148
MC_FORCE_MNNVL: "1"
149+
MOONCAKE_PROTOCOL: rdma
150+
MOONCAKE_DEVICE: "@fabric.rdma-devices"
151+
MOONCAKE_STANDALONE_STORAGE: "0"
152+
MC_STORE_CLIENT_METRIC_INTERVAL: "5"
153+
MC_ENABLE_DEST_DEVICE_AFFINITY: "1"
154+
MC_TCP_BIND_ADDRESS: "{node}"
155+
WITH_NVIDIA_PEERMEM: "0"
130156
args:
131157
served-model-name: deepseek-ai/DeepSeek-V4.1-Flash
132158
trust-remote-code: true
133159
tensor-parallel-size: 16
160+
tokenizer-backend: fastokens
161+
router-mode: round-robin
162+
load-balance-method: total_tokens
134163
data-parallel-size: 16
135164
expert-parallel-size: 16
136165
enable-dp-attention: true
137166
enable-dp-lm-head: true
167+
enable-dp-attention-local-control-broadcast: true
138168
moe-a2a-backend: megamoe
139-
dsv4-attn-backend: flashmla
169+
dsv4-attn-backend: flashmla_mega
140170
disaggregation-transfer-backend: mooncake
171+
disaggregation-decode-polling-interval: 8
172+
stream-interval: 32
173+
swa-prefix-tails: 2048
174+
enable-decoder-swa-bounded-replay: true
175+
enable-unified-cache-external-linker: true
176+
unified-cache-external-linker-backend: mooncake
141177
mem-fraction-static: 0.85
142178
speculative-algorithm: DSPARK
143179
speculative-dspark-block-size: 5
180+
speculative-num-steps: 5
144181
reasoning-parser: auto
145182
tool-call-parser: auto
146183
watchdog-timeout: 3600
147184
enable-metrics: true
148185
weight-loader-prefetch-checkpoints: true
149186
weight-loader-drop-cache-after-load: false
150187
model-loader-extra-config: '{"enable_multithread_load":true}'
188+
kv_events: false
151189
sbatch_directives:
152190
mem: "0"
153191
exclusive: ""
@@ -167,30 +205,50 @@ base:
167205
AIPERF_EXTRA_INPUTS: '{"chat_template_kwargs":{"reasoning_effort":"high","thinking":true},"temperature":1}'
168206
AIPERF_DATASET_MMAP_CACHE_ENABLED: "true"
169207

170-
override_4p1d_dep2_dep16_c1228:
171-
name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-4p1d-dep2-dep16-c1228
208+
override_3p1d_dep2_dep16_c1228:
209+
name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-3p1d-dep2-dep16-c1228
172210
roles:
173211
prefill:
212+
# Three DEP2 workers leave two GPUs of the second node idle.
174213
nodes: 2
175-
workers: 4
176-
args:
177-
hicache-ratio: 1.02
214+
workers: 3
178215
decode:
216+
env:
217+
MOONCAKE_GLOBAL_SEGMENT_SIZE: 140gb
179218
args:
180219
max-running-requests: 384
181220
cuda-graph-max-bs-decode: 24
182221
disaggregation-decode-extra-slots: 36
183222

184-
override_6p1d_dep2_dep16_c2884:
185-
name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-6p1d-dep2-dep16-c2884
223+
override_5p1d_dep2_dep16_c2884:
224+
name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-5p1d-dep2-dep16-c2884
186225
roles:
187226
prefill:
227+
# Five DEP2 workers leave two GPUs of the third node idle.
188228
nodes: 3
189-
workers: 6
229+
workers: 5
190230
args:
191-
hicache-ratio: 1.1
231+
max-prefill-tokens: 32768
192232
decode:
233+
env:
234+
MOONCAKE_GLOBAL_SEGMENT_SIZE: 140gb
193235
args:
194236
max-running-requests: 640
195237
cuda-graph-max-bs-decode: 40
196238
disaggregation-decode-extra-slots: 80
239+
240+
override_8p1d_dep2_dep16_c4096:
241+
name: dsv41flash-fp4-gb300-dynamo-sglang-agentx-disagg-8p1d-dep2-dep16-c4096
242+
roles:
243+
prefill:
244+
nodes: 4
245+
workers: 8
246+
args:
247+
max-prefill-tokens: 32768
248+
decode:
249+
env:
250+
MOONCAKE_GLOBAL_SEGMENT_SIZE: 140gb
251+
args:
252+
max-running-requests: 1024
253+
cuda-graph-max-bs-decode: 64
254+
disaggregation-decode-extra-slots: 128

‎inferencex-e2e/configs/nvidia-master.yaml‎

Lines changed: 23 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -9322,13 +9322,13 @@ dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg:
93229322
dp-attn: false
93239323

93249324
dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg:
9325-
image: lmsysorg/sglang:nightly-dev-cu13-20261015-a6a8c560@sha256:0a1ba33aa4e032b8c5bcebdc69c45664074c032074add4f7db01629ec1fcc381
9325+
image: lmsysorg/sglang:nightly-dev-cu13-20261021-c1c287a4@sha256:b0473834763b2753e069cb96c1acaf235ba7fd632ccd9bbeb44da809c437fdeb
93269326
model: deepseek-ai/DeepSeek-V4.1-Flash
93279327
model-prefix: dsv41flash
93289328
runner: cluster:gb300-nv
93299329
precision: fp4
93309330
framework: dynamo-sglang
9331-
router: { name: dynamo-router, version: "1.6.0.dev20261014" }
9331+
router: { name: dynamo-router, version: "1.6.0.dev20261019" }
93329332
kv-p2p-transfer: mooncake
93339333
multinode: true
93349334
disagg: true
@@ -9339,11 +9339,11 @@ dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg:
93399339
search-space:
93409340
- spec-decoding: mtp
93419341
conc-list: [1228]
9342-
srt-recipe: disagg-variants.yaml:override_4p1d_dep2_dep16_c1228
9342+
srt-recipe: disagg-variants.yaml:override_3p1d_dep2_dep16_c1228
93439343
kv-offloading: dram
9344-
kv-offload-backend: { name: hicache }
9344+
kv-offload-backend: { name: mooncake, version: "0.3.13.post1" }
93459345
prefill:
9346-
num-worker: 4
9346+
num-worker: 3
93479347
tp: 2
93489348
ep: 2
93499349
dp-attn: true
@@ -9354,11 +9354,26 @@ dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg:
93549354
dp-attn: true
93559355
- spec-decoding: mtp
93569356
conc-list: [2884]
9357-
srt-recipe: disagg-variants.yaml:override_6p1d_dep2_dep16_c2884
9357+
srt-recipe: disagg-variants.yaml:override_5p1d_dep2_dep16_c2884
93589358
kv-offloading: dram
9359-
kv-offload-backend: { name: hicache }
9359+
kv-offload-backend: { name: mooncake, version: "0.3.13.post1" }
93609360
prefill:
9361-
num-worker: 6
9361+
num-worker: 5
9362+
tp: 2
9363+
ep: 2
9364+
dp-attn: true
9365+
decode:
9366+
num-worker: 1
9367+
tp: 16
9368+
ep: 16
9369+
dp-attn: true
9370+
- spec-decoding: mtp
9371+
conc-list: [4096]
9372+
srt-recipe: disagg-variants.yaml:override_8p1d_dep2_dep16_c4096
9373+
kv-offloading: dram
9374+
kv-offload-backend: { name: mooncake, version: "0.3.13.post1" }
9375+
prefill:
9376+
num-worker: 8
93629377
tp: 2
93639378
ep: 2
93649379
dp-attn: true

‎inferencex-e2e/perf-changelog.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10806,3 +10806,20 @@
1080610806
description:
1080710807
- "Route llm-d DEP8 disagg and agg workers with data-parallel-external-lb (one vLLM API per DP rank for the EPP) and VLLM_USE_RUST_FRONTEND=1, matching the Sep 28 standalone llm-d wide-EP launch; agg TP8 keeps a single TP8 worker without external LB. Drop 2P×1D disagg agentic concurrency 576 from the master search space after repeated warmup stalls."
1080810808
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3741
10809+
10810+
- config-keys:
10811+
- dsv41flash-fp4-gb300-dynamo-sglang-agentic-dspark-disagg
10812+
- dsv41flash-fp4-gb300-dynamo-sglang-agentic-agg
10813+
scenario-type:
10814+
- agentic-coding
10815+
description:
10816+
- "Include the unchanged aggregated TP4 c1/c4/c16 and TP2 c16 points in the sweep and evaluation coverage, retaining their pinned October 15 image."
10817+
- "Move the disaggregated points to lmsysorg/sglang:nightly-dev-cu13-20261021-c1c287a4 (digest sha256:b0473834763b2753e069cb96c1acaf235ba7fd632ccd9bbeb44da809c437fdeb) with the ai-dynamo 1.6.0.dev20261019 it ships and the ZMQ event plane. One CPU node hosts etcd, the Mooncake master, the frontend and the AgentX client."
10818+
- "Replace prefill HiCache DRAM offload with the Mooncake external cache linker on both roles (MOONCAKE_GLOBAL_SEGMENT_SIZE 80gb on prefill, 140gb on decode). The higher prefix hit rate lets c1228 run 3x DEP2 prefill instead of 4x and c2884 5x instead of 6x."
10819+
- "Add c4096 with 8x DEP2 prefill (4 nodes) and one DEP16 decode: 64 requests per decode rank, 128 extra decode slots."
10820+
- "Other engine changes: flashmla_mega attention and local DP control broadcast on both roles; on decode, 2048 SWA prefix tails, decoder SWA bounded replay, stream interval 32 and polling interval 8, with the NCCL DP-batch sync env removed; c2884 and c4096 cap prefill batches at 32768 tokens."
10821+
- "Use the image's upstream DeepSeek-V4.1 external linker pool support (sgl-project/sglang#44623) without SGLang source patches."
10822+
- "Keep KV routing and session affinity on prefill; use round-robin routing on decode so SGLang places requests with total_tokens load balancing."
10823+
- "Use fastokens in both worker roles and BF16 decode index logits."
10824+
- "Disable decode KV-event publishing for round-robin routing; retain prefill KV events and metrics on both roles."
10825+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4303

0 commit comments

Comments
 (0)