Skip to content

[AMD][AgentX] DeepSeek-V4.1-Flash MI355X vLLM: add DP2 concurrency 128-384, with UMBP KV offloading at 256 and 384 / [AMD][AgentX] DeepSeek-V4.1-Flash MI355X vLLM:新增 DP2 并发 128–384 点,256、384 启用 UMBP KV offloading - #5036

Open
shen-shanshan wants to merge 16 commits into
mainfrom
amd-agentx-dsv41-flash-umbp

Conversation

@shen-shanshan

@shen-shanshan shen-shanshan commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The MI355X DeepSeek-V4.1-Flash vLLM AgentX curve stops at concurrency 64. Above that, the GPU KV cache stops holding every live session's prefix. Evicted prefixes are then recomputed, and prefill crowds out decode. vllm-project/vllm#64715 (merged 2026-11-10) adds UMBP standalone support for DeepSeek-V4.1-Flash on ROCm. With it, the engine offloads KV blocks to a node-local host-DRAM pool owned by MORI's umbp_standalone_server and restores them on a prefix hit instead of recomputing.

This PR adds a new config key, dsv41flash-fp4-mi355x-vllm-agentic-dspark-umbp, with three points: concurrency 128 without KV offloading, and 256 and 384 with UMBP. Evals run at c384. The c128 point is carried over from the closed #4992 unchanged; UMBP is off there because it did not help at 128 in local runs (see below).

  • Image: vllm/vllm-openai-rocm:nightly-60bd6dd2edd175a1c55fd3488f871e4c3d2ef27d (2026-11-16, digest sha256:67a3a0fb7989…). It carries #64715 (UMBP standalone for DeepSeek-V4.1-Flash), #66036 (per-tensor fp8 KV for sparse MLA on ROCm) and #67377 (AITER FlyDSL dense MXFP8 GEMMs for DeepSeek-V4.1-Flash), and its MORI v1.2.3.post1 is built with UMBP. The existing TP4/TP2 key and the TP8 c1 key are untouched and are not rerun.
  • KV cache dtype: fp8_e4m3 (#66036): plain per-tensor fp8 KV rows for the AITER sparse MLA instead of the default block-scaled fp8_ds_mla record. It applies to all three points.
  • Topology: two GPUs per server with attention DP2. The MoE stays tensor-parallel across both ranks (tensor-parallel-size 1, data-parallel-size 2, no expert parallelism). vllm-router 0.1.14 fronts the ranks with consistent_hash, so each session keeps its rank and its prefix. Every point uses the settings of the closed [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 with prefill cadence / [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 启用 prefill 节奏 #4992's DP2 points: prefill cadence (prefill-schedule-interval 11) and no --monokernel. The UMBP points differ from c128 only by UMBP.
  • Recipe: one override per row, each selected explicitly: override_dp2_c128 (no offloading) and override_dp2_umbp_c{256,384}.
  • KV offloading (256 and 384): the recipe launches umbp_standalone_server as a critical srt-slurm generic service that starts before the workers and becomes ready once it logs [StandaloneServer] listening.
    • The server owns one 449 GB pool: dram-utilization 0.60 of the two-GPU node share, on 4 KiB pages. Both DP ranks share it.
    • srtctl gives every step of a job its own mount namespace, so the server's /tmp is not the workers'. The recipe mounts the node-local /dev/shm into every step at /umbp-host, and the gRPC and fd sockets live in /umbp-host/umbp/. A new server rebinds over socket files an earlier job left behind.
    • The engine attaches through UMBPStoreConnector (mode: standalone, kv_load_failure_policy: recompute).
  • Draft: weights unchanged. It is the shipped DSpark head (mtp.0-2), loaded with the checkpoint's FP8 config. Its attention layers share the target's KV cache dtype, so target and draft both use the per-tensor fp8_e4m3 cache.

vLLM recipe PR:

Why: local no-offload (B0) vs UMBP (S1) runs

Local AgentX replays on one MI355X node (GPUs 0-1) with this recipe's engine settings before the fp8 KV change (default fp8_ds_mla KV): the nightly-96710245 container with vLLM main 595b1514e8 Python code. B0 is #4992's DP2 configuration; S1 adds UMBP with the 449 GB pool. Each point is a single InferenceX agentx-fast run (1200 s profile, prefix cache cold at the start), not a CI run. As in CI, UMBP turns off the extensible KV cache, so S1 has 4.2% less GPU KV than B0 (26.71M vs 27.89M tokens per rank).

Metric c128 B0 → S1 c256 B0 → S1 c384 B0 → S1 c544 B0 → S1 (not in this PR)
Total throughput per GPU (tok/s) 193,099 → 183,984 (−4.7%) 250,435 → 275,836 (+10.1%) 143,888 → 291,619 (+103%) 46,641 → 287,025 (+515%)
TTFT p50 / p90 (s) 0.87 / 2.32 → 0.88 / 2.62 1.60 / 5.12 → 1.45 / 4.10 7.36 / 264.9 → 2.41 / 7.92 66.1 / 387.3 → 21.9 / 58.7
TPOT mean (ms) 24.8 → 26.6 50.0 → 43.6 102.7 → 69.1 144.9 → 87.5
Interactivity p90 (tok/s/user) 29.2 → 27.4 12.9 → 16.4 6.6 → 10.4 6.8 → 8.5
Prompt tokens from GPU prefix cache (B0 / S1) 90.0% / 89.4% 87.4% / 88.1% 73.0% / 64.6% 27.8% / 21.7%
Prompt tokens restored from UMBP 0.46% 0.67% 21.5% 62.0%
Prompt tokens computed 10.0% → 10.1% 12.6% → 11.3% 27.0% → 13.9% 72.2% → 16.3%
  • The gain tracks how far the working set exceeds GPU KV. At c384 the no-offload server is overloaded: its GPU prefix hit rate falls to 73%, TTFT p90 reaches 265 s, and throughput drops below c256. UMBP restores 21.5% of prompt tokens, halves the prefill computed, doubles throughput and brings TTFT p90 down to 7.9 s.
  • At c256 UMBP restores 0.67% of prompt tokens, cuts prefill computed by 11%, and raises throughput by 10% and p90 interactivity by 27%.
  • At c128 GPU KV holds the working set (90% prefix hits). UMBP restores 0.46% and does not pay for the smaller GPU KV and its store traffic: throughput is 4.7% lower, near the ~4.5% noise of a single fast run, with TTFT and TPOT moving the same way. The c128 point therefore runs without offloading.
  • c544 is beyond the knee and is not part of this PR. Without offloading the server collapses: 27.8% GPU prefix hits, 72% of prompt tokens recomputed, 46,641 tok/s/GPU. UMBP restores 62% of prompt tokens and holds 287,025 tok/s/GPU, level with c384, but TTFT p50/p90 rise to 21.9/58.7 s, 3.6% of steps exceed the cudagraph capture limit, and there are 20 preemptions. Throughput with UMBP therefore peaks around c384.
  • No UMBP eviction (the pool never reached its watermark) at any point, and no preemption up to c384. Steps of 1,024-2,048 tokens, which include decode steps above the 1,024-token cudagraph capture limit, are 0.7% at c384, as at c256.
  • The first S1 run at c544 failed during warmup, and a rerun succeeded. On the first requests, DP0's worker JIT-compiled Triton kernels that warmup does not cover (indexer prefill metadata among them) while holding the GIL. DP1 waited in the cross-DP all-reduce of a dummy batch until VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS (300 s) expired and the engine died. In the rerun, with a partly warm Triton cache, the same stall lasted about 75 s. Every run, including B0, logs runtime JIT warnings, so this is not specific to UMBP.
  • [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 with prefill cadence / [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 启用 prefill 节奏 #4992 reports 310,739 tok/s/GPU at c256 from a one-hour local run on nightly-083fbac02. The B0 numbers above come from shorter, cold-cache fast runs, so compare UMBP against B0 here rather than against that absolute figure.

Accuracy: GSM8K. (copy from vllm-project/vllm#64715)

  • lm_eval with 5-shot, all 1319 questions, local-completions, num_concurrent=64, max_gen_toks=2048.
  • The baseline has no connector and runs once. The UMBP deployment runs twice: pass 1 stores, then /reset_prefix_cache, then pass 2 restores the same prompts from the pool.
Run strict-match flexible-extract Mean acceptance length Prompt tokens from UMBP
Baseline 0.9212 ± 0.0074 0.9204 ± 0.0075 3.43 0
UMBP pass 1 0.9333 ± 0.0069 0.9325 ± 0.0069 3.42 0
UMBP pass 2 0.9295 ± 0.0071 0.9287 ± 0.0071 3.44 92.7%
  • Every gap is within about 1.2 combined standard errors.
  • Draft acceptance stays flat. Corrupted restored KV would lower it, because probabilistic rejection sampling keeps outputs exact even with bad drafts.

Risks

  • The first CI run (https://github.com/SemiAnalysisAI/InferenceX/actions/runs/43151259956) failed on every job because the sockets were under /tmp: the workers could not attach to the server (cannot attach to the UMBP standalone server at unix:///tmp/umbp/standalone.grpc.sock). The host-directory mount above fixes the path. The server and the workers also exchange GPU memory handles over HIP IPC; enroot shares the host IPC and PID namespaces, so this should work, but the next sweep is the first confirmation.
  • The local runs below predate nightly-60bd6dd and fp8_e4m3: they used the nightly-96710245 image's native binaries with newer vLLM Python code, the default fp8_ds_mla KV cache, and fast mode. The fp8 KV layout changes the GPU KV capacity and UMBP block size and was not measured locally; the sweep is its first end-to-end run.
  • A cold Triton cache on a CI node can make the first-request JIT stall exceed the 300 s execute-model timeout, as in the first local c544 run, and fail the point. The sweep shows whether this happens at c256 or c384.
AI model disclosure
  • Model/version: claude-opus-5-5[1m] (Claude Opus 5.5, 1M context), via Claude Code.
  • Role: read the existing config, recipe and srt-slurm service code; drafted the master-config key, recipe overrides, perf-changelog.yaml entry and this description; ran matrix generation, recipe rendering with the pinned srtctl, infx.matrix.changed and the sweep planner; ran and summarized the local B0/S1 AgentX benchmarks. No delegated agents. A human reviewed the plan and chose the topology (DP2), the concurrencies and the UMBP mode.
  • Model/version: Grok 4.7, via Cursor.
  • Role: moved the pinned image to vllm/vllm-openai-rocm:nightly-60bd6dd2edd175a1c55fd3488f871e4c3d2ef27d (digest sha256:67a3a0fb7989…), which includes [Perf][ROCm][DSv4.1] DeepSeek-V4.1-Flash: dense MXFP8 GEMMs and wo_a on AITER's tuned FlyDSL kernels, and a faster MXFP8 activation quant (MI355X) vllm-project/vllm#67377, and updated the changelog entry and this description.
PR checklist and change type

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other (please describe)

Checklist

  • I have completed the AI model disclosure and kept it current
  • I have tested my changes locally
  • I have updated documentation if necessary
  • If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries
  • If this PR appends an inferencex-e2e/perf-changelog.yaml entry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
  • Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.

vLLM recipe PR:

中文

MI355X 上 DeepSeek-V4.1-Flash 的 vLLM AgentX 曲线目前只到并发 64。并发更高时,GPU KV cache 放不下所有活跃会话的前缀,被淘汰的前缀只能重新计算,prefill 会挤占 decode。vllm-project/vllm#64715(2026-11-10 合入)在 ROCm 上为 DeepSeek-V4.1-Flash 支持了 UMBP standalone 模式:引擎把 KV block 卸载到节点本地的主机内存池(由 MORI 的 umbp_standalone_server 管理),前缀命中时直接恢复,不再重算。

本 PR 新增配置键 dsv41flash-fp4-mi355x-vllm-agentic-dspark-umbp,包含三个点:c128 不开 KV offloading,c256 和 c384 开 UMBP,eval 在 c384 上跑。c128 这个点原样搬自已关闭的 #4992;本地测试中 UMBP 在 c128 没有收益(见下文),所以这个点不开 offloading。

  • 镜像:vllm/vllm-openai-rocm:nightly-60bd6dd2edd175a1c55fd3488f871e4c3d2ef27d(2026-11-16,digest sha256:67a3a0fb7989…),包含 #64715(DSv4.1-Flash 的 UMBP standalone 支持)、#66036(ROCm 上 sparse MLA 的 per-tensor fp8 KV)和 #67377(DSv4.1-Flash 的 AITER FlyDSL dense MXFP8 GEMM),其中的 MORI v1.2.3.post1 编译时开启了 UMBP。已有的 TP4/TP2 键和 TP8 c1 键不变,也不会重跑。
  • KV cache 类型:fp8_e4m3(#66036),即给 AITER sparse MLA 用的 per-tensor fp8 KV,而不是默认的块缩放 fp8_ds_mla。三个点都使用这种格式。
  • 拓扑:每个服务 2 张卡,attention 用 DP2,MoE 在两个 rank 间做张量并行(tensor-parallel-size 1、data-parallel-size 2,不开 expert parallelism)。前端是 vllm-router 0.1.14,策略为 consistent_hash,同一会话固定落在同一个 rank 上,前缀也留在那里。所有点都沿用已关闭的 [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 with prefill cadence / [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 启用 prefill 节奏 #4992 中 DP2 点的参数:prefill cadence 为 11(prefill-schedule-interval 11),不开 --monokernel。两个 UMBP 点和 c128 唯一的区别就是开了 UMBP。
  • Recipe:每一行对应一个 override,并显式指定:override_dp2_c128(不开 offloading),以及 override_dp2_umbp_c{256,384}。
  • KV offloading(c256、c384):recipe 把 umbp_standalone_server 作为 srt-slurm 的 generic 服务启动,标记为 critical,在 worker 之前启动,日志出现 [StandaloneServer] listening 即视为就绪。
    • server 管理一个 449 GB 的池子,大小是两卡份额的 dram-utilization 0.60,使用 4 KiB 页,两个 DP rank 共用。
    • srtctl 会给同一个作业里的每个 step 分配独立的 mount namespace,server 看到的 /tmp 和 worker 看到的不是同一个。所以 recipe 把节点本地的 /dev/shm 挂载到每个 step 的 /umbp-host,gRPC socket 和 fd socket 都放在 /umbp-host/umbp/ 下。之前作业留下的 socket 文件不影响新 server 重新绑定。
    • 引擎通过 UMBPStoreConnector 接入(mode: standalone,kv_load_failure_policy: recompute)。
  • Draft:权重不变,仍是随 checkpoint 发布的 DSpark head(mtp.0-2),按 checkpoint 的 FP8 配置加载。它的 attention 层和 target 使用同一种 KV cache 类型,所以 target 和 draft 都用 per-tensor fp8_e4m3 KV。

依据:本地「不开 offloading(B0)」与「UMBP(S1)」的对比。

结果(每 GPU 总吞吐,B0 → S1):

并发 吞吐 从 UMBP 恢复的 prompt 说明
c128 193,099 → 183,984(−4.7%) 0.46% TTFT p90 2.32 → 2.62 s,TPOT 均值 24.8 → 26.6 ms
c256 250,435 → 275,836(+10.1%) 0.67% TTFT p90 5.12 → 4.10 s,p90 interactivity +27%
c384 143,888 → 291,619(+103%) 21.5% B0 的 GPU prefix 命中降到 73%,TTFT p90 从 265 s 降到 7.9 s
c544(不在本 PR 中) 46,641 → 287,025(+515%) 62.0% B0 的 GPU prefix 命中只剩 27.8%;S1 的吞吐与 c384 持平,但 TTFT p50 / p90 为 21.9 / 58.7 s,有 20 次 preemption
  • 收益取决于 GPU KV 放不下的程度。
    • c384 下 B0 已经过载:吞吐比 c256 还低。UMBP 让需要计算的 prefill 减半,吞吐翻倍。
    • c256 下 UMBP 让需要计算的 prefill 减少 11%,吞吐提升 10%。
  • c128 没有收益。 GPU KV 基本够用,GPU prefix 命中 90%,UMBP 只恢复了 0.46% 的 prompt,抵不过 GPU KV 变少和 store 的开销。吞吐 −4.7%,接近单次 fast 运行约 4.5% 的噪声,但 TTFT 和 TPOT 的变化方向一致,所以 c128 这个点不开 offloading。
  • c544 已经过了拐点,不放进本 PR。 不开 offloading 时服务崩溃:GPU prefix 命中只剩 27.8%,72% 的 prompt 要重算,吞吐只有 46,641。UMBP 恢复了 62% 的 prompt,吞吐保持在 287,025,与 c384 持平,但 TTFT p50 / p90 升到 21.9 / 58.7 s,3.6% 的 step 超过 cudagraph 上限,并出现 20 次 preemption。因此开 UMBP 时吞吐在 c384 附近达到上限。
  • c544 的 S1 第一次运行挂死,重跑成功。 第一批请求到达时,DP0 的 worker 在运行时 JIT 编译 warmup 没覆盖到的 Triton kernel(包括 indexer 的 prefill 元数据),并一直持有 GIL。DP1 在 dummy batch 的跨 DP all_reduce 里等待,超过 VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS(300 s)后引擎崩溃。重跑时 Triton cache 已部分预热,同样的停顿约 75 s 后恢复。所有运行(包括 B0)都有运行时 JIT 的警告,所以这不是 UMBP 特有的问题。
  • 其他检查都正常。
    • 所有点都没有发生 UMBP 淘汰;c384 及以下没有 preemption。
    • c384 下 1,024–2,048 token 的 step 只占 0.7%,和 c256 相当,cudagraph 上限 1,024 不需要调高。
  • 和 [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 with prefill cadence / [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 启用 prefill 节奏 #4992 的绝对值不能直接比。 [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 with prefill cadence / [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 启用 prefill 节奏 #4992 的 c256 是 1 小时本地运行,镜像 nightly-083fbac02,结果为 310,739 tok/s/GPU。上表 B0 是较短、开始时 cache 为冷的 fast 运行,所以 UMBP 的收益应以表中的 B0 为基准,不要和这个绝对值比。

风险:

  • 第一次 CI 运行(https://github.com/SemiAnalysisAI/InferenceX/actions/runs/43151259956)所有 job 都失败了,原因是 socket 放在 /tmp 下,worker 连不上 server(cannot attach to the UMBP standalone server at unix:///tmp/umbp/standalone.grpc.sock)。上面的宿主机目录挂载已修复了这个路径问题。server 和 worker 之间还要通过 HIP IPC 交换 GPU 显存句柄;enroot 共享宿主机的 IPC 和 PID namespace,这一步应该没问题,但要等下一次 sweep 才能确认。
  • 下面的本地测试是在换成 nightly-60bd6dd 和 fp8_e4m3 之前做的:用的是 nightly-96710245 镜像的 native 部分加上更新的 vLLM Python 代码,KV 是默认的 fp8_ds_mla,而且是 fast 模式。fp8 KV 会改变 GPU KV 容量和 UMBP 每个 block 的大小,本地没有测过,这次 sweep 是它第一次端到端运行。
  • CI 节点上的 Triton cache 如果是冷的,第一批请求触发的 JIT 停顿可能超过 300 s 的 execute-model 超时,就像本地 c544 第一次运行那样,导致该点失败。c256 和 c384 会不会遇到,要看 sweep 结果。

AI 模型使用说明:使用 claude-opus-5-5[1m](Claude Opus 5.5,1M context),通过 Claude Code。工作内容:阅读现有配置、recipe 和 srt-slurm 服务代码;起草 master config 键、recipe override、perf-changelog.yaml 记录和本描述;运行 matrix 生成、用固定版本的 srtctl 渲染 recipe、infx.matrix.changed 和 sweep planner;运行本地 B0 / S1 的 AgentX 测试并汇总结果。没有委派给其他 agent。方案经人工审阅,拓扑(DP2)、并发点和 UMBP 模式由人工决定。随后使用 Grok 4.7(通过 Cursor)把镜像更新到 vllm/vllm-openai-rocm:nightly-60bd6dd2edd175a1c55fd3488f871e4c3d2ef27d(包含 vllm-project/vllm#67377),并同步 changelog 和本描述。

改动类型:配置变更。

vLLM recipe PR:

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@shen-shanshan
shen-shanshan force-pushed the amd-agentx-dsv41-flash-umbp branch from c0f85e6 to 506b46d Compare November 12, 2026 07:48

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

This review covers commit c0f85e6, which is no longer the latest commit on this pull request; later commits are not covered by it.

@shen-shanshan shen-shanshan added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Nov 12, 2026
@Fangzhou-Ai Fangzhou-Ai added non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures and removed full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) labels Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

@Fangzhou-Ai Fangzhou-Ai added non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) and removed full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) labels Nov 12, 2026
@shen-shanshan shen-shanshan added non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) and removed non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) labels Nov 12, 2026
@chunfangamd
chunfangamd marked this pull request as draft November 12, 2026 18:23
@shen-shanshan
shen-shanshan force-pushed the amd-agentx-dsv41-flash-umbp branch from c3680f0 to 3a4e067 Compare November 13, 2026 03:13
@shen-shanshan shen-shanshan removed the non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) label Nov 13, 2026
shen-shanshan and others added 2 commits November 14, 2026 09:47
…7189 with fp8 KV

Move the UMBP key to vllm/vllm-openai-rocm:nightly-be47189f37 (2026-11-12),
which adds vllm-project/vllm#66036 on top of #64715, and serve both DP2
points with kv-cache-dtype fp8_e4m3: per-tensor fp8 KV rows for the AITER
sparse MLA instead of the block-scaled fp8_ds_mla record. The DSpark
draft layers share the target's cache dtype; draft weights are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…oading

Carry over concurrency 128 from the closed #4992 into the UMBP key: the
same DP2 engine and router settings, prefill cadence 11 and fp8_e4m3 KV
cache on the same nightly, with no KV offloading. Local runs showed UMBP
does not help at 128, where the GPU KV cache holds the working set, so
the point runs without it while 256 and 384 keep UMBP.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@shen-shanshan
shen-shanshan force-pushed the amd-agentx-dsv41-flash-umbp branch from 013edc0 to 86b7d7d Compare November 14, 2026 09:50
@chunfangamd chunfangamd added the full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures label Nov 14, 2026
@Fangzhou-Ai

Copy link
Copy Markdown
Collaborator

Let's sweep with this PR vllm-project/vllm#67377 included before merging

Fangzhou-Ai and others added 2 commits November 15, 2026 07:01
…16ae

The 2026-11-15 ROCm nightly includes vllm-project/vllm#67377 (AITER FlyDSL
dense MXFP8 GEMMs for DeepSeek-V4.1-Flash) on top of #64715 and #66036.

Co-authored-by: Cursor <cursoragent@cursor.com>
…h-umbp

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
@Fangzhou-Ai Fangzhou-Ai added non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) and removed full-sweep-enabled Full sweep with canary gate; matrix jobs run to completion despite failures labels Nov 15, 2026
The c256 UMBP point landed on that node twice. Slurm exclude keeps the rerun off it.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Fangzhou-Ai Fangzhou-Ai added non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) and removed non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) labels Nov 15, 2026
Comment thread inferencex-e2e/configs/runners.yaml Outdated
partition: compute
exclusive: true
# mia1-p01-g11 is a known bad node.
exclude: [mia1-p01-g11]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@functionstackx @chunfangamd please take this node down, this one is broken with very bad performance across different frameworks.

Fangzhou-Ai and others added 3 commits November 16, 2026 07:45
…d6dd

The 2026-11-16 ROCm nightly is 50 commits ahead of nightly-9bc16ae and still
carries vllm-project/vllm#64715, #66036, and #67377.

Co-authored-by: Cursor <cursoragent@cursor.com>
…h-umbp

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml
Co-authored-by: Cursor <cursoragent@cursor.com>
@Fangzhou-Ai Fangzhou-Ai added non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) and removed non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) labels Nov 16, 2026
partition: compute
exclusive: true
# Known bad nodes.
exclude: [mia1-p01-g09, mia1-p01-g11]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

g09 is also down

Comment on lines +317 to +318
# The mi355x-amds host setup releases hugepages, so the pool uses 4 KiB pages.
UMBP_DRAM_USE_HUGEPAGES: '0'

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if u wanna use hugepages, cant u use madvise like mooncake?

madvise is enabled on the cluster

kvcache-ai/Mooncake#4393
kvcache-ai/Mooncake#5151

@chunfangamd

Copy link
Copy Markdown
Collaborator

/use 43861172833

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

@chunfangamd staged run 43861172833.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

@chunfangamd chunfangamd left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Checklist items
  • Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
  • Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
  • Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/43861172833
  • Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/43861172833
  • Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
  • Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
  • For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
  • Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
  • Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
  • Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md.
  • For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
  • If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
  • For official InferenceX (non-preview), verified that established models use upstream engine images built from master (or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md.
  • If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
  • Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
    • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
  • Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
  • If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
  • If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
  • Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

  • Check 3 (Upstream recipe merged and complete): vllm-project/recipes#1254 and vllm-project/vllm#67378
  • Check 2 (Evals pass, accuracy): Upstream vLLM #64715 reports a two-pass GSM8K test with GPU prefix-cache reset between passes. The restore pass achieved 92.95% accuracy with 92.7% of prompt tokens restored from UMBP, versus 92.12% without the connector. This provides supplementary restore-correctness evidence; the upstream TP2 test does not directly establish restore coverage for this PR’s DP2/fp8_e4m3 evaluation.

Signed: @chunfangamd

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

❌❌❌ REJECTED ❌❌❌

⚠️ Pareto coverage needs additional review. At least 5 points per affected throughput-versus-E2EL frontier are highly recommended. Below 5, or when coverage cannot be verified, merge only with an explicit, recorded admin bypass for the assessed commit; this advisory comment does not grant or enforce a bypass.

@chunfangamd Blocking: no eval shows that the UMBP KV-restore path ran. That path is this PR's conditional backend change, and the eval in run 43861172833 cannot show it was exercised.

❌ Check 2 (Evals pass): FAIL — No eval covers the changed conditional path (UMBP restore of offloaded fp8_e4m3 KV). The eval-only job (run 43861172833, job 133481016316, SHA 94b81ec4) launched with UMBPStoreConnector and --kv-cache-dtype fp8_e4m3. But a single pass of GSM8K (1319 prompts) at c384 fits in the GPU prefix cache, so a pool load happens only on an eviction. Nothing in the job log shows a UMBP restore, and the server-log tarball's per-worker connector hit counters could not be inspected here. The sign-off's own detail section admits the upstream two-pass test (vllm#64715, TP2, fp8_ds_mla) "does not directly establish restore coverage for this PR's DP2/fp8_e4m3 evaluation". Add an eval that forces restores (for example, store pass, /reset_prefix_cache, then a restore pass) and attach logs showing nonzero UMBP-restored tokens. Parts (a) and (b) pass: the image is right, and gsm8k em_strict is 0.9727±0.0045 against the cohort's 0.9780±0.0040 (run 42335637576). That gap is 0.0053, under the 2σ threshold of 0.0121.

⚠️ Check 13 (Draft precision): WARN — Draft precision could not be verified as required: the sign-off's additional detail section does not record the DSpark draft's checkpoint, stored precision, the pinned image's default handling, or the effective precision. Independent evidence points to compliance but is incomplete. The DeepSeek-V4.1-Flash config.json has quant_method: fp8 / expert_dtype: fp4 with no draft exclusion (num_nextn_predict_layers: 3). The launched --speculative-config has no quantization or dtype fields. kv-cache-dtype fp8_e4m3 applies consistently to target and draft, which is allowed. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not set (vLLM path). How nightly-60bd6dd loads mtp.0-2 by default was not inspected, so human review is needed.

⚠️ Check 14 (Pareto coverage): WARN — curve dsv41flash / agentic-coding / mi355x_vllm / fp4 / 2026-11-16 / p90 E2EL has 3/5 frontier points (c128 36.64 s at 271,329 tok/s/GPU; c256 67.95 s at 401,896; c384 103.56 s at 427,451), all measured in run 43861172833 at SHA 94b81ec4. No admin bypass found (admin bypass not verified); an explicit admin exception for SHA 7ef99702 and this curve is required.

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @chunfangamd is a named owner of inferencex-e2e/configs/amd-master.yaml and of the mi355x-* srt-slurm recipe tree. runners.yaml and perf-changelog.yaml fall under the catch-all.

✅ Check 1 (In-PR sweep + evals): PASS — In-PR commit 94b81ec4 has green agentic / c128/c256/c384 and agentic eval / c384 check runs (run 43861172833). The later merges e300db6d, 79129d38 and 7ef99702 are non-material: the PR's recipe, master key and runners.yaml are net-unchanged, and the only launcher change (the head_frontend gate in recipe.py/lanes.py) does not reach the mi355x-amds lane.

✅ Check 3 (Upstream recipe): PASS — vllm#67378 (merged 2026-11-13) adds the mi355x_agentx_dp2_umbp preset, and recipes#1254 (merged 2026-11-13) adds the UMBP option and server companion. Together they cover DP2/no-EP, Engram host tables, MXFP4 indexer, block 128, fp8_e4m3 KV, prefill interval 11, batched tokens 16384, cudagraph 1024, the AMD AITER/breakable-cudagraph env, QR NONE, DSpark spec config, the consistent-hash router, and per-point UMBP on/off. Informational: the preset pins nightly-be47189 rather than nightly-60bd6dd (which adds #67377), and /umbp-host mount and router telemetry flags are InferenceX-only.

✅ Check 4 (Reuse command): PASS — /use 43861172833 was posted by chunfangamd (permission write, role write) on 2026-11-17, and it names the Check 1 run.

✅ Check 5 (Checklist template): PASS — Every current PR_REVIEW_CHECKLIST.md item has a checked counterpart; the main-branch provenance item differs only in wording.

✅ Check 6 (Upstream images): PASS — vllm/vllm-openai-rocm:nightly-60bd6dd2…@sha256:67a3a0fb… is from the upstream vllm org, and vLLM commit 60bd6dd2 is on main (170 commits behind the tip). vLLM-only entry, so no ordering issue.

✅ Check 7 (Deprecated models): PASS — dsv41flash agentic coding is active in MODELS.md with no deprecation.

✅ Check 8 (No architecture hacks): PASS — No --hf-overrides or architecture edits. VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE is in the worker env. The fp8_e4m3 KV is eval-covered at c384.

✅ Check 9 (Chat template): PASS — AgentX replay drives the server through the chat endpoint with the deepseek_v41 tokenizer and reasoning parser.

✅ Check 10 (No engine patches): PASS — No patches. The vllm-router.sh setup installs the released vllm-router==0.1.14, and the UMBP service runs the umbp_standalone_server shipped in the image's MORI package.

✅ Check 11 (Golden AL): PASS — Benchmark launch uses rejection_sample_method: synthetic, synthetic_acceptance_length: 3.51, which equals golden deepseek-v4.1-flash__dspark__native__probabilistic-block (5 tokens, thinking_on, block verify).

➖ Check 12 (Append-only): N/A — The new changelog entry has no append-only: true.

✅ Check 15 (PR description): PASS — Title, body and changelog match the pinned config: image, DP2/TP1 topology, c128 without offload, UMBP at c256/c384 with a 449 GB pool, fp8_e4m3 KV, eval at c384, and other keys not rerun.

➖ Check 16 (NVIDIA overview): N/A — The only affected hardware is AMD MI355X.

➖ Check 17 (Metrics publication): N/A — No Dynamo or TRT-LLM serving path is affected.

✅ Check 18 (Recipe consistency): PASS — All 3 points generate. Master tp:2, dp-attn matches the recipe's TP1×DP2 on 2 GPUs, and the conc and KV-offload fields match. The image digest resolves. Nothing is removed or re-tuned and there are no regressions. Stream interval is the vLLM default (1, at most 60).

Assessed commit: 7ef99702001c6efab33ff0ff25dbffb7f9562c86.

…h-umbp

# Conflicts:
#	inferencex-e2e/perf-changelog.yaml

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim)

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

4 participants