Skip to content

[ROCm][Docs] Add DeepSeek-V4.1-Flash MI355X DP2 AgentX preset with UMBP KV offload - #67378

Merged
esmeetu merged 2 commits into
vllm-project:mainfrom
shen-shanshan:recipes-dsv41flash-mi355x-umbp-offload
Nov 13, 2026
Merged

esmeetu merged 2 commits into
vllm-project:mainfrom
shen-shanshan:recipes-dsv41flash-mi355x-umbp-offload

Conversation

@shen-shanshan

@shen-shanshan shen-shanshan commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Adds the mi355x_agentx_dp2_umbp preset to recipes/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml for InferenceX #5036 (dsv41flash-fp4-mi355x-vllm-agentic-dspark-umbp), and records UMBP as verified for the model.

Claims

Validation

  • From a vllm-project/recipes checkout with the umbp option, node scripts/validate-source.mjs --source vllm --dir <this branch>/recipes passes: ✓ vllm recipes … are valid against the renderer.

  • The rendered command matches the InferenceX recipe flag for flag: the per-point vllm serve / docker run, the UMBP Server companion (socket address equal to the connector endpoint) and the vLLM Router (--intra-node-data-parallel-size 2). At 128 the command has no --kv-transfer-config and no UMBP Server tab; 256 and 384 have both. pnpm test:presets and pnpm test:kv-offload pass with this branch as the vLLM source.

  • Local MI355X AgentX replays (agentx-fast, 1200 s, single runs) of this configuration on the default fp8_ds_mla KV and an earlier nightly, against the same configuration without offloading:

    Concurrency Total tok/s/GPU, no offload → UMBP Prompt tokens restored from UMBP
    128 (served without offload) 193,099 → 183,984 (−4.7%) 0.46%
    256 250,435 → 275,836 (+10.1%) 0.67%
    384 143,888 → 291,619 (+103%) 21.5%

    The fp8_e4m3 KV cache and the pinned nightly are qualified by the InferenceX sweep of [Kernel][RFC] Refactor the punica kernel based on Triton #5036.

Details

  • single_node_dep does not inherit the MI355X kernels the TP rows take from strategy_overrides.single_node_tp (MXFP4 indexer, block size 128, AITER sparse MLA), so the preset restates them. --monokernel stays off, since mono decode rejects DP.
  • The preset is opt-in, not default: true: it is a benchmark lane for high concurrency, where the working set outgrows GPU KV. At concurrency 128 UMBP did not help, so that point turns the KV Offload row off while keeping the rest of the lane.
  • The runbook records the pool sizing InferenceX uses (449 GB per two-GPU server) and the evidence above, with no launch commands, as SCHEMA.md requires.

Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity. (Recipe YAML only; reuses the DEP strategy, its router companion and the preset delta fields.)

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

AI assistance: drafted with claude-opus-5-5[1m] (Claude Opus 5.5, 1M context) via Claude Code, which read the recipe and renderer, wrote the preset, ran the validation and wrote this description; reviewed by a human.

🤖 Generated with Claude Code

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify

mergify Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

@mergify mergify Bot added recipe deepseek Related to DeepSeek models rocm Related to AMD ROCm labels Nov 13, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Nov 13, 2026
@shen-shanshan
shen-shanshan force-pushed the recipes-dsv41flash-mi355x-umbp-offload branch from 45b3afe to c7ba160 Compare November 13, 2026 06:52
github-actions Bot added a commit to vllm-project/recipes that referenced this pull request Nov 13, 2026
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@shen-shanshan

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #102400 for commit c7ba1601a541.

shen-shanshan and others added 2 commits November 13, 2026 10:13
…BP KV offload

Add the mi355x_agentx_dp2_umbp preset for InferenceX vllm-project#5036
(dsv41flash-fp4-mi355x-vllm-agentic-dspark-umbp): two GPUs per server with
attention DP2 and the MoE tensor-parallel across both ranks, behind the
consistent-hash vLLM Router, at concurrency 256 and 384. It selects the UMBP
KV Offload option (UMBPStoreConnector, standalone server) and serves a
per-tensor fp8_e4m3 KV cache with prefill cadence 11 on the nightly that
carries vllm-project#64715 and vllm-project#66036.

The preset restates the MI355X kernels the TP rows take from
strategy_overrides.single_node_tp, since single_node_dep does not inherit
them, and drops --enable-expert-parallel. kv_offload_support records UMBP as
verified for this model.

The `umbp` KV Offload option is added to the renderer's taxonomy in
vllm-project/recipes; this recipe validates only once that lands.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: shen-shanshan <467638484@qq.com>
…28 without offloading

InferenceX vllm-project#5036 now also runs DP2 at concurrency 128, carried over from the
closed vllm-project#4992, with no KV offloading: UMBP did not help there in local runs,
since the GPU KV cache holds the working set. Add that point to the
mi355x_agentx_dp2_umbp preset with `kv_offload: off`; 256 and 384 keep UMBP.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: shen-shanshan <467638484@qq.com>
@shen-shanshan

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #102456 for commit ad77d06dcea0.

github-actions Bot added a commit to vllm-project/recipes that referenced this pull request Nov 13, 2026
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@esmeetu
esmeetu merged commit 10a854e into vllm-project:main Nov 13, 2026
15 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Nov 13, 2026
vllm-recipes-publisher Bot pushed a commit to vllm-project/recipes that referenced this pull request Nov 13, 2026
vllm-project/vllm@main:recipes c410543d9fff → 10a854ed6b3b [latest.vllm]
vllm-project/vllm@c410543...10a854e

Commits:
- vllm-project/vllm@10a854ed6b3b [ROCm][Docs] Add DeepSeek-V4.1-Flash MI355X DP2 AgentX preset with UMBP KV offload (vllm-project/vllm#67378)

Recipes:
- deepseek-ai/DeepSeek-V4.1-Flash (modified): https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4.1-Flash

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
github-actions Bot added a commit to wombat/recipes that referenced this pull request Nov 13, 2026
…mni@2c148ceb1373, vllm-project/vllm-ascend@775899991f96

vllm-project/vllm@main:recipes c410543d9fff → cfd86e432065 [latest.vllm]
vllm-project/vllm@c410543...cfd86e4

Commits:
- vllm-project/vllm@10a854ed6b3b [ROCm][Docs] Add DeepSeek-V4.1-Flash MI355X DP2 AgentX preset with UMBP KV offload (vllm-project/vllm#67378)

Recipes:
- deepseek-ai/DeepSeek-V4.1-Flash (modified): https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4.1-Flash

vllm-project/vllm-omni@main:recipes 4dee28cac13d → 2c148ceb1373 [latest.vllm-omni]
vllm-project/vllm-omni@4dee28c...2c148ce

Commits:
- vllm-project/vllm-omni@6c8f69303faf [Bugfix][Qwen-Image-2.1] Fix decode graph pool lifetime and stream recovery (vllm-project/vllm-omni#10268)
- vllm-project/vllm-omni@b5d54010ed06 [Doc][NPU] Add Atlas 800T A3 performance to the MiniMax-H3-Image recipe (vllm-project/vllm-omni#10292)

vllm-project/vllm-ascend@main:recipes 02ac27a19cb7 → 775899991f96 [latest.vllm-ascend]
vllm-project/vllm-ascend@02ac27a...7758999

Commits:
- vllm-project/vllm-ascend@931bfd948e04 [Doc][Recipe] Add Qwen3-VL-Reranker-2B Ascend recipe (vllm-project/vllm-ascend#20817)
- vllm-project/vllm-ascend@5e9559edb202 [Doc][Recipe] Add Qwen3-Embedding-0.6B Ascend recipe (vllm-project/vllm-ascend#20816)
- vllm-project/vllm-ascend@61d127541ffd [BugFix][Recipes] Fix Ascend recipe TP/DP configuration (vllm-project/vllm-ascend#21808)
- vllm-project/vllm-ascend@63134f7c6b99 [Doc][Recipe] Add Ascend overlay for Qwen3-VL-235B-A22B-Instruct (vllm-project/vllm-ascend#21553)
- vllm-project/vllm-ascend@0d8b1f8ba171 [Doc][Recipe] Add Ascend overlay for Qwen3-Next-80B-A3B-Instruct (vllm-project/vllm-ascend#21552)
- vllm-project/vllm-ascend@7e18331d207f [Doc][Recipe] Add historical Qwen3-Reranker Ascend deployment (vllm-project/vllm-ascend#20994)
- vllm-project/vllm-ascend@3d3a5c038bdd [Doc][Recipes] Add MiniMax-M3 Ascend overlay (vllm-project/vllm-ascend#21065)
- vllm-project/vllm-ascend@548ee4c116f0 [Doc][Recipe] Add Ascend overlay for Qwen3.6-35B-A3B (vllm-project/vllm-ascend#20808)

Recipes:
- Qwen/Qwen3-Embedding-0.6B (added): https://recipes.vllm.ai/Qwen/Qwen3-Embedding-0.6B
- Qwen/Qwen3-Reranker-0.6B (added): https://recipes.vllm.ai/Qwen/Qwen3-Reranker-0.6B
- Qwen/Qwen3-VL-Reranker-2B (added): https://recipes.vllm.ai/Qwen/Qwen3-VL-Reranker-2B
- MiniMaxAI/MiniMax-M3 overlay (added): https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3
- Qwen/Qwen3-Next-80B-A3B-Instruct overlay (added): https://recipes.vllm.ai/Qwen/Qwen3-Next-80B-A3B-Instruct
- Qwen/Qwen3-VL-235B-A22B-Instruct overlay (added): https://recipes.vllm.ai/Qwen/Qwen3-VL-235B-A22B-Instruct
- Qwen/Qwen3.5-397B-A17B overlay (modified): https://recipes.vllm.ai/Qwen/Qwen3.5-397B-A17B
- Qwen/Qwen3.6-27B overlay (modified): https://recipes.vllm.ai/Qwen/Qwen3.6-27B
- Qwen/Qwen3.6-35B-A3B overlay (added): https://recipes.vllm.ai/Qwen/Qwen3.6-35B-A3B
- deepseek-ai/DeepSeek-V4-Flash overlay (modified): https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

This branch was previously deployed

1 inactive deployment
recipes-preview — ad77d06d Deployed Nov 13, 2026 by esmeetu via dispatch #564
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek Related to DeepSeek models recipe rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants