Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
887dbc3
[DBO] Size every microbatch's workspace after the memory profile
ChengYM5685 Oct 31, 2026
a2bcb44
[DBO][ROCm] Create hipBLASLt workspaces on the model's side streams b…
ChengYM5685 Nov 1, 2026
5872e3a
[DBO] Run a microbatched step eager when a rank's last microbatch is …
ChengYM5685 Nov 2, 2026
ffee11a
[DBO] Count microbatched CUDA graphs in the V1 memory profile
ChengYM5685 Nov 4, 2026
68207bf
[ROCm][DSV4] Handle a DBO microbatch of pure DP padding
ChengYM5685 Oct 31, 2026
1b517e1
[MORI] Support dual batch overlap with MORI EPv2
ChengYM5685 Oct 31, 2026
b971627
[MORI] Size later DBO microbatches' EPv2 ops to their share of the batch
ChengYM5685 Nov 1, 2026
f275a02
[MORI] Run MORI EPv2 on AsyncLL under MORI_ENABLE_SDMA, with DBO
ChengYM5685 Nov 4, 2026
76e7885
Merge branch 'main' into feat/mori-epv2-dbo
vllm-agent Nov 6, 2026
79179bf
Merge branch 'main' into feat/mori-epv2-dbo
ChengYM5685 Nov 9, 2026
a0043d3
Merge branch 'main' into feat/mori-epv2-dbo
vllm-agent Nov 9, 2026
cfeb6ef
Merge branch 'main' into feat/mori-epv2-dbo
ChengYM5685 Nov 9, 2026
405d464
[MORI] Keep EPv1 under MORI_ENABLE_SDMA=1 unless VLLM_MORI_EP_V2=1
ChengYM5685 Nov 11, 2026
b3383ec
Merge branch 'main' into feat/mori-epv2-dbo
vllm-agent Nov 11, 2026
52d3204
Merge branch 'main' into feat/mori-epv2-dbo
ChengYM5685 Nov 11, 2026
e7082d5
Merge branch 'main' into feat/mori-epv2-dbo
vllm-agent Nov 11, 2026
30f4fed
Merge branch 'main' into feat/mori-epv2-dbo
ChengYM5685 Nov 13, 2026
0e37106
Merge branch 'main' into feat/mori-epv2-dbo
vllm-agent Nov 13, 2026
88a39fd
Merge branch 'main' into feat/mori-epv2-dbo
ChengYM5685 Nov 14, 2026
841a7ee
Merge branch 'main' into feat/mori-epv2-dbo
ChengYM5685 Nov 15, 2026
dc0c34b
Merge branch 'main' into feat/mori-epv2-dbo
vllm-agent Nov 15, 2026
c99c1f3
Merge branch 'main' into feat/mori-epv2-dbo
ChengYM5685 Nov 16, 2026
73ff6d1
Merge branch 'main' into feat/mori-epv2-dbo
ChengYM5685 Nov 16, 2026
680f9fa
[Core][DBO] Allow dual batch overlap with pipeline parallelism
ChengYM5685 Nov 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/design/dbo.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ To enable the DBO system pass in the `--enable-dbo` argument to your vllm serve
* `--dbo-decode-token-threshold` the minimum number of tokens in a decode-only batch required to enable DBO for that batch
* `--dbo-prefill-token-threshold` the minimum number of tokens in a batch containing at least one prefill required to enable DBO for that batch

Currently, DBO is only supported with DeepEP, so DeepEP must be installed and the `--all2all-backend` argument must be set to `deepep_low_latency` if your workload is primarily decode requests, or `deepep_high_throughput` if your workload is primarily prefill requests.
DBO needs an all2all backend that supports it: `deepep_low_latency` if your workload is primarily decode requests or `deepep_high_throughput` if it is primarily prefill requests (both need DeepEP installed), or `nixl_ep`. On ROCm, `allgather_reducescatter` and `mori` (when it runs MoRI EPv2, see [MoRI EP](../serving/expert_parallel_deployment.md#mori-ep-rocm)) also support it.

Below is a command that will spin up a two DP rank server with expert parallelism and DBO enabled.
EX: `vllm serve deepseek-ai/DeepSeek-V2-Lite --trust-remote-code --data-parallel-size 2 --enable-expert-parallel --enable-dbo --all2all-backend deepep_low_latency`
Expand All @@ -43,7 +43,7 @@ Note that there must be at least two GPUs visible in `CUDA_VISIBLE_DEVICES`

## Limitations

DBO cannot be combined with LoRA, pipeline parallelism, prefill context parallelism, multimodal, encoder-decoder or hybrid models, or encoder-only models. Multimodal models are supported under `all-reduce-overlap`. CUDA graphs must be Full or disabled: the runner rejects `PIECEWISE`.
DBO cannot be combined with LoRA, prefill context parallelism, multimodal, encoder-decoder or hybrid models, or encoder-only models. Multimodal models are supported under `all-reduce-overlap`. CUDA graphs must be Full or disabled: the runner rejects `PIECEWISE`. With pipeline parallelism, each stage microbatches its own forward pass (non-first stages slice the received intermediate tensors per microbatch) and the stages still send and receive whole batches.

## DBO Components

Expand Down
6 changes: 5 additions & 1 deletion docs/serving/expert_parallel_deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,10 +112,14 @@ vllm serve deepseek-ai/DeepSeek-V4-Pro \

| `VLLM_MORI_EP_V2` | Behavior |
| ----------------- | -------- |
| unset | EPv2, falling back to EPv1 with a warning when MoRI is older than 1.2.4, the GPU is not gfx950 or gfx1250, EP spans hosts (except on gfx1250, whose UALink team can span hosts), `MORI_ENABLE_SDMA=1`, `--moe-backend aiter_mega_moe`, the model is not BF16, or MoE activations use per-tensor FP8 |
| unset | EPv2, falling back to EPv1 with a warning when MoRI is older than 1.2.4, the GPU is not gfx950 or gfx1250, EP spans hosts (except on gfx1250, whose UALink team can span hosts), `MORI_ENABLE_SDMA=1` on gfx950 (EPv2's AsyncLL kernels are opt-in, see below), `--moe-backend aiter_mega_moe`, the model is not BF16, or MoE activations use per-tensor FP8 |
| `1` | Always EPv2. The checks above only log a warning; startup fails if the EP ranks are not in one CCO-LSA team |
| `0` | EPv1 |

On gfx950, `MORI_ENABLE_SDMA=1` moves tokens by SDMA on EPv1 by default. With `VLLM_MORI_EP_V2=1` as well, EPv2 runs its AsyncLL kernels instead, which move tokens by SDMA, when the installed MoRI has them. gfx1250 keeps native EPv2.

On EPv2, `mori` supports `--enable-dbo` with native and AsyncLL kernels. DBO speeds up prefill but slows decode here, so set a large `--dbo-decode-token-threshold` or enable DBO on prefill instances only.

On EPv1, `mori` uses the IntraNode kernel within one node. Across nodes it uses InterNodeV1LL when the per-rank token budget (`--max-num-batched-tokens`) is at most `VLLM_MORI_DISPATCH_INTER_KERNEL_SWITCH_THRESHOLD` (default 256), and InterNodeV1 otherwise. `mori_high_throughput` and `mori_low_latency` always run EPv1 and pin the inter-node kernel to InterNodeV1 and InterNodeV1LL respectively.

Choosing a MoRI backend: use EPv2 wherever it applies, and EPv1 for deployments EPv2 does not support. EPv2 cuts its receive buffers to the rows a step can actually receive and lets AITER fused MoE write its output straight into the combine buffer, so it does less work per step than EPv1.
Expand Down
Loading
Loading