[Perf][PP] Copy-engine PP transport, on by default where supported (NVLink only) by gau-nernst · Pull Request #65896 · vllm-project/vllm

gau-nernst · 2026-11-05T11:14:57Z

Human note: This PR adds copy engine option for PP handoff from one rank to the next. The copy is one-sided by nature (sender sets a flag after sending the data, receiver releases a flag once finished consuming it). The main goals are (1) 0-SM copy to remove contention with compute kernels, especially persistent kernels sized by number of SMs (instead of CLC for example), and (2) replace NCCL rendezvous with GPU-side wait

AMD equivalent solution: #65658

Overview

Moves PP activations between stages with CUDA copy engines instead of NCCL send/recv. Selected by VLLM_PP_TRANSPORT:

auto (default): copy engines if every PP rank supports them, NCCL otherwise. All ranks make the same choice, and the log says which and why.
nccl: always NCCL.
copy_engine: always copy engines; startup fails if they are not usable.

Summary

Near saturation, median TTFT drops 13–20% with 1.5–10K new tokens and 1–12% with 16–30K (100K cached prefix), and p99 drops with it (by 16–19% and 3–6% at the highest loads).
At light load, copy engines are no slower than NCCL: at the lowest load of each workload, the paired median per request is between −14 and +0.1 ms.
The light-load regression of earlier revisions is fixed. The copy-engine receive waited for the previous stage before the model runner's input preparation, so that GPU work could not overlap the transfer, at every stage boundary. The receive now queues its wait and copy when the runner first reads the tensors, as the NCCL path does (AsyncIntermediateTensors). At PP=4 this took light-load TTFT from +7–9 ms to NCCL's level and decode ITL from +4% to NCCL's.
Outputs match NCCL exactly (temperature 0, tokens and logprobs, prompts up to 8K tokens).

Known weaknesses

Few setups measured: one model at PP=2 and PP=4, on GB300. auto also turns copy engines on for unmeasured setups (e.g. H100/H200 with NVLink; B300 was only measured before the receive fix).
Why copy engines help near saturation is not established (see Details).

Validation

MiniMaxAI/MiniMax-M3.1-Flash, TP=1, GB300 (NVLink), Rust frontend, with the deferred receive.

Synthetic load: Poisson arrivals at the given mean rate for 120 s, replayed open-loop. Each request is the same 100K-token prefix (cached) + new tokens drawn uniformly from the stated range + 1 output token. One unscored warm-up per workload; a prefix-cache reset and re-prime before every scored pass.

Setup: 2 nodes, one seed each. On each node, NCCL and copy engines run at the same time on separate GPU pairs (PP=2 each) with identical requests, so both arms see the same node conditions; the GPU pairs are swapped between the two nodes. No request failed in either arm (~38K scored requests per arm).

Columns: each cell gives node 1, node 2. "Change" is copy engine vs NCCL. "Paired median" is the median over requests of (copy-engine TTFT − NCCL TTFT) for the same request; its bootstrap 95% CI excludes zero at most points.

Server command

VLLM_USE_RUST_FRONTEND=1 vllm serve MiniMaxAI/MiniMax-M3.1-Flash --pipeline-parallel-size 2 \
  --block-size 128 --disable-hybrid-kv-cache-manager --language-model-only --enable-prefix-caching \
  --max-model-len 1048576 --max-num-batched-tokens 32768 --max-num-seqs 64 \
  --gpu-memory-utilization 0.9 --trust-remote-code

VLLM_PP_TRANSPORT=nccl for the baseline. The copy-engine arm logs PP transport: copy engines.

TTFT, 100K cached + 1.5–2.5K new

Load	p50 NCCL	p50 change	p99 NCCL	p99 change	Paired median, ms
8 req/s	234 / 231 ms	−1%, −1%	632 / 488 ms	−1%, +1%	−2.6, +0.1
12 req/s	354 / 396 ms	+5%, +2%	1134 / 1368 ms	+7%, −1%	−5.4, −1.8
16 req/s	902 / 918 ms	−11%, −9%	1922 / 1889 ms	−4%, −1%	−45, −12
19 req/s	1238 / 1226 ms	+1%, −3%	1974 / 1927 ms	−3%, −1%	−34, −18
22 req/s	1238 / 1406 ms	−3%, −15%	1989 / 2267 ms	−1%, −16%	−12, −237
25 req/s	1569 / 1536 ms	−20%, −19%	2400 / 2368 ms	−19%, −18%	−373, −274

TTFT, 100K cached + 6–10K new

Load	p50 NCCL	p50 change	p99 NCCL	p99 change	Paired median, ms
3 req/s	691 / 564 ms	0%, 0%	1879 / 1700 ms	−2%, −1%	−12, −1.6
4.5 req/s	1214 / 1046 ms	−12%, −1%	2352 / 1854 ms	−10%, 0%	−18, −7.6
5.5 req/s	1519 / 1379 ms	−10%, −6%	2449 / 2138 ms	−11%, −8%	−119, −9.3
6 req/s	1605 / 1515 ms	−8%, −15%	2713 / 2318 ms	−4%, −7%	−60, −104
6.5 req/s	1755 / 1614 ms	−13%, −14%	2969 / 2531 ms	−4%, −15%	−117, −222
7 req/s	1892 / 1892 ms	−15%, −16%	3694 / 3200 ms	−18%, −16%	−280, −354

TTFT, 100K cached + 16–30K new

Load	p50 NCCL	p50 change	p99 NCCL	p99 change	Paired median, ms
1 req/s	1122 / 900 ms	+2%, 0%	2033 / 2275 ms	+1%, +1%	−14, −2.8
1.5 req/s	1519 / 1361 ms	−5%, −1%	2837 / 2480 ms	−3%, −4%	−13, −6.4
1.8 req/s	1864 / 1475 ms	−5%, −5%	3588 / 3039 ms	−4%, 0%	−25, −10
2.1 req/s	2243 / 1788 ms	−8%, −8%	5191 / 3516 ms	−13%, −2%	−69, −68
2.4 req/s	2813 / 2099 ms	−1%, −12%	6120 / 3912 ms	−1%, −8%	−34, −112
2.7 req/s	4646 / 3524 ms	−4%, −7%	8924 / 5007 ms	−3%, −6%	−341, −259

At 12 req/s with 1.5–2.5K new tokens, node 1's p50 and p99 rise (+5%, +7%) while its paired median falls (−5.4 ms): this rate sits at the knee, where bursts move the percentiles. Earlier revisions saw the same at this point on one of three loads.

PP=4, light load and decode (same model, 4 GPUs, 90 s passes at 8 req/s with 1.5–2.5K new tokens on the 100K prefix; servers run one after another on one node):

Server	TTFT p50	Paired median vs NCCL #1	Decode ITL p50 (128 output tokens)
NCCL #1	204 ms	—	41.66 ms
NCCL #2	200 ms	−0.9 ms	41.64 ms
Copy engines, receive before input prep (earlier revisions)	211 ms	+7.3 ms	43.49 ms
Copy engines, deferred receive #1	206 ms	+2.0 ms	42.05 ms
Copy engines, deferred receive #2	204 ms	−1.2 ms	41.36 ms

Outputs: temperature-0 completions (16 tokens, with logprobs) for prompts of 1.5K–8K tokens are identical between NCCL and copy engines at PP=2 and PP=4. Longer prompts were not compared: on this model they differ between two runs on the same NCCL server.

GSM8K (before the receive fix; 8x B300, 5-shot, chat, greedy, 1319 questions):

	Server 1	Server 2
NCCL	0.955	0.958
copy engine	0.958	0.958

A paired per-question McNemar test gives p = 0.74 and p = 1.0, so there is no difference.

Tests: tests/distributed/test_pp_transport.py covers 2- and 4-stage chains with varying sizes and lagging stages, and a 4-stage chain where one stage fails setup: every rank raises under copy_engine, every rank falls back to NCCL under auto, and the memory is freed. The scenarios share one Ray worker per GPU, so the file takes ~36 s on 4x GB200.

Details

How it works: each stage shares a receive buffer over CUDA IPC. The sender packs its outputs on the main stream; a side stream peer-copies them (cudaMemcpyAsync) and sets a "full" flag (cuStreamWriteValue64). The receiver waits on that flag (cuStreamWaitValue64), copies the data out and sets an "empty" flag, queued when the model runner first reads the tensors. Metadata goes over the CPU group.

Requirements: CUDA, TP=1, PCP=1, one node, 64-bit stream memory ops, and NVLink between all PP GPUs. Checks that can differ between ranks run before an all-ranks error exchange, so all ranks pick the same transport; on fallback, the buffers are freed before memory profiling.

Why NVLink: x86 hosts don't support CU_STREAM_WAIT_VALUE_FLUSH, so the flag waits run without it and rely on NVLink ordering, as NCCL's zero-SM copy-engine path does.

Memory: a send and a receive buffer per stage boundary, each the size of the persistent intermediate tensors. Memory profiling counts them.

Why it might help (not established): NCCL send/recv runs kernels on SMs on both stages, and the receiver's kernel can stay on the GPU for milliseconds per step; copy engines use no SMs. An nsys profile at light load did not show model kernels running slower next to NCCL. NCCL's send also waits for the receiver once a tensor exceeds its eager threshold (64–96 MiB here, i.e. 5.5–8K tokens per step), which stalls the sending stage near saturation; copy engines never wait on the CPU.

Follow-ups, not in this PR: TP>1, PP across nodes over multi-node NVLink, and freeing the buffers in sleep mode.

Pull Request Checklist

I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).
AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

mergify · 2026-11-05T12:30:10Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @gau-nernst.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

njhill · 2026-11-05T20:49:18Z

Maybe we could make this deafult?

gau-nernst · 2026-11-06T08:26:20Z

@njhill There are certain conditions that this PR is consistently worse (e.g. 1.5-2.5K new inputs, 8 QPS are consistently 2% slower). I'm not sure what's the reason. Hence I'm a bit hesitant to make this the default. But of course if you think it's fine, we can switch it on by default (and listen to issue reports to harden it)

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

gau-nernst · 2026-11-06T08:51:13Z

/ci run

github-actions · 2026-11-06T08:51:42Z

✅ Triggered Buildkite CI #100516 for commit 67412ccfb54e.

ℹ️ This PR is 1 commit behind upstream main. Merged upstream main into this branch as 67412ccfb54e before starting CI. Pull the branch before pushing more changes.

mergify · 2026-11-06T15:06:23Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @gau-nernst.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

njhill · 2026-11-08T06:46:11Z

Thanks @gau-nernst, this looks great. I made some further changes with claude here which recovers the regression at the low load / shorter query cases, this should also apply to the AMD version: njhill@e351d23

Probably the case for making this default is then stronger - it could fall back to NCCL when not supported. Also I think we should probably be able to extend this to support TP>1 and MNNVL?

A non-last PP stage sends its intermediate tensors with NCCL isend, and a send reads its source until the next stage posts the matching receive. The worker therefore waits for the previous step's send before each forward, since the forward may overwrite the tensors that send is still reading. Send from a ring of two buffers sized like the profile run's outputs instead: the next forward can overwrite its outputs at once, and a send is only waited when its buffer comes around again. The buffers are allocated after the profile run and added to the non-KV-cache memory. The hand-off moves behind a small PPTransport interface (vllm/distributed/pp_transport), with NCCL as its only backend for now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>

NCCL send/recv runs a kernel on SMs on both stages of a PP boundary. Add a copy_engine PPTransport backend: peer cudaMemcpyAsync into the next stage's CUDA IPC receive buffer, synchronized with GPU-side flags written and awaited with stream memory operations, so neither SMs nor the CPU wait on the transfer. The flags are awaited without CU_STREAM_WAIT_VALUE_FLUSH, which x86 hosts (e.g. HGX B300) do not support; as in NCCL's zero-SM RMA path, the backend therefore requires the PP group's GPUs to be NVLink-connected. VLLM_PP_TRANSPORT selects the backend: nccl (default), auto (copy engines when every stage qualifies, i.e. CUDA, V2 model runner, TP=1, PCP=1, one node, NVLink; NCCL otherwise), or copy_engine (fail startup instead of falling back). Setup is a collective, so all ranks pick the same backend. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>

recv() queued the wait for the previous stage and the copy into the persistent tensors on the main stream before the model runner's input preparation, so that GPU work waited for the transfer at every stage boundary. Return AsyncIntermediateTensors instead, queuing the wait, copy and empty-flag write when .tensors is first read, as the NCCL path does. At PP=4 on MiniMax-M3.1-Flash (GB300) this removes the light-load TTFT and decode ITL regression vs NCCL (decode ITL p50 43.5 -> 41.4-42.1 ms, NCCL 41.6 ms). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xgy3B6LNvoZJuQY4nadJiG Signed-off-by: Nick Hill <nickhill123@gmail.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>

VLLM_PP_TRANSPORT now defaults to "auto": copy engines when every PP rank can set them up, NCCL otherwise. Each check that can differ between ranks (cuda-python, stream memory ops, buffer allocation, NVLink, opening the peers' buffers, the send buffer) runs before an all-ranks exchange of errors, so every rank makes the same choice; on fallback each rank frees its buffers and closes its peer mappings before memory profiling. "copy_engine" still fails startup, on every rank alike. The startup-plan fingerprint records the transport in use rather than the env value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>

Import _custom_ops inside the Ray worker of the one-rank-failure test, since Ray cannot pickle the module as a global. Pin the compiled-graphs load_model test to NCCL: it has no PP group, and "auto" would set up a PP transport. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>

gau-nernst · 2026-11-09T02:44:12Z

/ci run

github-actions · 2026-11-09T02:44:35Z

✅ Triggered Buildkite CI #100955 for commit 18007ee786c9.

ℹ️ This PR is 2 commits behind upstream main. Merged upstream main into this branch as 18007ee786c9 before starting CI. Pull the branch before pushing more changes.

…enarios One shared Ray actor per GPU now serves all three scenarios: the worker pays the vLLM import and CUDA context setup once, and each scenario only re-initializes and tears down its distributed environment. Tests keep their distinct names and run in one process (multi_gpu_marks instead of multi_gpu_test's per-test fork, which a shared Ray connection cannot survive). A failing scenario kills the pool so its peers fail fast instead of hanging on the abandoned collective, and the next test starts a fresh pool. 3 passed in 36s, down from 120s, on a 4x GB200 tray. Co-authored-by: Kimi Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>

njhill

Thanks @gau-nernst! I updated the PR description with the new results that I collected.

njhill · 2026-11-09T20:50:25Z

/ci run

github-actions · 2026-11-09T20:50:55Z

✅ Triggered Buildkite CI #101240 for commit e785660ce5db.

ℹ️ This PR is 79 commits behind upstream main. Merged upstream main into this branch as e785660ce5db before starting CI. Pull the branch before pushing more changes.

njhill · 2026-11-09T23:27:27Z

/ci retry

github-actions · 2026-11-09T23:27:46Z

✅ Triggered Buildkite CI #101285 for commit c986424d4bfe, running 7 failed step(s) from Buildkite CI #101240.

njhill · 2026-11-10T00:00:16Z

@gau-nernst I've opened #66677 based on top of this one to extend the support to TP>1 and MNNVL. And verified it improves performance in those cases too.

khluu

stamp for CI

vllm-project#65896 added the PPTransport interface (VLLM_PP_TRANSPORT) at the same hooks of gpu_worker.py; both conflicts are resolved to main's side, and the next commit moves the SDMA hand-off onto that interface. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Taoyu Zhu <zhutaoyu97@gmail.com>

Moves the SDMA hand-off onto vllm-project#65896's PPTransport instead of its own flag: `SdmaTransport` (vllm/distributed/pp_transport/sdma.py) subclasses `CopyEngineTransport` and keeps its protocol, buffers, metadata path and deferred receive; only the platform hooks differ. - copy_engine.py: the CUDA-only steps become overridable hooks (_platform_unavailable, _device_info, _setup_sender, _peer_copy, _wait_done, _wait_flag, _write_flag, _close_peer); CUDA behaviour is unchanged. - sdma.py: the peer copy is hsa_amd_memory_async_copy_on_engine on engines 2 and 4, queued by the host behind a dependency signal that the side stream opens after its `empty` wait; two 8-byte SDMA copies of the step number, queued behind the data copies, publish `full` to the receiver and `sent` to this rank (the next pack waits for `sent`). Signals rotate through a ring of 16 steps. Flags use the ROCm shared buffer, which is uncached. - VLLM_PP_TRANSPORT=copy_engine selects it on ROCm gfx950; "auto" keeps NCCL on ROCm. - Removes --pp-sdma-handoff, ParallelConfig.pp_sdma_handoff, pp_sdma.py and test_pp_sdma.py; test_pp_transport.py runs on MI355X too (MI355 2-GPU step). TODO (nothing run yet; the code is up for review first): - test_pp_transport.py on 2 and 4 x MI355X; MiniMax-M3.1-Flash TP1 x PP2 prefill vs NCCL, eager and with CUDA graphs; gsm8k. - Unverified on ROCm: hipStreamWriteValue64 into the previous stage's IPC-mapped uncached `empty` word; several copies decrementing one completion signal; torch.as_tensor over __cuda_array_interface__ (`_device_bytes`). - HIP maps streams onto 4 HW queues round-robin: if the side stream shares the compute stream's queue, its `empty` wait also stalls compute. - Whether "auto" should pick SDMA on gfx950. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Taoyu Zhu <zhutaoyu97@gmail.com>

mergify Bot added ci/build nvidia mrv2 Model Runner V2 specific labels Nov 5, 2026

github-project-automation Bot added this to NVIDIA Nov 5, 2026

gau-nernst force-pushed the pp-transport-copy-engine branch from f8131f1 to 8c5f7bb Compare November 5, 2026 12:29

mergify Bot added the needs-rebase label Nov 5, 2026

gau-nernst force-pushed the pp-transport-copy-engine branch from 8c5f7bb to 0abd809 Compare November 5, 2026 12:32

gau-nernst changed the title ~~[Perf][PP] Opt-in PP transport on CUDA copy engines~~ [Perf][PP] Opt-in PP transport on CUDA copy engines (NVLink only) Nov 5, 2026

gau-nernst marked this pull request as ready for review November 6, 2026 08:31

gau-nernst requested a review from a team November 6, 2026 08:31

gau-nernst requested review from Harry-Chen, WoosukKwon, khluu, njhill and yewentao256 as code owners November 6, 2026 08:31

claude Bot reviewed Nov 6, 2026

View reviewed changes

gau-nernst force-pushed the pp-transport-copy-engine branch from 0abd809 to 5406f62 Compare November 6, 2026 08:47

mergify Bot removed the needs-rebase label Nov 6, 2026

mergify Bot added the needs-rebase label Nov 6, 2026

gau-nernst force-pushed the pp-transport-copy-engine branch from 67412cc to a0030ef Compare November 7, 2026 01:28

mergify Bot removed the needs-rebase label Nov 7, 2026

gau-nernst and others added 2 commits November 9, 2026 01:39

njhill and others added 3 commits November 9, 2026 01:39

[Perf][PP] Single backticks in the deferred-receive docstring

70eab13

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>

gau-nernst force-pushed the pp-transport-copy-engine branch from e384498 to cd73a08 Compare November 9, 2026 01:39

gau-nernst changed the title ~~[Perf][PP] Opt-in PP transport on CUDA copy engines (NVLink only)~~ [Perf][PP] Copy-engine PP transport, on by default where supported (NVLink only) Nov 9, 2026

Merge branch 'main' into pp-transport-copy-engine

18007ee

njhill approved these changes Nov 9, 2026

View reviewed changes

Merge branch 'main' into pp-transport-copy-engine

e785660

Merge branch 'main' into pp-transport-copy-engine

c986424

njhill mentioned this pull request Nov 9, 2026

[Perf][PP] Copy-engine PP transport: TP > 1, PCP > 1 and multi-node NVLink #66677

Open

1 of 6 tasks

khluu approved these changes Nov 10, 2026

View reviewed changes

github-project-automation Bot moved this to Ready in NVIDIA Nov 10, 2026

njhill merged commit ce896d6 into vllm-project:main Nov 10, 2026
34 checks passed

github-project-automation Bot moved this from Ready to Done in NVIDIA Nov 10, 2026

gau-nernst deleted the pp-transport-copy-engine branch November 10, 2026 01:04

This was referenced Nov 10, 2026

[Bugfix][MRV2] Apply EVS-recomputed M-RoPE positions on every PP stage #66632

Open

[MRV2][PP] Support different CUDA graph padding across pipeline stages #66465

Merged

Conversation

gau-nernst commented Nov 5, 2026 • edited by njhill Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Summary

Known weaknesses

Validation

Details

Uh oh!

mergify Bot commented Nov 5, 2026

Uh oh!

njhill commented Nov 5, 2026

Uh oh!

gau-nernst commented Nov 6, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

gau-nernst commented Nov 6, 2026

Uh oh!

github-actions Bot commented Nov 6, 2026

Uh oh!

mergify Bot commented Nov 6, 2026

Uh oh!

njhill commented Nov 8, 2026

Uh oh!

gau-nernst commented Nov 9, 2026

Uh oh!

github-actions Bot commented Nov 9, 2026

Uh oh!

njhill left a comment

Choose a reason for hiding this comment

Uh oh!

njhill commented Nov 9, 2026

Uh oh!

github-actions Bot commented Nov 9, 2026

Uh oh!

njhill commented Nov 9, 2026

Uh oh!

github-actions Bot commented Nov 9, 2026

Uh oh!

njhill commented Nov 10, 2026

Uh oh!

khluu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gau-nernst commented Nov 5, 2026 •

edited by njhill

Loading