Repository navigation
[Feat] pp: Add CPU-staged Gloo transport for pipeline parallelism - #45790
Open
zeyuan-lcl wants to merge 3 commits into
Open
zeyuan-lcl wants to merge 3 commits into
zeyuan-lcl wants to merge 3 commits into
Conversation
…elism Add an opt-in Gloo PP transport that snapshots outgoing tensors in CPU memory and restores received tensors to their original device before TP all-gather. Preserve the default communicator policy and TP backend. Cover configuration validation, backend-specific communicator selection, real four-rank Gloo shard reassembly, and CUDA graph payload buffer reuse.
Compare eager and full decode CUDA graph outputs over Gloo PP2 and require replay on both stages without logged decode fallback. Drain captured server output after shutdown and propagate capture failures so late fallback messages cannot be missed. Include regressions for delayed output and failed sinks.
Document the PP transport option, a two-host deployment example, supported configurations, and synchronous staging costs. Clarify network protection requirements and that TP and PD KV transfer are configured separately.
zeyuan-lcl
requested review from
JustinTong0323,
ch-wan,
merrymercy,
sogalin,
wisclmy0611,
yizhang2077 and
zijiexia
as code owners
October 22, 2026 09:17
Contributor
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
Author
This was referenced Nov 9, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
TDX / Confidential Computing (CC) deployments that require host-memory PP communication need a selectable
GPU → CPU → Gloo → CPU → GPUpath while retaining their existing TP backend. The tested upstream baseline (16eb87c06fca) does not expose this PP path. This PR adds--pp-transport gloo, stages PP payloads through owned CPU buffers and restores received tensors to CUDA. Default transport selection and TP collectives are preserved.The existing PP tensor send path does not automatically stage CUDA tensors on the CPU for Gloo. In the tested environment, explicitly creating Gloo process groups and sending CUDA tensors directly triggers
gloo::IoException: writev ... Bad address. This PR adds GPU-to-CPU staging on send and CPU-to-GPU restoration on receive so that PP payloads can travel through Gloo.A matched end-to-end comparison on two H20 hosts exercised Qwen3.5-35B-A3B with cross-host TP1/PP2, NEXTN and full decode CUDA graphs. The unmodified upstream default NCCL path completed eight serial warmup requests but failed to complete the concurrency-4 warmup block within the common 180-second limit. A second control using the PR implementation at the time with default NCCL also exceeded the limit: its first serial warmup block eventually took 184.01 seconds. Gloo completed all 32 warmup requests and 48 measured requests. Its measured output token sequences matched across repeats and concurrency settings. The same model, runtime, input tokens and serving settings were used, apart from transport selection and the implementation revisions being compared.
Startup and two full-shape warmup rounds were excluded from timing. Populated Triton caches were shared across cases on each host; server logs and cache-artifact timestamps showed no new compilation during the measured blocks or failed NCCL warmup windows.
Modifications
This option currently requires CUDA and PP > 1; independent DP replicas and elastic rank offsets are rejected. Host staging is synchronous; asynchronous work overlaps Gloo transmission only. Gloo manages its own peer connections, so no fixed PP port range is exposed.
Gloo communication does not add encryption, peer authentication or attestation. Object communication uses pickle and requires trusted peers. Deployments must protect inter-host traffic separately. PD KV transfer remains a separate configuration.
Accuracy Tests
Isolated Linux CPU validation passed 66 tests and 22 subtests, including configuration validation. The four-rank Gloo test exercises actual PP sends and TP all-gather with noncontiguous, non-divisible, empty, scalar and BF16 payloads; it also checks asynchronous snapshots, bidirectional exchange, broadcasts and tagged object receives. Default communicator-policy tests cover CUDA, NPU, XPU and Mooncake policy without requiring those accelerators.
Cross-host Qwen3.5-35B-A3B validation on H20 completed all eight Gloo configurations: TP1/TP2, PP2, ordinary decoding and NEXTN, each in eager and full decode CUDA graph modes. All four pairs matched 20/20 measured 64-token outputs (80/80 paired comparisons; 160 requests). Replay counts refer only to the four full decode CUDA graph configurations, including warmup. Both PP stages had the same count within each configuration, with no logged decode fallback: 952 each for TP1 ordinary and TP2 ordinary, 337 for TP1 NEXTN, and 336 for TP2 NEXTN. Counts are decode-batch log entries. The 80/80 comparisons pair eager and full execution within each TP configuration; request-level draft acceptance statistics differ between TP1 and TP2. Prefill graphs were disabled.
The updated GPU CI test compares Gloo eager and full decode CUDA graphs, requires replay on both stages and rejects logged decode fallback after shutdown and output capture have completed. The test passed on two H20 GPUs using a hash-verified local copy of Qwen2.5-1.5B-Instruct (1 test, 169.3 seconds).
The two-rank GPU regression uses real Gloo process groups and contiguous CUDA graph outputs, exercising the actual PP tensor-dictionary send/receive methods. Three consecutive graph replays pass data, dtype, receiver-device and asynchronous send-buffer snapshot checks. Disabling only CPU staging reproduces the
Bad addresserror described above.The new contiguous CUDA-payload regression passed on two H20 GPUs (1 test, 22.143 seconds).
Speed Tests and Profiling
The matched TP1/PP2 + NEXTN comparison used 256 input tokens / 128 output tokens, two unmeasured rounds followed by three measured rounds, each with eight requests at concurrency 1 and 4. Median measured output throughput:
Gloo logged 1,788 decode graph replays per PP stage, including warmup, with no logged decode fallback. Its c4 repeats ranged from 245.81 to 285.42 output tokens/s. Failed warmup timings are diagnostics, not steady-state performance measurements.
The following broader Gloo-only measurements were collected separately:
Completed on two H20 hosts with full decode CUDA graphs and Gloo PP transport: 256 input tokens / 128 output tokens, concurrency 1/4, three repeats of eight requests per concurrency (48 requests per configuration). Throughput is the median of three runs; TTFT and TPOT are medians over requests. Profiling is excluded from timing.
TP2 has lower throughput than TP1 in three of the four comparisons, with NEXTN c4 as the exception. TP2 retains synchronous CPU staging and adds TP collectives and all-gather for PP shard reassembly.
At TP1, NEXTN c4 had lower median throughput than ordinary decoding (237.51 vs 246.44 tokens/s); at TP2, it was higher (285.21 vs 221.21 tokens/s). The two independent TP1 NEXTN c4 measurements had medians of 237.51 and 275.88 tokens/s, with three-repeat ranges of 237.38–284.26 and 245.81–285.42 tokens/s, respectively, showing substantial run-to-run variation.
Within each configuration, full output token sequences matched across repeats and concurrency settings for the same prompts. TP2 NEXTN c4 throughput ranged from 237.43 to 285.99 tokens/s. NCCL did not complete warmup, so no steady-state throughput is available for a speedup calculation.
Process-tree CPU use averaged 0.53–1.60 cores per host across measurement blocks. Peak aggregate process RSS ranged from 3.50 to 7.74 GiB per host; shared pages may be counted more than once. Twelve per-rank CPU traces were collected. CPU receive scopes include synchronization and waiting.
The validation targets Confidential Computing (CC) deployments. Both hosts reported GPU CC State OFF; GPU CC-On and attestation were not tested.
The environment previously rejected GPU CUPTI activities with
CUPTI_ERROR_CONFIDENTIAL_COMPUTING_NOT_SUPPORTED (41). The collected traces contain CPU activities only.Reproduction
Environment, two-host commands and standalone client
Tested runtime: Linux, Python 3.12.7, PyTorch 2.14.1+cu130, CUDA 13.0, NCCL 2.30.7, NVIDIA H20, driver 570.148.08. Use identical model files on both hosts. The commands below pin the current upstream baseline and PR head for a new matched comparison. Update
BASEandPR_HEADif the commit history changes. Save the inline blocks aslaunch.shandclient.pyoutside the source worktrees.On each host, run the launcher in a dedicated terminal. Use a fresh
RUN_DIRper case. It must expose both live logs to the client (shared storage or equivalent live log forwarding). Reserve the selected GPUs and ports independently from other workloads.For the broader Gloo matrix, use the same serving settings and client workload. On both hosts, set
TP_SIZE=1 GPU_IDS=2orTP_SIZE=2 GPU_IDS=2,3, using one or two GPUs per host, respectively. SetNEXTN=0for ordinary decoding orNEXTN=1for NEXTN, keepingTRANSPORT=gloo. Run the four combinations sequentially with a freshRUN_DIRand separate startup, warmup, measurement and cleanup for each case.Run cases sequentially: PR/Gloo, upstream/default, PR/default. Select
SOURCE_DIRaccordingly and setTRANSPORTtogloo,upstream,default, respectively. Stop only the two servers belonging to the completed case and wait for their processes to exit before reusing ports. The client never terminates servers.Use the same persistent per-host cache across cases. The reported experiment seeded it from prior runs; on a fresh setup, first populate it with untimed runs of the same shapes. Every recorded run still requires 32 warmup requests (two rounds at c1/c4), then 48 measured requests (three rounds). Startup and warmup are excluded. New compile log messages invalidate a measured block. If warmup times out, check the server logs and cache files for ongoing compilation. The client enforces 180 seconds per block and 900 seconds overall, saves completed requests and exits on timeout. The original diagnostic harness also captured server stacks, so its log tail after a timeout may differ.
Save and run this client (dependencies:
requests,transformers). It records input/output token IDs and timing for each completed block. Check complete output sequences across repeats and concurrency settings, requirestatus == "complete"before reporting throughput, and take the median of the three measured blocks for each concurrency. Inspect both server logs for decode graph replay and absence of decode fallback; prefill graphs are deliberately disabled.Documentation validation limitation: the previous docs-site build failed on an existing extra closing brace in
popular-models.jsxand related import errors. That file is outside this PR diff.mint broken-linkspassed.Checklist
CI States
Latest PR Test (Base): ❌ Run #39754326914
Latest PR Test (Extra): ❌ Run #39754326860
Latest PR Test (AMD ROCm 10): ❌ Run #39754326857