Skip to content

[Feat] pp: Add CPU-staged Gloo transport for pipeline parallelism - #45790

Open
zeyuan-lcl wants to merge 3 commits into
sgl-project:mainfrom
aliyun:feat/tdx-cc-pp-gloo
Open

zeyuan-lcl wants to merge 3 commits into
sgl-project:mainfrom
aliyun:feat/tdx-cc-pp-gloo

Conversation

@zeyuan-lcl

@zeyuan-lcl zeyuan-lcl commented Oct 22, 2026 •

Copy link
Copy Markdown

Motivation

TDX / Confidential Computing (CC) deployments that require host-memory PP communication need a selectable GPU → CPU → Gloo → CPU → GPU path while retaining their existing TP backend. The tested upstream baseline (16eb87c06fca) does not expose this PP path. This PR adds --pp-transport gloo, stages PP payloads through owned CPU buffers and restores received tensors to CUDA. Default transport selection and TP collectives are preserved.

The existing PP tensor send path does not automatically stage CUDA tensors on the CPU for Gloo. In the tested environment, explicitly creating Gloo process groups and sending CUDA tensors directly triggers gloo::IoException: writev ... Bad address. This PR adds GPU-to-CPU staging on send and CPU-to-GPU restoration on receive so that PP payloads can travel through Gloo.

A matched end-to-end comparison on two H20 hosts exercised Qwen3.5-35B-A3B with cross-host TP1/PP2, NEXTN and full decode CUDA graphs. The unmodified upstream default NCCL path completed eight serial warmup requests but failed to complete the concurrency-4 warmup block within the common 180-second limit. A second control using the PR implementation at the time with default NCCL also exceeded the limit: its first serial warmup block eventually took 184.01 seconds. Gloo completed all 32 warmup requests and 48 measured requests. Its measured output token sequences matched across repeats and concurrency settings. The same model, runtime, input tokens and serving settings were used, apart from transport selection and the implementation revisions being compared.

Startup and two full-shape warmup rounds were excluded from timing. Populated Triton caches were shared across cases on each host; server logs and cache-artifact timestamps showed no new compilation during the measured blocks or failed NCCL warmup windows.

Modifications

  • Select Gloo for the PP process groups and avoid creating a PP PyNccl communicator when CPU staging is enabled.
  • Copy outgoing PP tensors into owned, contiguous CPU buffers; retain asynchronous send buffers until completion. Restore received tensors to their original device before TP all-gather reassembles sharded payloads.
  • Cover tensor dictionaries, bidirectional exchanges, tensor broadcasts and raw tensor P2P. Route PP object-list broadcasts through the CPU group; existing scheduler control messages continue to use their Gloo path.
  • Add a four-rank Gloo regression with real TP all-gather, configuration/communicator-policy tests, and an eager-versus-CUDA-graph generation test with replay and log-drain assertions.
  • Document cross-host deployment and synchronous staging costs.

This option currently requires CUDA and PP > 1; independent DP replicas and elastic rank offsets are rejected. Host staging is synchronous; asynchronous work overlaps Gloo transmission only. Gloo manages its own peer connections, so no fixed PP port range is exposed.

Gloo communication does not add encryption, peer authentication or attestation. Object communication uses pickle and requires trusted peers. Deployments must protect inter-host traffic separately. PD KV transfer remains a separate configuration.

Accuracy Tests

Isolated Linux CPU validation passed 66 tests and 22 subtests, including configuration validation. The four-rank Gloo test exercises actual PP sends and TP all-gather with noncontiguous, non-divisible, empty, scalar and BF16 payloads; it also checks asynchronous snapshots, bidirectional exchange, broadcasts and tagged object receives. Default communicator-policy tests cover CUDA, NPU, XPU and Mooncake policy without requiring those accelerators.

Cross-host Qwen3.5-35B-A3B validation on H20 completed all eight Gloo configurations: TP1/TP2, PP2, ordinary decoding and NEXTN, each in eager and full decode CUDA graph modes. All four pairs matched 20/20 measured 64-token outputs (80/80 paired comparisons; 160 requests). Replay counts refer only to the four full decode CUDA graph configurations, including warmup. Both PP stages had the same count within each configuration, with no logged decode fallback: 952 each for TP1 ordinary and TP2 ordinary, 337 for TP1 NEXTN, and 336 for TP2 NEXTN. Counts are decode-batch log entries. The 80/80 comparisons pair eager and full execution within each TP configuration; request-level draft acceptance statistics differ between TP1 and TP2. Prefill graphs were disabled.

The updated GPU CI test compares Gloo eager and full decode CUDA graphs, requires replay on both stages and rejects logged decode fallback after shutdown and output capture have completed. The test passed on two H20 GPUs using a hash-verified local copy of Qwen2.5-1.5B-Instruct (1 test, 169.3 seconds).

The two-rank GPU regression uses real Gloo process groups and contiguous CUDA graph outputs, exercising the actual PP tensor-dictionary send/receive methods. Three consecutive graph replays pass data, dtype, receiver-device and asynchronous send-buffer snapshot checks. Disabling only CPU staging reproduces the Bad address error described above.

The new contiguous CUDA-payload regression passed on two H20 GPUs (1 test, 22.143 seconds).

Speed Tests and Profiling

The matched TP1/PP2 + NEXTN comparison used 256 input tokens / 128 output tokens, two unmeasured rounds followed by three measured rounds, each with eight requests at concurrency 1 and 4. Median measured output throughput:

Transport / source Warmup outcome Measured requests Output tokens/s, c1 / c4
Default NCCL / unmodified upstream c4 exceeded 180-second block limit 0 N/A
Default NCCL / tested PR implementation first c1 block took 184.01 s; exceeded limit 0 N/A
Gloo / tested PR implementation 32/32 completed 48 123.98 / 275.88

Gloo logged 1,788 decode graph replays per PP stage, including warmup, with no logged decode fallback. Its c4 repeats ranged from 245.81 to 285.42 output tokens/s. Failed warmup timings are diagnostics, not steady-state performance measurements.

The following broader Gloo-only measurements were collected separately:

Completed on two H20 hosts with full decode CUDA graphs and Gloo PP transport: 256 input tokens / 128 output tokens, concurrency 1/4, three repeats of eight requests per concurrency (48 requests per configuration). Throughput is the median of three runs; TTFT and TPOT are medians over requests. Profiling is excluded from timing.

Configuration Output tokens/s, c1 Output tokens/s, c4 TTFT ms, c1 / c4 TPOT ms, c1 / c4
TP1/PP2, ordinary 86.02 246.44 198.4 / 400.9 10.15 / 13.20
TP1/PP2, NEXTN 126.93 237.51 224.0 / 394.6 5.99 / 10.40
TP2/PP2, ordinary 75.97 221.21 227.9 / 530.2 11.33 / 14.36
TP2/PP2, NEXTN 124.01 285.21 258.2 / 458.6 6.00 / 9.98

TP2 has lower throughput than TP1 in three of the four comparisons, with NEXTN c4 as the exception. TP2 retains synchronous CPU staging and adds TP collectives and all-gather for PP shard reassembly.

At TP1, NEXTN c4 had lower median throughput than ordinary decoding (237.51 vs 246.44 tokens/s); at TP2, it was higher (285.21 vs 221.21 tokens/s). The two independent TP1 NEXTN c4 measurements had medians of 237.51 and 275.88 tokens/s, with three-repeat ranges of 237.38–284.26 and 245.81–285.42 tokens/s, respectively, showing substantial run-to-run variation.

Within each configuration, full output token sequences matched across repeats and concurrency settings for the same prompts. TP2 NEXTN c4 throughput ranged from 237.43 to 285.99 tokens/s. NCCL did not complete warmup, so no steady-state throughput is available for a speedup calculation.

Process-tree CPU use averaged 0.53–1.60 cores per host across measurement blocks. Peak aggregate process RSS ranged from 3.50 to 7.74 GiB per host; shared pages may be counted more than once. Twelve per-rank CPU traces were collected. CPU receive scopes include synchronization and waiting.

The validation targets Confidential Computing (CC) deployments. Both hosts reported GPU CC State OFF; GPU CC-On and attestation were not tested.

The environment previously rejected GPU CUPTI activities with CUPTI_ERROR_CONFIDENTIAL_COMPUTING_NOT_SUPPORTED (41). The collected traces contain CPU activities only.

Reproduction

Environment, two-host commands and standalone client

Tested runtime: Linux, Python 3.12.7, PyTorch 2.14.1+cu130, CUDA 13.0, NCCL 2.30.7, NVIDIA H20, driver 570.148.08. Use identical model files on both hosts. The commands below pin the current upstream baseline and PR head for a new matched comparison. Update BASE and PR_HEAD if the commit history changes. Save the inline blocks as launch.sh and client.py outside the source worktrees.

BASE=d0a75654368a
# Run inside a clone containing the upstream base and the submitted PR commit.
PR_HEAD=5dc552c0b9f9
git worktree add --detach ../pp-upstream "$BASE"
git worktree add --detach ../pp-gloo "$PR_HEAD"

On each host, run the launcher in a dedicated terminal. Use a fresh RUN_DIR per case. It must expose both live logs to the client (shared storage or equivalent live log forwarding). Reserve the selected GPUs and ports independently from other workloads.

#!/usr/bin/env bash
set -euo pipefail
: "${SOURCE_DIR:?Absolute path to the selected source checkout}"
: "${MODEL_PATH:?Local Qwen3.5-35B-A3B snapshot, identical on both hosts}"
: "${NODE0_HOST:?Reachable address of rank 0}"
: "${NODE_RANK:?0 or 1}"
: "${RUN_DIR:?Shared per-case log directory visible from both hosts and client}"
: "${CACHE_DIR:?Dedicated persistent cache directory on this host}"
: "${IFACE:?Network interface reachable by both hosts}"
: "${TRANSPORT:?upstream, default, or gloo}"
export CUDA_VISIBLE_DEVICES="${GPU_IDS:-2}"
export PYTHONPATH="$SOURCE_DIR/python"
export SGLANG_ENABLE_PP_SPEC=1
export GLOO_SOCKET_IFNAME="$IFACE" NCCL_SOCKET_IFNAME="$IFACE"
export TRITON_CACHE_DIR="$CACHE_DIR/triton"
export TORCHINDUCTOR_CACHE_DIR="$CACHE_DIR/inductor"
export PYTHONPYCACHEPREFIX="$CACHE_DIR/pycache"
mkdir -p "$RUN_DIR" "$CACHE_DIR"
extra=()
case "$TRANSPORT" in
  upstream) ;; # The unmodified base has no --pp-transport flag.
  default|gloo) extra=(--pp-transport "$TRANSPORT") ;;
  *) exit 2 ;;
esac
spec_args=()
case "${NEXTN:-1}" in
  0) ;;
  1) spec_args=(--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --mamba-radix-cache-strategy extra_buffer --mamba-track-interval 128) ;;
  *) exit 2 ;;
esac
exec python -m sglang.launch_server \
  --model-path "$MODEL_PATH" --dist-init-addr "$NODE0_HOST:34010" \
  --node-rank "$NODE_RANK" --host 0.0.0.0 --port 34000 \
  --tp-size "${TP_SIZE:-1}" --pp-size 2 --nnodes 2 --load-format safetensors --skip-tokenizer-init --random-seed 42 --dtype bfloat16 --attention-backend triton --linear-attn-prefill-backend triton --enable-deterministic-inference --cuda-graph-backend-decode full --cuda-graph-backend-prefill disabled --disable-overlap-schedule --mem-fraction-static 0.9 --max-total-tokens 8192 --max-running-requests 4 --context-length 2048 --chunked-prefill-size 128 --page-size 1 --disable-custom-all-reduce --language-only --trust-remote-code --max-mamba-cache-size 32 --cuda-graph-max-bs-decode 4 --decode-log-interval 1 ${spec_args[@]+"${spec_args[@]}"} \
  ${extra[@]+"${extra[@]}"} > "$RUN_DIR/node${NODE_RANK}.log" 2>&1
# On each host, set these to your own dedicated paths and reachable interface.
export SOURCE_DIR=/absolute/path/to/pp-gloo
export MODEL_PATH=/absolute/path/to/Qwen3.5-35B-A3B
export NODE0_HOST=rank0.example.net
export NODE_RANK=0  # Set to 1 on the second host.
export IFACE=eth0 GPU_IDS=2 TP_SIZE=1 NEXTN=1 TRANSPORT=gloo
export CACHE_DIR=/absolute/path/to/dedicated-pp-cache
export RUN_DIR=/absolute/shared/path/to/unique-gloo-run
bash launch.sh

For the broader Gloo matrix, use the same serving settings and client workload. On both hosts, set TP_SIZE=1 GPU_IDS=2 or TP_SIZE=2 GPU_IDS=2,3, using one or two GPUs per host, respectively. Set NEXTN=0 for ordinary decoding or NEXTN=1 for NEXTN, keeping TRANSPORT=gloo. Run the four combinations sequentially with a fresh RUN_DIR and separate startup, warmup, measurement and cleanup for each case.

# On rank 0, after both servers have launched; startup is outside client timing.
python - <<'PYREADY'
import time, requests
end = time.monotonic() + 300
while time.monotonic() < end:
    try:
        if requests.get("http://127.0.0.1:34000/ready", timeout=2).status_code == 200:
            break
    except requests.RequestException:
        pass
    time.sleep(2)
else:
    raise SystemExit("Server readiness exceeded 300 seconds")
PYREADY
python client.py --url http://127.0.0.1:34000 --model "$MODEL_PATH" \
  --output "$RUN_DIR/bench.json" \
  --node0-log "$RUN_DIR/node0.log" --node1-log "$RUN_DIR/node1.log"

Run cases sequentially: PR/Gloo, upstream/default, PR/default. Select SOURCE_DIR accordingly and set TRANSPORT to gloo, upstream, default, respectively. Stop only the two servers belonging to the completed case and wait for their processes to exit before reusing ports. The client never terminates servers.

Use the same persistent per-host cache across cases. The reported experiment seeded it from prior runs; on a fresh setup, first populate it with untimed runs of the same shapes. Every recorded run still requires 32 warmup requests (two rounds at c1/c4), then 48 measured requests (three rounds). Startup and warmup are excluded. New compile log messages invalidate a measured block. If warmup times out, check the server logs and cache files for ongoing compilation. The client enforces 180 seconds per block and 900 seconds overall, saves completed requests and exits on timeout. The original diagnostic harness also captured server stacks, so its log tail after a timeout may differ.

Save and run this client (dependencies: requests, transformers). It records input/output token IDs and timing for each completed block. Check complete output sequences across repeats and concurrency settings, require status == "complete" before reporting throughput, and take the median of the three measured blocks for each concurrency. Inspect both server logs for decode graph replay and absence of decode fallback; prefill graphs are deliberately disabled.

"""Same e2e workload, explicit full-shape warmup, atomic progress and timing gates."""

import argparse
import json
import os
import threading
import time
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path

import requests
from transformers import AutoTokenizer

p = argparse.ArgumentParser()
p.add_argument("--url")
p.add_argument("--model")
p.add_argument("--output")
p.add_argument("--node0-log", required=True)
p.add_argument("--node1-log", required=True)
a = p.parse_args()
out = Path(a.output)
assert not out.exists()
lock = threading.RLock()
tok = AutoTokenizer.from_pretrained(a.model, local_files_only=True)
base = tok.encode(
    "Explain pipeline parallelism and CPU communication. " * 80,
    add_special_tokens=False,
)
inputs = [
    (tok.encode(f"Question {i}: ", add_special_tokens=False) + base)[:256]
    for i in range(8)
]
r = dict(
    input_ids=inputs,
    input_tokens=256,
    output_tokens=128,
    status="running",
    warmups=[],
    runs=[],
    phase="initializing",
    phase_status="idle",
    events=[],
)


def save():
    with lock:
        temp = out.with_suffix(".tmp")
        temp.write_text(json.dumps(r, indent=2))
        temp.replace(out)


def log_counts():
    return {
        str(i): sum(
            "took" in l and "compile" in l
            for l in Path([a.node0_log, a.node1_log][i]).read_text().splitlines()
        )
        for i in [0, 1]
    }


def generate(item):
    i, ids = item
    start = time.perf_counter()
    first = None
    final = None
    chunks = []
    with requests.post(
        a.url + "/generate",
        json={
            "input_ids": ids,
            "stream": True,
            "sampling_params": {
                "temperature": 0,
                "max_new_tokens": 128,
                "ignore_eos": True,
            },
        },
        stream=True,
        timeout=200,
    ) as res:
        res.raise_for_status()
        for line in res.iter_lines(chunk_size=1):
            if not line.startswith(b"data: ") or line == b"data: [DONE]":
                continue
            obj = json.loads(line[6:])
            if obj.get("error"):
                raise RuntimeError(obj)
            if obj.get("output_ids"):
                now = time.perf_counter()
                if first is None:
                    first = now
                    with lock:
                        r["events"].append(
                            dict(
                                phase=r["phase"],
                                request=i,
                                event="first_token",
                                time=time.time(),
                            )
                        )
                        save()
                final = obj
                chunks.append([now - start, len(obj["output_ids"])])
    end = time.perf_counter()
    assert final and len(final["output_ids"]) == 128
    result = dict(
        ids=final["output_ids"],
        ttft_s=first - start,
        latency_s=end - start,
        tpot_s=(end - first) / 127,
        chunk_arrivals=chunks,
        meta_info=final["meta_info"],
    )
    with lock:
        r["events"].append(
            dict(phase=r["phase"], request=i, event="complete", time=time.time())
        )
        save()
    return result


save()


def enforce_deadlines():
    start = time.monotonic()
    while True:
        time.sleep(0.2)
        with lock:
            if r["status"] != "running":
                return
            if (
                r.get("phase_status") == "running"
                and time.time() - r["phase_started_at"] > 180
            ) or time.monotonic() - start > 900:
                r["status"] = "failed"
                r["error"] = "Request block exceeded 180 s or client exceeded 900 s"
                save()
                # ThreadPoolExecutor cannot cancel an in-flight HTTP request. Exit only
                # this client; the operator retains control of both model servers.
                os._exit(2)


threading.Thread(target=enforce_deadlines, daemon=True).start()
try:
    for stage, repeats in [("warmup", 2), ("measure", 3)]:
        for repeat in range(repeats):
            for c in [1, 4]:
                requests.get(a.url + "/flush_cache", timeout=30).raise_for_status()
                with lock:
                    r.update(
                        phase=f"{stage}-{repeat}-c{c}",
                        phase_status="running",
                        phase_started_at=time.time(),
                    )
                    save()
                counts = log_counts()
                start = time.perf_counter()
                started_at = time.time()
                with ThreadPoolExecutor(max_workers=c) as pool:
                    rows = list(pool.map(generate, enumerate(inputs)))
                elapsed = time.perf_counter() - start
                new_counts = log_counts()
                entry = dict(
                    repeat=repeat,
                    concurrency=c,
                    started_at=started_at,
                    finished_at=time.time(),
                    elapsed_s=elapsed,
                    output_tokens_per_s=1024 / elapsed,
                    requests=rows,
                    compile_messages_before=counts,
                    compile_messages_after=new_counts,
                )
                with lock:
                    r["warmups" if stage == "warmup" else "runs"].append(entry)
                    r["phase_status"] = "complete"
                    save()
                print(
                    stage,
                    repeat,
                    c,
                    elapsed,
                    "new_compilation",
                    counts != new_counts,
                    flush=True,
                )
                if stage == "measure" and counts != new_counts:
                    raise RuntimeError(
                        "Compilation detected in measured window; do not report this timing"
                    )
    r["status"] = "complete"
    r["benchmark_status"] = "complete"
    save()
except BaseException as exc:
    r["status"] = "failed"
    r["error"] = repr(exc)
    save()
    raise

Documentation validation limitation: the previous docs-site build failed on an existing extra closing brace in popular-models.jsx and related import errors. That file is outside this PR diff. mint broken-links passed.

Checklist

  • Run all applicable repository pre-commit hooks on the 13 changed files (Python 3.12).
  • Add real Gloo and configuration regressions.
  • Add deployment documentation.
  • Complete current Gloo GPU and cross-host validation.
  • Provide Gloo speed/resource measurements and CPU profiling.
  • Complete code review.

CI States

Latest PR Test (Base): ❌ Run #39754326914
Latest PR Test (Extra): ❌ Run #39754326860
Latest PR Test (AMD ROCm 10): ❌ Run #39754326857

…elism

Add an opt-in Gloo PP transport that snapshots outgoing tensors in CPU memory and restores received tensors to their original device before TP all-gather. Preserve the default communicator policy and TP backend.

Cover configuration validation, backend-specific communicator selection, real four-rank Gloo shard reassembly, and CUDA graph payload buffer reuse.
Compare eager and full decode CUDA graph outputs over Gloo PP2 and require replay on both stages without logged decode fallback.

Drain captured server output after shutdown and propagate capture failures so late fallback messages cannot be missed. Include regressions for delayed output and failed sinks.
Document the PP transport option, a two-host deployment example, supported configurations, and synchronous staging costs. Clarify network protection requirements and that TP and PD KV transfer are configured separately.
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Oct 22, 2026
@mintlify

mintlify Bot commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 5dc552c.

Approve preview

@zeyuan-lcl zeyuan-lcl changed the title [Feat][PP] Add CPU-staged Gloo transport for pipeline parallelism [Feat] pp: Add CPU-staged Gloo transport for pipeline parallelism Oct 22, 2026
@zeyuan-lcl

Copy link
Copy Markdown
Author

Hi @stepinto, could you take a look at this PR? It adds CPU-staged Gloo transport for pipeline parallelism in TDX/CC deployments. Given your work on PP communication in #38959, your feedback on the send/receive path and cross-host validation would be helpful. Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant