Skip to content

[Feat] [SM120] distributed: Add optional b12x PCIe all-reduce (one-shot + fused RMSNorm + FP8 DMA ring) - #32479

Open
AliceChenyy wants to merge 7 commits into
sgl-project:mainfrom
AliceChenyy:feat/b12x-pcie-allreduce
Open

AliceChenyy wants to merge 7 commits into
sgl-project:mainfrom
AliceChenyy:feat/b12x-pcie-allreduce

Conversation

@AliceChenyy

@AliceChenyy AliceChenyy commented Jul 27, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

On Blackwell workstation / server parts (SM120 & SM121 — RTX PRO 6000, RTX 6000D, RTX 50-series, GB10) the GPUs are connected over PCIe with no NVLink. The per-layer tensor-parallel all-reduce therefore dominates both decode latency and long-prompt prefill: on 8x RTX 6000D running GLM-5.2-NVFP4, NCCL needs 45 us for a 4 KB all-reduce and 344 us for 64 KB, which is most of the per-token budget at low batch sizes.

b12x (Apache-2.0) is an SM120/SM121-only CuTe DSL kernel library that ships IPC-backed PCIe all-reduce runtimes for exactly this topology. This PR wires two of them into SGLang behind an opt-in flag.

What this adds

A new B12xPCIeCommunicator and a three-tier dispatch inside GroupCoordinator._all_reduce:

message size path
<= 8 MB b12x PCIe one-shot all-reduce (decode-sized hidden states)
8 MB .. 256 MB b12x PCIe DMA ring, optional FP8 wire format (prefill chunks)
otherwise existing NCCL path, unchanged

The one-shot runtime also exposes a fused all-reduce + residual add + RMSNorm, plugged into the existing fused_allreduce_rmsnorm coordinator hook that AITER already uses (layers/communicator.py + layers/layernorm.py). That kernel is specialized for <= 36 rows, so the dispatch helpers cap on token count and fall back when it does not apply — including an explicit all-reduce fallback in _forward_with_allreduce_fusion, since the caller has already skipped its own all-reduce when it takes the fusion path.

Safety / blast radius

  • Off by default: SGLANG_B12X_PCIE_AR and SGLANG_B12X_PCIE_DMA are EnvBool(False).
  • The communicator is only constructed for the tensor-parallel group (other groups would just pin IPC buffers).
  • b12x is an optional dependency; when it is not installed the setup logs a warning and everything stays on NCCL. No changes to any default code path.
  • Every tier re-checks dtype / contiguity / size and returns to NCCL when the kernel declines.

Results

GLM-5.2-NVFP4, 8x RTX 6000D (SM120, PCIe), driver 595.71.05, KV cache FP8, greedy, prefix caching off.

Kernel microbenchmark (8 ranks, NCCL vs b12x one-shot): 4 KB 45.0 -> 7.3 us (6.1x), 12 KB 58.5 -> 15.3 us (3.8x), 64 KB 344.5 -> 38.5 us (8.9x).

TP8 decode (ISL 1 / OSL 256), median TPOT / output tok/s:

bs baseline (NCCL) + b12x
1 23.9 ms / 41 14.7 ms / 66
8 70.6 ms / 112 29.8 ms / 264
32 105.8 ms / 300 69.2 ms / 457

TP8 e2e 8K (ISL 8192 / OSL 64), median TTFT: 6103 ms -> 1384 ms (DMA ring tier).

TP4xPP2 + MTP, 64K (ISL 65536 / OSL 1024), bs1: TTFT 11.5 s -> 9.3 s, TPOT 10.3 ms -> 9.5 ms, accept length 3.00.

Accuracy: GSM8K (20 questions) stays at 0.900 for both TP8 and TP4xPP2+MTP — identical to the NCCL baseline — with Invalid 0.000, and greedy output matches token-for-token on a fixed prompt.

Test plan

  • TP8 and TP4xPP2(+MTP) launch with both tiers active (8/8 ranks report one-shot + DMA enabled, no init failures)
  • Communicator is built for the TP group only (world=4 under TP4xPP2, PP group untouched)
  • GSM8K 0.900 / Invalid 0.000, greedy output matches baseline
  • decode (bs 1/8/32), e2e 8K, 64K sweeps as above
  • Gate off (default): unchanged NCCL behaviour

Notes for reviewers

  • Requires pip install b12x (plus nvidia-cutlass-dsl, pulled in as a wheel dependency; kernels are JIT-compiled on first use).
  • The DMA ring allocates a scratch slab sized by SGLANG_B12X_DMA_MAX_BYTES (default 256 MB); on memory-tight setups this competes with the KV pool, so raise --mem-fraction-static accordingly when enabling it together with speculative decoding.
  • _FUSED_MAX_ROWS = 36 mirrors the current kernel limit and is duplicated in communicator.py as _B12X_FUSED_MAX_ROWS; happy to expose it from the package instead if maintainers prefer.
  • This is orthogonal to Pipeline parallelism x speculative decoding (EAGLE/MTP) compatibility #30775 (PP x speculative decoding). Stacking both is what produces the 64K number above.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #39701958292
Latest PR Test (Extra): ❌ Run #39701958289
Latest PR Test (AMD ROCm 10): ❌ Run #39701958309

… FP8 DMA ring)

On Blackwell workstation/server parts (SM120/SM121, e.g. RTX PRO 6000 / 6000D)
GPUs are connected over PCIe without NVLink, so the per-layer NCCL all-reduce
dominates decode latency and long-prompt prefill time.

This adds an opt-in communicator backed by the `b12x` package
(https://pypi.org/project/b12x/, Apache-2.0, SM120-only CuTe DSL kernels) with a
three-tier dispatch inside GroupCoordinator:

  <= 8 MB          -> b12x PCIe one-shot all-reduce (IPC, decode-sized messages)
  8 MB .. 256 MB   -> b12x PCIe DMA ring with optional FP8 wire format (prefill)
  otherwise        -> existing NCCL path (unchanged)

The one-shot runtime also provides a fused all-reduce + residual add + RMSNorm
entry point, wired through the existing `fused_allreduce_rmsnorm` coordinator
hook that AITER already uses. That kernel is specialized for <= 36 rows, so the
dispatch helpers cap on token count and fall back otherwise.

Everything is gated behind SGLANG_B12X_PCIE_AR / SGLANG_B12X_PCIE_DMA (default
off) and the communicator is only built for the tensor-parallel group. When the
`b12x` package is absent the setup logs a warning and NCCL is used, so existing
deployments are unaffected.

Measured on GLM-5.2-NVFP4, 8x RTX 6000D, driver 595.71.05:

  kernel microbench (8 ranks, NCCL -> b12x one-shot)
    4 KB   45.0 us -> 7.3 us   (6.1x)
    64 KB  344.5 us -> 38.5 us (8.9x)

  TP8 decode (ISL 1 / OSL 256), median TPOT
    bs1  23.9 ms -> 14.7 ms
    bs8  70.6 ms -> 29.8 ms
    bs32 105.8 ms -> 69.2 ms
  TP8 e2e 8K (ISL 8192 / OSL 64), median TTFT
    6103 ms -> 1384 ms
  TP4xPP2 + MTP, 64K (ISL 65536 / OSL 1024), bs1
    TTFT 11.5 s -> 9.3 s, TPOT 10.3 ms -> 9.5 ms

GSM8K (20 questions) stays at 0.900 for both TP8 and TP4xPP2+MTP, and greedy
output matches the NCCL baseline.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Resolve the one conflict in parallel_state.py: upstream factored the
out-of-place all-reduce dispatch chain into
`_resolve_outplace_all_reduce_method`. Move the b12x one-shot / DMA branches
into that helper (still ahead of `ca`), keeping the b12x logic itself
unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AliceChenyy and others added 2 commits August 18, 2026 03:09
…educe

# Conflicts:
#	python/sglang/srt/environ.py
b12x >= 1.x reorganised the PCIe all-reduce API in five ways, each of which
this wrapper hit in turn -- the first three at startup, the last two only
once a forward ran, by which point the communicator had already logged
"enabled" and looked healthy:

  1. the classes moved from b12x.distributed to b12x.comm.pcie and dropped
     their PCIe prefix, the package path now carrying it;
  2. a pool spanning ranks refuses to pick a channel implicitly and wants a
     name every rank agrees on;
  3. those names have to be allocated collectively before first use;
  4. channels are stream-affine, and the caller now owns the stream-to-
     channel mapping the pool used to keep itself;
  5. recording a one-shot into a CUDA graph requires holding that graph's
     channel for the duration of the capture.

Points 2-5 are one design change: the old release let each rank pick a
channel on its own, and since channels are stream-affine, two ranks could
silently bind different ones and stop reducing with each other. It worked
because a single channel happened to make everyone agree.

_StreamChannels hands out a reserved channel per stream in first-seen
order. That needs no extra communication: ranks run the same forward on the
same streams in the same order, so rank 3 and rank 5 give a given stream the
same channel. Streams past the reservation keep the NCCL path rather than
share a channel, which would be a correctness bug and not a slowdown.

Every branch is guarded on the installed signature, so b12x 0.30.x still
works unchanged. Verified on 8x SM120 with b12x 1.2.4: all 8 ranks enable
one-shot and DMA, GSM8K 0.925-0.975 across five configurations, and against
NCCL at TP8 the output throughput goes 21.4 -> 49.2 tok/s at batch 1 and
33.5 -> 117.0 at batch 32.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Conflicts:
- environ.py: main dropped DsparkFoldedSampling (sgl-project#43668) and
  SGLANG_ENABLE_CUSTOM_ALL_REDUCE_V2_MULTINODE (sgl-project#35061) and added the
  PCIe-IPC knobs (sgl-project#34528); keep B12xOneshotCap and the SGLANG_B12X_* knobs
  on top of that.
- layers/communicator.py: deleted on main by the layer_boundary refactor
  (sgl-project#41439, sgl-project#41554, sgl-project#41555). Port the b12x hooks to their new homes:
  b12x_ar_fusion_applies next to the flashinfer/aiter predicates in
  layer_boundary/residual/add_norm.py, consulted by fused_attn_input and
  fused_ffn_input (the old prepare_attn / CommunicateWithAllReduceAndLayerNormFn
  sites) and by _batch_allows_deferred_sum (the old
  should_fuse_mlp_allreduce_with_next_layer).
test_custom_allreduce_precedes_symmetric_memory_pynccl (from sgl-project#35061) builds a
GroupCoordinator with __new__ and sets each communicator by hand. The b12x tier
is checked ahead of custom all-reduce, so the dispatch now reads b12x_comm and
the test raised AttributeError before it reached the ca path.
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
… C closure -- the post-merge port queue is now decided

ARRIVAL SCAN (git log 12dc623..2a6c204 --grep) + LIVE gh STATE +
head-sha comparison for EVERY deferred item. Verdicts:
  RELEASED-BY-MERGE: sgl-project#42689 (@ 7a632cf in-range -- the SECTION-B
    cherry is now redundant; behavior note rides the restart).
  CLOSED (C-items whose PRs arrived and whose integration this stack
    performed): C1 (the dedicated merge week WAS merge sgl-project#5; all eight
    precursor PRs in-range), C2 (sgl-project#44993/sgl-project#44994 -- B5), C3 (sgl-project#45232/
    sgl-project#45236/sgl-project#45334 -- B2), C4 (sgl-project#45238/sgl-project#45241/sgl-project#45242 -- B3), C5 (sgl-project#44912/
    sgl-project#44913 -- arrived; entry-230 canary re-planted as its guard),
    C9 (sgl-project#39727 -- verified OPT-IN, no purge; BOOT_DELTA anti-needle),
    C0a (arrival shas already recorded at entry 310 -- Section C now
    carries NO open code-integration items: C6/C10 retain only their
    explicitly-named docs/boot-measurement remainders, re-stated).
  STILL-OPEN: sgl-project#44342 (head unchanged; auto-merge comfort LOST to
    #46970x), sgl-project#46153/sgl-project#37106/sgl-project#32479/sgl-project#45391 (gh CONFLICTING -- stale
    bases, rebase before any port), sgl-project#41624/sgl-project#42525/sgl-project#42660/sgl-project#44462/sgl-project#35617
    (open, heads unchanged), sgl-project#46191/sgl-project#39845-class (watch-list, none in
    range).
  HOLD STANDS (operator decision, binding): sgl-project#43356/sgl-project#46452/sgl-project#43240/
    sgl-project#43471 KV route -- step-1 'MERGE sgl-project#5 lands first' is NOW MET; the
    sequence may be re-visited ONLY with the post-merge PCIe-IPC
    measurement at an operator sitting. sgl-project#44732: the 'one purge if done
    WITH the merge' premise LAPSED when merge sgl-project#5 completed without it
    -- later execution costs the SAME single purge once; no penalty,
    no haste. Recorded for the operator, not decided by B6.
detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
…gl-project#44462(317)/sgl-project#44342-trunk(318) in both the FAM-A3b block and the B6 re-triage verdicts; 10 dated corrections (mergeable flips: sgl-project#46191 REAL -> WATCH-HIGH with the re-derive-vs-QSA_RING_GROUPS=2 instruction, sgl-project#39845 DISPUTED as UNKNOWN-not-true plus the general 'UNKNOWN is no signal' rule; sgl-project#44344 -> SECTION R DOA with the entry-268 unconditional-clear subsumption citation and the mergeable-is-a-trap note; sgl-project#42525 target path moved fla->linear; sgl-project#45520 STILL OPEN -> re-apply rule ARMED FOR MERGE sgl-project#6; b12x-package archived claim DISPUTED with the frozen cutlass-dsl==4.8.0 pin risk standing; sgl-project#46153+sgl-project#41624 gated as ONE QAD bundle; sgl-project#37106 PARKED on eroded motivation with named revive conditions; sgl-project#32479 gated behind the PCIe-IPC measurement and DE-COUPLED from sgl-project#45391; sgl-project#44462's NEXT note spent); NEW S-item S-B12XDENSE filed
@Jiminator Jiminator changed the title [SM120] Add optional b12x PCIe all-reduce (one-shot + fused RMSNorm + FP8 DMA ring) [Feat] [SM120] distributed: Add optional b12x PCIe all-reduce (one-shot + fused RMSNorm + FP8 DMA ring) Nov 3, 2026

@Jiminator Jiminator left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some requests from a initial review. will dig further

channel = self._channels.kwargs()
if channel is None:
return None
return self._oneshot.all_reduce(inp, **channel)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please port this to b12x 1.5.1’s plan API, check the version at construction, and drop the untested b12x.distributed import path. 1.5.1 is the release that installs with our nvidia-cutlass-dsl==4.8.0 pin, and its all-reduce calls require plan=, so init logs “enabled” and the first all-reduce raises TypeError.

channel, so the binding has to exist before the first node is recorded.
Older releases had no such requirement and get a no-op here.
"""
channel = None if self._channels is None else self._channels.kwargs(stream)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we give each graph_capture() session its own channel id instead of one per stream? The decode, EAGLE draft, prefill and row-bucket runners all capture on the global capture stream, so the second capture fails on every rank with logical channel was already captured.

maybe_b12x_context = (
nullcontext()
if b12x_comm is None or b12x_comm.disabled
else b12x_comm.capture(stream=stream)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we enter the b12x capture context in ViTCudaGraphRunner._capture_context and the InternVL ViT runner too, or keep b12x off for captures outside graph_capture()? With SGLANG_VIT_ENABLE_CUDA_GRAPH=1 and a TP vision tower, the all-reduce inside the graph fails with requires an active pool.capture() graph-owned channel context.

buf=buf,
group=group,
)
if b12x_us * _PROBE_MARGIN > nccl_us:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove AUTOTUNE (SGLANG_B12X_ONESHOT_MAX_BYTES=-1) or make it rank-consistent. Each rank breaks out of the ladder on its own timing and the catch-all except hides the probe timeout, so two ranks can end with caps of 0 and 8 MiB and send the same 4 KiB all-reduce to NCCL on one and b12x on the other.

# Prefer the communicator-native fused AR+RMSNorm when one exists
# (AITER on AMD, b12x on SM120); both go through the same coordinator
# entry point, which returns None when the fused path does not apply.
if _use_aiter or envs.SGLANG_B12X_PCIE_AR.get():

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we take the b12x branch only when the full TP group owes the sum? This path always reduces over get_tp_group(), so under attention CP (e.g. TP8 with --attn-cp-size 2) a sum owed over the 4 attention-TP ranks is reduced over all 8, and the attention-TP group is never used.

return False
if inp.dtype not in _SUPPORTED_DTYPES or not inp.is_contiguous():
return False
return inp.numel() * inp.element_size() <= self.oneshot_max_size

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please match b12x’s eligibility here, or call the channel’s should_allreduce. b12x also requires the byte count to be a multiple of 16 and raises ValueError instead of declining, so fp32 [1,1], [2,1], [3,1] and bf16 [1,3] crash instead of falling back to NCCL.

# the caller own that mapping, so reserve a few and hand them out in first-seen
# order. Streams beyond this keep the NCCL path rather than reusing a channel,
# which would be a correctness bug, not a slowdown.
_MAX_STREAM_CHANNELS = 8

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: could we document the memory the eight prepared channels cost? Enabling b12x takes about 224 MiB per GPU at TP2 and TP4 with 1.2.6, and 480 MiB at TP2 with 1.5.1, while the PR description only mentions the DMA slab.

# the model can issue (chunked_prefill_size x hidden x dtype).
SGLANG_B12X_DMA_MAX_BYTES = EnvInt(256 * 1024 * 1024)
# DMA ring wire format: "ring" / "a2a" for FP8, empty string for BF16.
SGLANG_B12X_DMA_FP8 = EnvStr("ring")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we default the DMA wire to BF16 and make FP8 opt-in? On a 16 MiB bf16 all-reduce at TP2, the FP8 ring gives 3.15% relative L2 error against 0.18% on the BF16 wire.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants