Skip to content

Commit 92d412b

Browse files
committed
CollectiveX: clamp the nccl-ep low-latency ladder around an unfixed upstream combine race
gb300 nccl-ep low-latency BF16 failed correctness at T=256 and nowhere else: rungs T=1..128 pass with max relative error 0.0039, T=256 gave 0.4704. Both chained checks were clean there (chain_last_output_error exactly 0.0), so it is the drained oracle catching it, and the error is specifically the combine_values elementwise check -- the magnitude was computed at all, so shapes matched and it is not a payload or metadata failure. STOCHASTIC, measured: 1 failure in 5 executions of this shard (30934943352, 30969695966, 30981959724, 31200578704 pass; 31191722264 fails; an earlier container-import failure excluded as infra). The distribution is BIMODAL with nothing between the two values, which is a discrete corrupted write rather than a tolerance sitting near normal variance. MECHANISM: nccl_ep's low_latency.cu is a port of DeepEP's PRE-FIX low-latency combine. In the combine recv pipeline the reduction warps read shared memory and then mbarrier_arrive(emptyBarriers[stageIdx]) with no fence.proxy.async.shared::cta between, so the producer's next TMA load can overwrite a stage while consumer reads are in flight. DeepEP closed exactly this with a one-line fence in PR #642, and its defect signature matches ours: stochastic, Blackwell-only, top rung, one wrong row. The fence is absent at our pin AND at NVIDIA/nccl master, so this is unfixed upstream rather than a stale pin. The adapter's own known defect -- LL handles aliasing parity signal slots, NVIDIA/nccl#2303 -- is ruled out: this adapter uses one handle per group rebound per shape, and that defect wedges rather than corrupting silently. THE CLAMP IS NOT A SAFETY BOUNDARY and the code says so in those words. The fence is missing on every combine recv; T=256 is only the rung with the most pipeline iterations, and its receive plane is not even full. Lower rungs are less likely to hit the race, not immune. What the clamp buys is operational: unclamped, gb300 alone reds roughly 1 run in 5 on a defect we have already fully characterized, autopublish requires every leg green, and each red burns a full-matrix attempt plus a triage cycle -- while a stochastic red at a known site no longer teaches anyone anything. Against that: publishing a T=256 latency for a kernel we know stochastically corrupts its output at that rung, with a correctness gate that samples only 3-5 invocations per case, is itself the safety-implying act. A clamped ladder with an explicit note is the more honest artifact. Implemented as two constants mirroring ep_deepep_v2, and the split matters here more than it did there: nccl-ep sized its receive from spec.max_tokens_per_rank, which follows the CLAMPED ladder, so a naive clamp would have shrunk the transport footprint too and quietly re-based what T=128 measures. create_buffer now reads _LL_BUFFER_CAP directly. Tests pin both halves. Restore _LL_LADDER_CAP to _LL_BUFFER_CAP when a fixed wheel ships. Upstream report to NVIDIA/nccl is the primary action and is not blocked by this. 176 tests green.
1 parent bfd6b4b commit 92d412b

2 files changed

Lines changed: 56 additions & 5 deletions

File tree

‎experimental/CollectiveX/bench/ep_nccl.py‎

Lines changed: 31 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,29 @@
6464
# change can't silently truncate the id on the non-root ranks.
6565
_UNIQUE_ID_MAX_BYTES = 256
6666

67+
# Low-latency receive sizing, deliberately two numbers, mirroring ep_deepep_v2: _LL_BUFFER_CAP
68+
# sizes the pre-allocated receive (and so the transport footprint), _LL_LADDER_CAP bounds which
69+
# token counts are measured. Separating them lets the ladder be clamped around a kernel defect
70+
# without moving the footprint and silently re-basing the rungs that remain.
71+
#
72+
# The ladder sits below the buffer because nccl_ep's low_latency.cu is a port of DeepEP's
73+
# PRE-FIX low-latency combine: in the combine recv pipeline the reduction warps read shared
74+
# memory and then mbarrier_arrive(emptyBarriers[stageIdx]) with no fence.proxy.async.shared::cta
75+
# between, so the producer's next TMA load can overwrite a stage while consumer reads are still
76+
# in flight. DeepEP closed exactly this with a one-line fence in PR #642; the fence is absent
77+
# both at our pin and at NVIDIA/nccl master, so it is unfixed upstream.
78+
#
79+
# Observed on gb300 EP8 BF16 at T=256: 1 failure in 5 executions, bimodal -- healthy rows give
80+
# max relative error 0.0039, the failure gave 0.4704, with nothing between, which is a discrete
81+
# corrupted write rather than tolerance noise.
82+
#
83+
# THIS CLAMP IS NOT A SAFETY BOUNDARY. The fence is missing on every combine recv; T=256 is only
84+
# the rung with the most pipeline iterations, and the receive plane is not even full there. Lower
85+
# rungs are LESS LIKELY to hit the race, not immune. Restore _LL_LADDER_CAP to _LL_BUFFER_CAP
86+
# once a fixed wheel ships.
87+
_LL_BUFFER_CAP = 256
88+
_LL_LADDER_CAP = 128
89+
6790

6891
class NCCLEPBackend(EPBackend):
6992
name = "nccl-ep"
@@ -122,10 +145,10 @@ def __init__(self, args, rank, world_size, local_rank, device):
122145

123146
def buffer_cap(self, args):
124147
if self._ll:
125-
# LL pre-allocates the fixed [num_local_experts, cap*num_ranks, hidden] receive
126-
# buffer, so cap is a hard per-rank dispatch-slot bound (same 256 as ep_deepep_v2 /
127-
# ep_uccl low-latency; the harness clamps the decode ladder and reports drops).
128-
return 256
148+
# Bounds which token counts are MEASURED. Below _LL_BUFFER_CAP today because the
149+
# combine recv pipeline races (see the constants above); the harness reports every
150+
# dropped rung rather than silently truncating.
151+
return _LL_LADDER_CAP
129152
return None
130153

131154
# ---- helpers -----------------------------------------------------------------------------
@@ -187,7 +210,10 @@ def _bootstrap_comm(self):
187210
def create_buffer(self, spec):
188211
"""Bootstrap the communicator, create the EP group sized from the ladder maximum, and
189212
allocate the persistent receive/combine buffers reused across every ladder shape."""
190-
self.max_dispatch = spec.max_tokens_per_rank
213+
# Sized from the BUFFER cap, not from the measured ladder, so clamping the ladder
214+
# around the combine race does not also shrink the transport footprint -- which drives
215+
# recv-slot memory traffic and would change what the remaining rungs measure.
216+
self.max_dispatch = _LL_BUFFER_CAP if self._ll else spec.max_tokens_per_rank
191217
hidden = self.args.hidden
192218
self._bootstrap_comm()
193219
# max_recv_tokens_per_rank: HT requires >0 and >= max_dispatch; LL auto-derives when 0.

‎experimental/CollectiveX/tests/test_backends.py‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -273,6 +273,31 @@ def test_a_backend_is_deployed_unless_it_says_otherwise(self):
273273
self.assertEqual(ep_backend.EPBackend.path_status, "deployed")
274274

275275

276+
class NcclLowLatencyLadderClamp(unittest.TestCase):
277+
"""The measured ladder is clamped below the receive buffer around an unfixed upstream race."""
278+
279+
def _module(self):
280+
with mock.patch.dict(sys.modules, _stub_modules()):
281+
import importlib
282+
import ep_nccl
283+
return importlib.reload(ep_nccl)
284+
285+
def test_the_ladder_is_clamped_below_the_buffer(self):
286+
# Two separate numbers on purpose: clamping the ladder must not shrink the transport
287+
# footprint, or the rungs that remain quietly measure a smaller receive plane.
288+
m = self._module()
289+
self.assertLess(m._LL_LADDER_CAP, m._LL_BUFFER_CAP)
290+
self.assertLessEqual(m._LL_BUFFER_CAP, 511)
291+
292+
def test_buffer_cap_bounds_the_ladder_not_the_allocation(self):
293+
# buffer_cap() is what the harness clamps the ladder against, so it must report the
294+
# LADDER cap; the allocation reads _LL_BUFFER_CAP directly.
295+
m = self._module()
296+
source = (ROOT / "bench" / "ep_nccl.py").read_text()
297+
self.assertIn("return _LL_LADDER_CAP", source)
298+
self.assertIn("_LL_BUFFER_CAP if self._ll else", source)
299+
300+
276301
class RoundtripStagingGate(unittest.TestCase):
277302
"""`roundtrip` must mean dispatch -> combine in every row, or it is not comparable: the gate
278303
is `stage_device_work` alone, with `CX_FP8_CONSUME=dequant` as the sole opt-out."""

0 commit comments

Comments
 (0)