Repository navigation
Commit 92d412b
committed
CollectiveX: clamp the nccl-ep low-latency ladder around an unfixed upstream combine race
gb300 nccl-ep low-latency BF16 failed correctness at T=256 and nowhere else: rungs T=1..128 pass
with max relative error 0.0039, T=256 gave 0.4704. Both chained checks were clean there
(chain_last_output_error exactly 0.0), so it is the drained oracle catching it, and the error is
specifically the combine_values elementwise check -- the magnitude was computed at all, so shapes
matched and it is not a payload or metadata failure.
STOCHASTIC, measured: 1 failure in 5 executions of this shard (30934943352, 30969695966,
30981959724, 31200578704 pass; 31191722264 fails; an earlier container-import failure excluded as
infra). The distribution is BIMODAL with nothing between the two values, which is a discrete
corrupted write rather than a tolerance sitting near normal variance.
MECHANISM: nccl_ep's low_latency.cu is a port of DeepEP's PRE-FIX low-latency combine. In the
combine recv pipeline the reduction warps read shared memory and then
mbarrier_arrive(emptyBarriers[stageIdx]) with no fence.proxy.async.shared::cta between, so the
producer's next TMA load can overwrite a stage while consumer reads are in flight. DeepEP closed
exactly this with a one-line fence in PR #642, and its defect signature matches ours: stochastic,
Blackwell-only, top rung, one wrong row. The fence is absent at our pin AND at NVIDIA/nccl master,
so this is unfixed upstream rather than a stale pin. The adapter's own known defect -- LL handles
aliasing parity signal slots, NVIDIA/nccl#2303 -- is ruled out: this adapter uses one handle per
group rebound per shape, and that defect wedges rather than corrupting silently.
THE CLAMP IS NOT A SAFETY BOUNDARY and the code says so in those words. The fence is missing on
every combine recv; T=256 is only the rung with the most pipeline iterations, and its receive plane
is not even full. Lower rungs are less likely to hit the race, not immune. What the clamp buys is
operational: unclamped, gb300 alone reds roughly 1 run in 5 on a defect we have already fully
characterized, autopublish requires every leg green, and each red burns a full-matrix attempt plus
a triage cycle -- while a stochastic red at a known site no longer teaches anyone anything.
Against that: publishing a T=256 latency for a kernel we know stochastically corrupts its output at
that rung, with a correctness gate that samples only 3-5 invocations per case, is itself the
safety-implying act. A clamped ladder with an explicit note is the more honest artifact.
Implemented as two constants mirroring ep_deepep_v2, and the split matters here more than it did
there: nccl-ep sized its receive from spec.max_tokens_per_rank, which follows the CLAMPED ladder,
so a naive clamp would have shrunk the transport footprint too and quietly re-based what T=128
measures. create_buffer now reads _LL_BUFFER_CAP directly. Tests pin both halves.
Restore _LL_LADDER_CAP to _LL_BUFFER_CAP when a fixed wheel ships. Upstream report to NVIDIA/nccl
is the primary action and is not blocked by this. 176 tests green.1 parent bfd6b4b commit 92d412b
2 files changed
Lines changed: 56 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
67 | 90 | | |
68 | 91 | | |
69 | 92 | | |
| |||
122 | 145 | | |
123 | 146 | | |
124 | 147 | | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
129 | 152 | | |
130 | 153 | | |
131 | 154 | | |
| |||
187 | 210 | | |
188 | 211 | | |
189 | 212 | | |
190 | | - | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
191 | 217 | | |
192 | 218 | | |
193 | 219 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
273 | 273 | | |
274 | 274 | | |
275 | 275 | | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
276 | 301 | | |
277 | 302 | | |
278 | 303 | | |
| |||
0 commit comments