Skip to content

fix(nccl): disable graph buffer registration for pausable graph pools - #40648

Merged
Oasis-Git merged 9 commits into
mainfrom
jiajun/nccl-graph-register-pausable-graph-pool
Oct 1, 2026
Merged

Oasis-Git merged 9 commits into
mainfrom
jiajun/nccl-graph-register-pausable-graph-pool

Conversation

@guapisolo

@guapisolo guapisolo commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Disable NCCL graph buffer registration when the CUDA graph pool is pausable under memory saver.

Symptom & Reproduction

  • Symptom: a two-node TP16/EP16 MoE engine with --enable-memory-saver and SGLANG_MEMORY_SAVER_CUDA_GRAPH=1 never finishes a decode step after a release_memory_occupation / resume_memory_occupation(kv_cache, cuda_graph) cycle; all 16 schedulers wait in copy_done.synchronize() until the 300 s scheduler watchdog fires.
  • Reproduction: zai-org/GLM-5.3 FP8 on two 8x H200 nodes with NCCL_NVLS_ENABLE=1 NCCL_CUMEM_ENABLE=1, --cuda-graph-max-bs-decode 256, memory saver, one release/resume cycle, then 512 requests (recipe); expected 512/512, main gives 0/512 and a 16-rank watchdog.

Root Cause

  1. NCCL_GRAPH_REGISTER (default on) registers graph-captured collective buffers for the graph lifetime.
  2. SGLANG_MEMORY_SAVER_CUDA_GRAPH captures the graphs into a torch_memory_saver region.
  3. release_memory_occupation(cuda_graph) releases its pages; resume maps new pages at the same addresses.
  4. Registered NVLS/network paths keep the released pages, so ranks diverge and the tp_group deadlocks.
  5. CustomAllreduce skips its registration under tms_cudagraph, but multi-node TP disables it.

Fix

handle_data_parallelism calls _disable_nccl_graph_buffer_registration (added in #38936 for the TP LM-head all-to-all) for a second trigger, _graph_pool_is_pausable (enable_memory_saver and SGLANG_MEMORY_SAVER_CUDA_GRAPH), so the schedulers inherit NCCL_GRAPH_REGISTER=0 from the launcher before any communicator exists. This is the mitigation torch_memory_saver documents for the same interaction (fzyzcjy/torch_memory_saver#88, fzyzcjy/torch_memory_saver#107). The helper takes a reason for its warning when NCCL_GRAPH_REGISTER is set explicitly, which still wins. Engines whose graph pool is not pausable, including DP-attention engines, are unchanged.

Verification

  • GLM-5.3 TP16 recipe, main vs the pausable-pool trigger: 0/512 with 30752 register comm lines per node vs 512/512 in 109 s with none, both on the first commit and after the rebase onto 81cb895 (re-run). The DP-attention rule dropped at this head never fires in this recipe.
  • GLM-5.3 DP-attention recipe (--enable-dp-attention --dp-size 2, same pool and cycle): main 1/512 in 2/2 runs vs 512/512 with the pausable-pool trigger alone (recipe).
  • Downstream tree (pausable-pool trigger, earlier base), two-node TP16/EP16, memory saver, seven-step release/resume lifecycle: unpatched 0/512 vs patched 512/512 with zero registrations and no throughput change.
  • Downstream RL colocated-training run, four engines with weight updates and graph release/resume every step: three rollout/train cycles completed.
  • test/registered/unit/server_args/test_server_args.py -k NcclGraphRegister: not rerun at this head; the test file now matches main.

Review Focus

  • handle_data_parallelism / parallel_hook.py: the guard must run in the launcher before any NCCL communicator exists.
  • Cost: memory-saver engines lose NCCL zero-copy registration for graph-captured collectives; the pausable-pool run measured no throughput change.
  • Scope: an earlier revision also disabled registration for DP-attention decode graphs (fix(nccl): disable graph buffer registration when DP attention replays decode graphs #40649); it is dropped because no run isolated a hang with registration on a non-pausable pool.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ✅ Run #36804137764
Latest PR Test (Extra): ❌ Run #36804137573
Latest PR Test (AMD ROCm 10): ❌ Run #36804137768

@guapisolo

Copy link
Copy Markdown
Collaborator Author

cc cmt

Reproduction recipe on open weights: zai-org/GLM-5.3 (FP8), two H200 nodes, TP16/EP16, upstream main @ 66f19f5 (control) vs this branch @ acc2f77 (fix). Same command on both nodes (--node-rank 1 on the second):

export NCCL_NVLS_ENABLE=1 NCCL_CUMEM_ENABLE=1        # with NVLS off, NCCL registers nothing at capture and the hang cannot occur
export SGLANG_MEMORY_SAVER_CUDA_GRAPH=1
export NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,TUNING,REG NCCL_DEBUG_FILE=/tmp/nccl-%h-%p.log   # optional: registration evidence
python -m sglang.launch_server --model-path zai-org/GLM-5.3 --trust-remote-code \
  --tp-size 16 --ep-size 16 --nnodes 2 --node-rank 0 --dist-init-addr <node0-ip>:20101 --nccl-port 20132 \
  --host <this-node-ip> --port 20100 --mem-fraction-static 0.8 --max-running-requests 256 --chunked-prefill-size 1024 \
  --context-length 8192 --cuda-graph-backend-prefill disabled --cuda-graph-max-bs-decode 256 \
  --watchdog-timeout 300 --skip-server-warmup --enable-memory-saver

One graph-pool release/resume cycle, then 512 concurrent /generate requests (temperature 1, max_new_tokens 1024, plain text prompts):

curl -X POST http://<node0-ip>:20100/release_memory_occupation -H 'Content-Type: application/json' -d '{"tags":["kv_cache","cuda_graph"]}'
curl -X POST http://<node0-ip>:20100/resume_memory_occupation  -H 'Content-Type: application/json' -d '{"tags":["kv_cache","cuda_graph"]}'

Results (paired, same nodes, same weights, minutes apart):

  • main: capture registers 30752 buffers per node (register comm … size 20971520 ×30296 and size 79691776 ×456 in the REG log; 5652 captured collectives are routed NVLS_TREE, e.g. AllReduce: 3145728 Bytes -> Algo NVLS_TREE proto SIMPLE). After the cycle: 18 prefill batches, 0 decode batches, 0/512 completions, all 16 schedulers hit the 300 s watchdog. py-spy dump --native before the watchdog: 7 of 8 ranks per node in torch.cuda.Event.synchronize (copy_done.synchronize() in process_batch_result_decode), the remaining rank in dist.broadcast, native frames inside libcuda, no CUDA or NCCL error.
  • this branch: identical NVLS_TREE picks, 0 register comm lines, schedulers inherit NCCL_GRAPH_REGISTER=0, 512/512 in 109 s.

Two knobs decide whether the bug surface exists at all, which is why smaller setups do not see it: (1) NCCL only graph-registers collectives it routes through NVLS/CollNet, so --cuda-graph-max-bs-decode 256 is what makes the decode all-reduce (~3 MiB at bs 256 × hidden 6144) large enough for NVLS_TREE; at bs ≤ 64 the ≤0.8 MiB all-reduces go Tree/Ring, nothing is registered, and the same release/resume cycle completes 512/512 on main. (2) NCCL_NVLS_ENABLE=1 (auto-detection left NVLS off in this container image; with it off nothing is registered either).

Environment: NCCL 2.29.7+cuda13.2, torch 2.13.0+cu130, sgl-kernel 0.4.7, driver 580.105.08, torch_memory_saver 0.0.9.post1, IB (8× mlx5) + NVLink between the nodes.

@guapisolo
guapisolo added this pull request to stack #40984 September 23, 2026 23:03
@guapisolo
guapisolo removed this pull request from stack #40984 September 23, 2026 23:04
@guapisolo guapisolo changed the title fix(nccl): disable graph buffer registration when the graph pool is pausable fix(nccl): disable graph buffer registration for pausable graph pools and DP-attention decode graphs Sep 23, 2026
@guapisolo

Copy link
Copy Markdown
Collaborator Author

cc cmt

Reproduction recipe on open weights: zai-org/GLM-5.3 (FP8), two H200 nodes, TP16/EP16 with --enable-dp-attention --dp-size 2 (attention TP8, one DP rank per node), upstream main @ 66f19f5 (control) vs this branch @ 9cd2c68 (fix). Same command on both nodes (--node-rank 1 on the second):

export NCCL_NVLS_ENABLE=1 NCCL_CUMEM_ENABLE=1          # with NVLS off, NCCL registers nothing at capture and the failure cannot occur
export SGLANG_MEMORY_SAVER_CUDA_GRAPH=1
export SGLANG_ENABLE_CUDA_GRAPH_DEDUP=1 SGLANG_ENABLE_GRAPH_POOL_BORROW=1 SGLANG_ENABLE_GRAPH_POOL_PRECARVE=1
export NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,TUNING,REG NCCL_DEBUG_FILE=/tmp/nccl-%h-%p.log   # optional: registration evidence
python -m sglang.launch_server --model-path zai-org/GLM-5.3 --trust-remote-code \
  --tp-size 16 --ep-size 16 --nnodes 2 --node-rank 0 --dist-init-addr <node0-ip>:20101 --nccl-port 20132 \
  --host <this-node-ip> --port 20100 --enable-dp-attention --dp-size 2 \
  --mem-fraction-static 0.8 --max-running-requests 256 --chunked-prefill-size 1024 --context-length 8192 \
  --cuda-graph-backend-prefill disabled --cuda-graph-max-bs-decode 256 --watchdog-timeout 300 --skip-server-warmup \
  --enable-memory-saver --enable-return-routed-experts

One graph-pool release/resume cycle (what a colocated RL engine does before every rollout), then 512 concurrent /generate requests (~1500-token prompts, max_new_tokens 4096, temperature 1):

curl -X POST http://<node0-ip>:20100/release_memory_occupation -H 'Content-Type: application/json' -d '{"tags":["kv_cache","cuda_graph"]}'
curl -X POST http://<node0-ip>:20100/resume_memory_occupation  -H 'Content-Type: application/json' -d '{"tags":["kv_cache","cuda_graph"]}'

Results (paired, same nodes, minutes apart):

  • main, two runs, identical outcome: capture registers 496 buffers per node (register comm … size 262144000, the pre-carved graph-pool block). After the cycle the first decode step completes, then the DP1 schedulers (node 1) die with RuntimeError: The size of tensor a (128) must match the size of tensor b (512) at non-singleton dimension 0 in cuda_graph_buffer_registry.fill_from (eager_runner.load_batch), i.e. the gathered token count no longer matches the local batch, while the DP0 schedulers (node 0) wait inside dp_scatter (dp_attention.py) until the 300 s watchdog. 1/512 requests completed.
  • this branch: 0 registrations, NCCL_GRAPH_REGISTER=0 inherited by the schedulers, 512/512 in 618 s.

Preconditions are the same as in #40648: NVLS on and --cuda-graph-max-bs-decode 256 (a ~3 MiB decode all-reduce is what NCCL routes through NVLS_TREE and registers; at bs ≤ 64, or with NVLS off, nothing is registered and main completes 512/512 in DP2 and DP4). Without the memory-saver graph pool the same DP2/DP4 runs also completed on main even though the plain-allocator graph buffers were registered, so on this model the failure needs the pool's physical pages to be replaced by the release/resume cycle; consequently the parent PR's pausable-pool trigger alone also passes this recipe (512/512), and the DP-attention trigger added here is the guard for DP-attention engines whose graph pool is not pausable.

Environment: NCCL 2.29.7+cuda13.2, torch 2.13.0+cu130, sgl-kernel 0.4.7, driver 580.105.08, torch_memory_saver 0.0.9.post1, IB (8× mlx5) + NVLink.

guapisolo and others added 4 commits September 23, 2026 17:59
…ausable

With --enable-memory-saver and SGLANG_MEMORY_SAVER_CUDA_GRAPH the decode
graphs are captured into a torch_memory_saver region. That region is a
cuMem (VMM) allocation, so NCCL's default graph registration
(NCCL_GRAPH_REGISTER=1) binds the NVLS multicast and network paths of
every captured collective to the pool's physical pages for the lifetime
of the graph. release_memory_occupation(cuda_graph) unmaps and releases
those pages and resume maps new ones at the same addresses, so after a
release/resume cycle each replay reads the new pages through the
addresses and the released pages through the registered paths. Ranks
receive different all-reduce results and the TP group deadlocks once
their batches diverge: a two-node TP16/EP16 MoE engine produced 0/512
completions with all 16 schedulers stuck in copy_done.synchronize().

Multi-node TP groups disable the custom all-reduce, which already skips
its own IPC registration under this region (tms_cudagraph), so every
all-reduce goes through NCCL and its registration was the uncovered
path. Reuse the existing guard for the TP LM-head all-to-all and set
NCCL_GRAPH_REGISTER=0 in the launcher process whenever the graph pool
is pausable; an explicit value still wins with a warning that names the
reason.

Paired bare-server runs with the same seven-step release/resume
lifecycle: unpatched 0/512 with a 16-rank watchdog and 2048 NCCL
registrations of the 150 MiB graph-pool span; patched 512/512 with zero
registrations and no throughput change. An RL colocated-training run
(four TP16/EP16 engines, weight updates and graph release/resume every
step) completed three rollout/train cycles.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s decode graphs (#40649)

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Restore the upstream TP LM-head paragraph of
_disable_nccl_graph_buffer_registration unchanged and state the pausable
graph pool and DP-attention hazards as the live constraints only; the
reproduction history stays in the PR description. The pausable-pool
predicate covers every CUDA graph captured into the torch_memory_saver
region, not only decode graphs. Comment-only change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
handle_cuda_graph_config runs before handle_data_parallelism and always
leaves cuda_graph_config set, so _dp_attention_replays_decode_graphs no
longer needs the defensive getattr and None check; a pipeline reorder now
fails loudly instead of silently keeping NCCL graph registration on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@guapisolo
guapisolo force-pushed the jiajun/nccl-graph-register-pausable-graph-pool branch from 1628784 to 5aa4433 Compare September 24, 2026 00:59
@guapisolo

Copy link
Copy Markdown
Collaborator Author

cc cmt

Re-run on the rebased head, and a correction on which batch sizes trigger the hang

Both recipes above were re-run after the rebase: upstream main @ 81cb895 (control) vs this branch @ 5aa4433 (fix). Setup: the same two nodes, the posted commands and load (512 requests), and one /release_memory_occupation + /resume_memory_occupation cycle with tags: ["kv_cache", "cuda_graph"] before the load. Harness-only differences: the DP recipe ran on ports 21100/21101/21132, and every run also set --random-seed 1234 and NCCL_SOCKET_IFNAME/GLOO_SOCKET_IFNAME.

recipe main @ 81cb895 this branch @ 5aa4433
pausable graph pool, TP16/EP16 hang: no decode progress, 0/512, all 16 schedulers hit the 300 s watchdog; 30752 buffers registered per node, as before 512/512 in 109 s, 0 registrations
DP attention, --dp-size 2 crash in both runs: 1/512 512/512 in 603 s, 0 registrations
  • [P1] Pausable graph pool on main: after one release/resume cycle the engine serves nothing until the watchdog kills it, so every request fails and the engine must be restarted. A colocated RL engine goes through this cycle before every rollout. py-spy dump --native before the watchdog shows the same rank layout as the original run:
    • node 0: 7 ranks in copy_done.synchronize() (process_batch_result_decode) and 1 rank in dist.broadcast.
    • node 1: 3 ranks (TP8-10) replaying a decode CUDA graph, and 5 ranks (TP11-15) still in an eager extend forward, loading the fused-MoE Triton kernel (cuModuleLoadData).
    • 15 of the 16 ranks are in libcuda; the broadcast rank waits in c10d::Work::wait.
    • Correction to the first comment: "7 of 8 ranks per node in Event.synchronize" is true only for node 0.
  • [P1] DP attention on main: the engine also goes down after the cycle, but the failure site has moved since the original report. About 55 s into the load, four of the eight DP1 schedulers (TP9-12 on node 1; the repeat also has exactly four) raise RuntimeError: Trying to create tensor with negative dimension -384 in ForwardBatch._pad_inputs_to_size, called from prepare_mlp_sync_batch in _prepare_eager_forward_batch. The DP0 schedulers then exit on a gloo Connection closed by peer instead of waiting for the watchdog. The pad target (this rank's synchronized token count) is 384 below its local input_ids length. A split of 128 synchronized vs 512 local tokens would match the earlier 128-vs-512 mismatch in cuda_graph_buffer_registry.fill_from, but only the difference is logged, and 128 and 512 are also this recipe's per-DP-rank max-running and chunked-prefill sizes. Each node still registers 496 buffers of size 262144000.

Correction: graphs whose decode all-reduce is routed TREE also register buffers

The earlier comments tied registration to a decode all-reduce large enough (about 3 MiB) for NVLS_TREE. That is incomplete. I assigned each register comm line in rank 0's capture log to the most recent decode all-reduce tuner line (bytes / 12288 = bs). Every rank registers the same number of buffers: 3844, which is 30752 per node / 8.

captured decode bs decode all-reduce algo buffers registered per rank while capturing that graph
1-104 TREE 0
112-160 TREE 4
168-256 NVLS_TREE 318

The log does not show which collective in a graph registered. The graphs at 112 and above are also the only ones that contain a RING/SIMPLE collective: 57 picks at cap 256 = 3 × 19 graphs, and 21 at cap 160 = 3 × 7. So the 4 registrations at 112-160 need not come from the TREE all-reduce.

Pausable-pool recipe on main, one run per row, changing only the capture cap and the client load (the server runs at most 256 requests):

--cuda-graph-max-bs-decode client concurrency result
256 512 hang (the run above)
256 160 hang: no decode progress, 16/16 watchdog
160 160 hang: no decode progress, 16/16 watchdog; no captured collective routed NVLS_TREE (the only NVLS_TREE picks are eager prefill all-reduces), 224 registrations per node (7 graphs × 4 buffers × 8 ranks)
160 512 512/512 in 484 s; all 51 logged decode batches ran eager at 210-256 running requests, so whether any registering graph was replayed is not visible

What this suggests on this setup (hidden 6144, 2×8 H200, NCCL 2.29.7, NCCL_NVLS_ENABLE=1 NCCL_CUMEM_ENABLE=1):

  • Registration, and therefore the hang, does not need NVLS_TREE. In all three hanging runs the decode batches fit a captured graph from a bucket that registered at capture (bs 160 or 256). In the one passing run, decode stayed above the cap and ran eager. The smallest trigger measured is --cuda-graph-max-bs-decode 160 with 160 concurrent requests.
  • Not measured on this head:
    • A load that keeps decode at 104 or below while a registering cap is captured. That no hang is expected there comes only from the 0-registration range in the capture log; the first comment's bs ≤ 64 pass was on the pre-rebase main.
    • Whether the 112-160 registrations still happen with NVLS or cuMem off.
    • Repeats of these four rows.
  • The fix does not depend on any of these sizes. This branch defaults NCCL_GRAPH_REGISTER=0 whenever the graph pool is pausable (--enable-memory-saver with SGLANG_MEMORY_SAVER_CUDA_GRAPH=1) or DP attention with --dp-size > 1 runs decode CUDA graphs. An explicit NCCL_GRAPH_REGISTER setting still wins, with a warning. Both fixed runs logged 0 register comm lines.

@Oasis-Git Oasis-Git left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some suggestions on code style

graph buffer registration: registered graph-pool temporaries deadlock the
exchange under DP-rank ramps."""

def _resolve(self, **kwargs):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can still keep the original function signiture

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc cmt

Done in 0854ad8: _resolve keeps its original signature; test_without_all_to_all_env_is_untouched now stubs _dp_attention_replays_decode_graphs instead of forcing eager decode.

Comment on lines +328 to +341
A pausable graph pool breaks the registrations too: torch_memory_saver
resume maps new physical pages behind the pool's virtual addresses while
the registrations made at capture keep the released pages, so replayed
collectives move data through pages the rest of the graph no longer uses
and the TP group deadlocks once the ranks diverge. The custom all-reduce
likewise skips its IPC registration in this pool (`tms_cudagraph`).

DP attention replays its `dp_gather` / `dp_scatter` collectives over the
TP group inside the decode graphs; with registered buffers the TP group
stops on its GPUs within the first decode steps, without a CUDA or NCCL
error, while eager decode completes.

Must run before the schedulers create their NCCL communicators, which
inherit this environment. An explicit setting wins.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can make the comment shorter

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc cmt

Done in a546be5: the two paragraphs this PR added are folded into one sentence; the upstream LM-head paragraph is unchanged.

@Oasis-Git Oasis-Git added run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) labels Sep 28, 2026
guapisolo and others added 2 commits September 27, 2026 23:35
Fold the two paragraphs this PR added to the
_disable_nccl_graph_buffer_registration docstring into one sentence per
hazard; the call-site reasons and the PR description carry the rest. The
upstream TP LM-head paragraph is unchanged. Comment-only change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Restore TestTpLmHeadAllToAllNcclGraphRegister._resolve to its upstream
form. test_without_all_to_all_env_is_untouched now keeps the DP-attention
decode-graph rule out of its check by stubbing
_dp_attention_replays_decode_graphs instead of forcing eager decode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
guapisolo added a commit to radixark/miles that referenced this pull request Sep 28, 2026
…ross nodes

The recipe script called U.execute_train / U.hf_download_dataset /
U.convert_checkpoint, which main no longer exports (#2415); it now goes
through args.create_backend() like run_deepseek_v4.py.

On 8xH200 nodes a BF16 engine (~1.1 TB of weights) does not fit one node, so
the engine has to span nodes, and o_groups=8 caps attention TP at 8, which
means DP attention. --sglang-dp-size > 1 now turns on
--sglang-enable-dp-attention and --sglang-enable-dp-lm-head (the 129280-row
vocab does not split over 24 TP ranks) and sizes --sglang-chunked-prefill-size
at dp*1024, since DP attention splits the budget per rank and each share must
stay a multiple of the dsv4 backend's 256-token page. With DP attention it
also sets NCCL_GRAPH_REGISTER=0: the DP-attention gather inside a pausable
decode graph hangs on buffer registration until sgl-project/sglang#40648 and
#40649 land.

--sglang-engram-host-table-layout picks the engram host-table layout; it
defaults to per_rank when the engine spans nodes, because the shared layout
opens rank 0's memfd through /proc and cannot cross hosts. --optimizer-offload
and --disk-offload now fail fast together, matching Megatron's own exclusion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aoshen02 added a commit to aoshen02/vllm that referenced this pull request Sep 29, 2026
Custom allreduce IPC-registers the buffers a captured graph reads, but
cudaIpcGetMemHandle rejects cuMem memory ("invalid argument" at
custom_all_reduce.cuh:164) and the worker exits. While a capture allocates
from a cuMem graph pool, copy the input into the pre-registered IPC buffer
instead (registered=False), as SGLang does for its cuMem graph pool
(sgl-project/sglang#7873). Custom all-gather and reduce-scatter already
always use the pre-registered buffer. Reproduced on GB300 with Qwen3-8B TP4,
VLLM_ALLREDUCE_USE_FLASHINFER=0, VLLM_ALLREDUCE_USE_SYMM_MEM=0 and
fuse_allreduce_rms disabled.

NCCL graph registration retains the cuMem handles of graph buffers, pinning
about 400-510 MiB per GPU through sleep, and leaves stale registrations
after wake remaps the pool (hangs and wrong results in SGLang,
fzyzcjy/torch_memory_saver#88, sgl-project/sglang#40648). When cuMem graph
pools are enabled, the worker defaults NCCL_GRAPH_REGISTER=0 before creating
communicators and refuses an explicit non-zero value.

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
guapisolo added a commit that referenced this pull request Sep 29, 2026
Remove _dp_attention_replays_decode_graphs and keep only the pausable
graph pool trigger. The DP-attention evidence never isolated
registration on a non-pausable pool: the open-weight GLM-5.3 runs
completed on main without the memory-saver graph pool, and the
pausable-pool trigger alone fixes the DP-attention recipe. Revert the
matching test stub so test_without_all_to_all_env_is_untouched is back
to its upstream form.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@guapisolo guapisolo changed the title fix(nccl): disable graph buffer registration for pausable graph pools and DP-attention decode graphs fix(nccl): disable graph buffer registration for pausable graph pools Oct 1, 2026
Reword the pausable graph pool warning reason, the _graph_pool_is_pausable
docstring, and the _disable_nccl_graph_buffer_registration paragraph after
fzyzcjy/torch_memory_saver#88: resume keeps the virtual addresses but maps
new physical pages, so NCCL's capture-time registrations point at released
pages and graph replay can hang. The docstring now names the `cuda_graph`
tag instead of one release_memory_occupation call, since an untagged
release pauses the pool as well. Comment and message text only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Oasis-Git Oasis-Git removed the run-ci-extra CI: also run the extra suite (requires run-ci) label Oct 1, 2026
@Oasis-Git

Copy link
Copy Markdown
Collaborator

@Oasis-Git
Oasis-Git merged commit dcf4e18 into main Oct 1, 2026
157 of 177 checks passed
@Oasis-Git
Oasis-Git deleted the jiajun/nccl-graph-register-pausable-graph-pool branch October 1, 2026 19:34
ziang-and pushed a commit to zianglih/sglang that referenced this pull request Oct 2, 2026
… graph registration (sgl-project#40648, sgl-project#40978, sgl-project#40979, sgl-project#40980, sgl-project#40977, sgl-project#41972)

Retain colocated training fixes above the release MiMo implementation, including address-stable Marlin reload and mandatory TopK layer IDs. Disable NCCL graph registration for TP LM-head all-to-all and pausable graph pools.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
nv-hwoo pushed a commit to nv-hwoo/sglang that referenced this pull request Oct 20, 2026
… graph registration (sgl-project#40648, sgl-project#40978, sgl-project#40979, sgl-project#40980, sgl-project#40977, sgl-project#41972)

Retain colocated training fixes above the release MiMo implementation, including address-stable Marlin reload and mandatory TopK layer IDs. Disable NCCL graph registration for TP LM-head all-to-all and pausable graph pools.
nanjiangwill pushed a commit to modal-projects/sglang that referenced this pull request Oct 21, 2026
… graph registration (sgl-project#40648, sgl-project#40978, sgl-project#40979, sgl-project#40980, sgl-project#40977, sgl-project#41972)

Retain colocated training fixes above the release MiMo implementation, including address-stable Marlin reload and mandatory TopK layer IDs. Disable NCCL graph registration for TP LM-head all-to-all and pausable graph pools.

(cherry picked from commit bc1ad00, partial: MXFP4 Marlin MoE in-place reload only; MiMo loader changes dropped)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VDwTQn4gGFgVhzGXFs4aJx
Rockdu pushed a commit that referenced this pull request Oct 30, 2026
… graph registration (#40648, #40978, #40979, #40980, #40977, #41972)

Retain colocated training fixes above the release MiMo implementation, including address-stable Marlin reload and mandatory TopK layer IDs. Disable NCCL graph registration for TP LM-head all-to-all and pausable graph pools.
Rockdu pushed a commit that referenced this pull request Oct 30, 2026
… graph registration (#40648, #40978, #40979, #40980, #40977, #41972)

Retain colocated training fixes above the release MiMo implementation, including address-stable Marlin reload and mandatory TopK layer IDs. Disable NCCL graph registration for TP LM-head all-to-all and pausable graph pools.
Rockdu pushed a commit that referenced this pull request Oct 31, 2026
… graph registration (#40648, #40978, #40979, #40980, #40977, #41972)

Retain colocated training fixes above the release MiMo implementation, including address-stable Marlin reload and mandatory TopK layer IDs. Disable NCCL graph registration for TP LM-head all-to-all and pausable graph pools.
Rockdu pushed a commit that referenced this pull request Oct 31, 2026
… graph registration (#40648, #40978, #40979, #40980, #40977, #41972)

Retain colocated training fixes above the release MiMo implementation, including address-stable Marlin reload and mandatory TopK layer IDs. Disable NCCL graph registration for TP LM-head all-to-all and pausable graph pools.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants