Repository navigation
fix(nccl): disable graph buffer registration for pausable graph pools - #40648
Conversation
3e9f652 to
acc2f77
Compare
|
cc cmt Reproduction recipe on open weights: zai-org/GLM-5.3 (FP8), two H200 nodes, TP16/EP16, upstream export NCCL_NVLS_ENABLE=1 NCCL_CUMEM_ENABLE=1 # with NVLS off, NCCL registers nothing at capture and the hang cannot occur
export SGLANG_MEMORY_SAVER_CUDA_GRAPH=1
export NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,TUNING,REG NCCL_DEBUG_FILE=/tmp/nccl-%h-%p.log # optional: registration evidence
python -m sglang.launch_server --model-path zai-org/GLM-5.3 --trust-remote-code \
--tp-size 16 --ep-size 16 --nnodes 2 --node-rank 0 --dist-init-addr <node0-ip>:20101 --nccl-port 20132 \
--host <this-node-ip> --port 20100 --mem-fraction-static 0.8 --max-running-requests 256 --chunked-prefill-size 1024 \
--context-length 8192 --cuda-graph-backend-prefill disabled --cuda-graph-max-bs-decode 256 \
--watchdog-timeout 300 --skip-server-warmup --enable-memory-saverOne graph-pool release/resume cycle, then 512 concurrent curl -X POST http://<node0-ip>:20100/release_memory_occupation -H 'Content-Type: application/json' -d '{"tags":["kv_cache","cuda_graph"]}'
curl -X POST http://<node0-ip>:20100/resume_memory_occupation -H 'Content-Type: application/json' -d '{"tags":["kv_cache","cuda_graph"]}'Results (paired, same nodes, same weights, minutes apart):
Two knobs decide whether the bug surface exists at all, which is why smaller setups do not see it: (1) NCCL only graph-registers collectives it routes through NVLS/CollNet, so Environment: NCCL 2.29.7+cuda13.2, torch 2.13.0+cu130, sgl-kernel 0.4.7, driver 580.105.08, torch_memory_saver 0.0.9.post1, IB (8× mlx5) + NVLink between the nodes. |
|
cc cmt Reproduction recipe on open weights: zai-org/GLM-5.3 (FP8), two H200 nodes, TP16/EP16 with export NCCL_NVLS_ENABLE=1 NCCL_CUMEM_ENABLE=1 # with NVLS off, NCCL registers nothing at capture and the failure cannot occur
export SGLANG_MEMORY_SAVER_CUDA_GRAPH=1
export SGLANG_ENABLE_CUDA_GRAPH_DEDUP=1 SGLANG_ENABLE_GRAPH_POOL_BORROW=1 SGLANG_ENABLE_GRAPH_POOL_PRECARVE=1
export NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,TUNING,REG NCCL_DEBUG_FILE=/tmp/nccl-%h-%p.log # optional: registration evidence
python -m sglang.launch_server --model-path zai-org/GLM-5.3 --trust-remote-code \
--tp-size 16 --ep-size 16 --nnodes 2 --node-rank 0 --dist-init-addr <node0-ip>:20101 --nccl-port 20132 \
--host <this-node-ip> --port 20100 --enable-dp-attention --dp-size 2 \
--mem-fraction-static 0.8 --max-running-requests 256 --chunked-prefill-size 1024 --context-length 8192 \
--cuda-graph-backend-prefill disabled --cuda-graph-max-bs-decode 256 --watchdog-timeout 300 --skip-server-warmup \
--enable-memory-saver --enable-return-routed-expertsOne graph-pool release/resume cycle (what a colocated RL engine does before every rollout), then 512 concurrent curl -X POST http://<node0-ip>:20100/release_memory_occupation -H 'Content-Type: application/json' -d '{"tags":["kv_cache","cuda_graph"]}'
curl -X POST http://<node0-ip>:20100/resume_memory_occupation -H 'Content-Type: application/json' -d '{"tags":["kv_cache","cuda_graph"]}'Results (paired, same nodes, minutes apart):
Preconditions are the same as in #40648: NVLS on and Environment: NCCL 2.29.7+cuda13.2, torch 2.13.0+cu130, sgl-kernel 0.4.7, driver 580.105.08, torch_memory_saver 0.0.9.post1, IB (8× mlx5) + NVLink. |
…ausable With --enable-memory-saver and SGLANG_MEMORY_SAVER_CUDA_GRAPH the decode graphs are captured into a torch_memory_saver region. That region is a cuMem (VMM) allocation, so NCCL's default graph registration (NCCL_GRAPH_REGISTER=1) binds the NVLS multicast and network paths of every captured collective to the pool's physical pages for the lifetime of the graph. release_memory_occupation(cuda_graph) unmaps and releases those pages and resume maps new ones at the same addresses, so after a release/resume cycle each replay reads the new pages through the addresses and the released pages through the registered paths. Ranks receive different all-reduce results and the TP group deadlocks once their batches diverge: a two-node TP16/EP16 MoE engine produced 0/512 completions with all 16 schedulers stuck in copy_done.synchronize(). Multi-node TP groups disable the custom all-reduce, which already skips its own IPC registration under this region (tms_cudagraph), so every all-reduce goes through NCCL and its registration was the uncovered path. Reuse the existing guard for the TP LM-head all-to-all and set NCCL_GRAPH_REGISTER=0 in the launcher process whenever the graph pool is pausable; an explicit value still wins with a warning that names the reason. Paired bare-server runs with the same seven-step release/resume lifecycle: unpatched 0/512 with a 16-rank watchdog and 2048 NCCL registrations of the 150 MiB graph-pool span; patched 512/512 with zero registrations and no throughput change. An RL colocated-training run (four TP16/EP16 engines, weight updates and graph release/resume every step) completed three rollout/train cycles. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s decode graphs (#40649) Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Restore the upstream TP LM-head paragraph of _disable_nccl_graph_buffer_registration unchanged and state the pausable graph pool and DP-attention hazards as the live constraints only; the reproduction history stays in the PR description. The pausable-pool predicate covers every CUDA graph captured into the torch_memory_saver region, not only decode graphs. Comment-only change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
handle_cuda_graph_config runs before handle_data_parallelism and always leaves cuda_graph_config set, so _dp_attention_replays_decode_graphs no longer needs the defensive getattr and None check; a pipeline reorder now fails loudly instead of silently keeping NCCL graph registration on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1628784 to
5aa4433
Compare
|
cc cmt Re-run on the rebased head, and a correction on which batch sizes trigger the hang Both recipes above were re-run after the rebase: upstream
Correction: graphs whose decode all-reduce is routed The earlier comments tied registration to a decode all-reduce large enough (about 3 MiB) for
The log does not show which collective in a graph registered. The graphs at 112 and above are also the only ones that contain a Pausable-pool recipe on
What this suggests on this setup (hidden 6144, 2×8 H200, NCCL 2.29.7,
|
Oasis-Git
left a comment
There was a problem hiding this comment.
Left some suggestions on code style
| graph buffer registration: registered graph-pool temporaries deadlock the | ||
| exchange under DP-rank ramps.""" | ||
|
|
||
| def _resolve(self, **kwargs): |
There was a problem hiding this comment.
I think we can still keep the original function signiture
There was a problem hiding this comment.
cc cmt
Done in 0854ad8: _resolve keeps its original signature; test_without_all_to_all_env_is_untouched now stubs _dp_attention_replays_decode_graphs instead of forcing eager decode.
| A pausable graph pool breaks the registrations too: torch_memory_saver | ||
| resume maps new physical pages behind the pool's virtual addresses while | ||
| the registrations made at capture keep the released pages, so replayed | ||
| collectives move data through pages the rest of the graph no longer uses | ||
| and the TP group deadlocks once the ranks diverge. The custom all-reduce | ||
| likewise skips its IPC registration in this pool (`tms_cudagraph`). | ||
|
|
||
| DP attention replays its `dp_gather` / `dp_scatter` collectives over the | ||
| TP group inside the decode graphs; with registered buffers the TP group | ||
| stops on its GPUs within the first decode steps, without a CUDA or NCCL | ||
| error, while eager decode completes. | ||
|
|
||
| Must run before the schedulers create their NCCL communicators, which | ||
| inherit this environment. An explicit setting wins. |
There was a problem hiding this comment.
maybe we can make the comment shorter
There was a problem hiding this comment.
cc cmt
Done in a546be5: the two paragraphs this PR added are folded into one sentence; the upstream LM-head paragraph is unchanged.
Fold the two paragraphs this PR added to the _disable_nccl_graph_buffer_registration docstring into one sentence per hazard; the call-site reasons and the PR description carry the rest. The upstream TP LM-head paragraph is unchanged. Comment-only change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Restore TestTpLmHeadAllToAllNcclGraphRegister._resolve to its upstream form. test_without_all_to_all_env_is_untouched now keeps the DP-attention decode-graph rule out of its check by stubbing _dp_attention_replays_decode_graphs instead of forcing eager decode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ross nodes The recipe script called U.execute_train / U.hf_download_dataset / U.convert_checkpoint, which main no longer exports (#2415); it now goes through args.create_backend() like run_deepseek_v4.py. On 8xH200 nodes a BF16 engine (~1.1 TB of weights) does not fit one node, so the engine has to span nodes, and o_groups=8 caps attention TP at 8, which means DP attention. --sglang-dp-size > 1 now turns on --sglang-enable-dp-attention and --sglang-enable-dp-lm-head (the 129280-row vocab does not split over 24 TP ranks) and sizes --sglang-chunked-prefill-size at dp*1024, since DP attention splits the budget per rank and each share must stay a multiple of the dsv4 backend's 256-token page. With DP attention it also sets NCCL_GRAPH_REGISTER=0: the DP-attention gather inside a pausable decode graph hangs on buffer registration until sgl-project/sglang#40648 and #40649 land. --sglang-engram-host-table-layout picks the engram host-table layout; it defaults to per_rank when the engine spans nodes, because the shared layout opens rank 0's memfd through /proc and cannot cross hosts. --optimizer-offload and --disk-offload now fail fast together, matching Megatron's own exclusion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Custom allreduce IPC-registers the buffers a captured graph reads, but
cudaIpcGetMemHandle rejects cuMem memory ("invalid argument" at
custom_all_reduce.cuh:164) and the worker exits. While a capture allocates
from a cuMem graph pool, copy the input into the pre-registered IPC buffer
instead (registered=False), as SGLang does for its cuMem graph pool
(sgl-project/sglang#7873). Custom all-gather and reduce-scatter already
always use the pre-registered buffer. Reproduced on GB300 with Qwen3-8B TP4,
VLLM_ALLREDUCE_USE_FLASHINFER=0, VLLM_ALLREDUCE_USE_SYMM_MEM=0 and
fuse_allreduce_rms disabled.
NCCL graph registration retains the cuMem handles of graph buffers, pinning
about 400-510 MiB per GPU through sleep, and leaves stale registrations
after wake remaps the pool (hangs and wrong results in SGLang,
fzyzcjy/torch_memory_saver#88, sgl-project/sglang#40648). When cuMem graph
pools are enabled, the worker defaults NCCL_GRAPH_REGISTER=0 before creating
communicators and refuses an explicit non-zero value.
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove _dp_attention_replays_decode_graphs and keep only the pausable graph pool trigger. The DP-attention evidence never isolated registration on a non-pausable pool: the open-weight GLM-5.3 runs completed on main without the memory-saver graph pool, and the pausable-pool trigger alone fixes the DP-attention recipe. Revert the matching test stub so test_without_all_to_all_env_is_untouched is back to its upstream form. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reword the pausable graph pool warning reason, the _graph_pool_is_pausable docstring, and the _disable_nccl_graph_buffer_registration paragraph after fzyzcjy/torch_memory_saver#88: resume keeps the virtual addresses but maps new physical pages, so NCCL's capture-time registrations point at released pages and graph replay can hang. The docstring now names the `cuda_graph` tag instead of one release_memory_occupation call, since an untagged release pauses the pool as well. Comment and message text only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… graph registration (sgl-project#40648, sgl-project#40978, sgl-project#40979, sgl-project#40980, sgl-project#40977, sgl-project#41972) Retain colocated training fixes above the release MiMo implementation, including address-stable Marlin reload and mandatory TopK layer IDs. Disable NCCL graph registration for TP LM-head all-to-all and pausable graph pools. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
… graph registration (sgl-project#40648, sgl-project#40978, sgl-project#40979, sgl-project#40980, sgl-project#40977, sgl-project#41972) Retain colocated training fixes above the release MiMo implementation, including address-stable Marlin reload and mandatory TopK layer IDs. Disable NCCL graph registration for TP LM-head all-to-all and pausable graph pools.
… graph registration (sgl-project#40648, sgl-project#40978, sgl-project#40979, sgl-project#40980, sgl-project#40977, sgl-project#41972) Retain colocated training fixes above the release MiMo implementation, including address-stable Marlin reload and mandatory TopK layer IDs. Disable NCCL graph registration for TP LM-head all-to-all and pausable graph pools. (cherry picked from commit bc1ad00, partial: MXFP4 Marlin MoE in-place reload only; MiMo loader changes dropped) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VDwTQn4gGFgVhzGXFs4aJx
Summary
Disable NCCL graph buffer registration when the CUDA graph pool is pausable under memory saver.
Symptom & Reproduction
--enable-memory-saverandSGLANG_MEMORY_SAVER_CUDA_GRAPH=1never finishes a decode step after arelease_memory_occupation/resume_memory_occupation(kv_cache, cuda_graph)cycle; all 16 schedulers wait incopy_done.synchronize()until the 300 s scheduler watchdog fires.NCCL_NVLS_ENABLE=1 NCCL_CUMEM_ENABLE=1,--cuda-graph-max-bs-decode 256, memory saver, one release/resume cycle, then 512 requests (recipe); expected 512/512,maingives 0/512 and a 16-rank watchdog.Root Cause
NCCL_GRAPH_REGISTER(default on) registers graph-captured collective buffers for the graph lifetime.SGLANG_MEMORY_SAVER_CUDA_GRAPHcaptures the graphs into atorch_memory_saverregion.release_memory_occupation(cuda_graph)releases its pages; resume maps new pages at the same addresses.tp_groupdeadlocks.CustomAllreduceskips its registration undertms_cudagraph, but multi-node TP disables it.Fix
handle_data_parallelismcalls_disable_nccl_graph_buffer_registration(added in #38936 for the TP LM-head all-to-all) for a second trigger,_graph_pool_is_pausable(enable_memory_saverandSGLANG_MEMORY_SAVER_CUDA_GRAPH), so the schedulers inheritNCCL_GRAPH_REGISTER=0from the launcher before any communicator exists. This is the mitigation torch_memory_saver documents for the same interaction (fzyzcjy/torch_memory_saver#88, fzyzcjy/torch_memory_saver#107). The helper takes areasonfor its warning whenNCCL_GRAPH_REGISTERis set explicitly, which still wins. Engines whose graph pool is not pausable, including DP-attention engines, are unchanged.Verification
mainvs the pausable-pool trigger: 0/512 with 30752register commlines per node vs 512/512 in 109 s with none, both on the first commit and after the rebase onto 81cb895 (re-run). The DP-attention rule dropped at this head never fires in this recipe.--enable-dp-attention --dp-size 2, same pool and cycle):main1/512 in 2/2 runs vs 512/512 with the pausable-pool trigger alone (recipe).test/registered/unit/server_args/test_server_args.py -k NcclGraphRegister: not rerun at this head; the test file now matchesmain.Review Focus
handle_data_parallelism/parallel_hook.py: the guard must run in the launcher before any NCCL communicator exists.🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ✅ Run #36804137764
Latest PR Test (Extra): ❌ Run #36804137573
Latest PR Test (AMD ROCm 10): ❌ Run #36804137768