Skip to content

[Fix] dp-attention: Carry a rank's ShutdownReq to its DP peers through the MLP-sync gather - #48447

Merged
hanming-lu merged 3 commits into
mainfrom
oss-sync-a35b6e939f
Nov 6, 2026
Merged

hanming-lu merged 3 commits into
mainfrom
oss-sync-a35b6e939f

Conversation

@metamergebot

@metamergebot metamergebot commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

With attention DP and a group-local control broadcast (which the Rust server forces on with enable_dp_attention_local_control_broadcast), a ShutdownReq reaches one DP group only: the Rust server's rank-0 listener sends it to its own group. Those ranks set gracefully_exit, finish the step and leave the event loop. The other DP groups start the next step and block in the MLP-sync gather (exchange_sync_rows), which then fails against the departed ranks with gloo Connection closed by peer, so an engine with more than one DP group exits non-zero on every stop. python/sglang/srt/rust_server/shutdown.py carried this as a TODO. The Python server's control broadcast spans the whole TP group, so it does not hit this.

Modifications

Every attention-DP rank already gathers once per scheduler step, so the shutdown rides that gather instead of adding a collective:

  • MLPSyncBatchInfo gains one column, is_shutting_down; the gather yields any_shutting_down (max over DP ranks). The fallback row carries 0.
  • SchedulerDPAttnAdapter.prepare_mlp_sync_batch passes the rank's gracefully_exit into the gather (DPShutdownVote) and, when a peer DP rank honored a ShutdownReq, calls Scheduler.handle_peer_shutdown, which sets gracefully_exit and logs Scheduler exiting with its DP peers. Every rank then leaves at the top of its next step, before any further gather. Scheduler.init_dp_attn_adapter wires both callbacks.
  • The overlap prefill loop launches nothing new once gracefully_exit is set, so a prefill rank that honored the ShutdownReq would leave before its vote reached a gather. It now runs one more batch selection while the rank's own vote is unsent (SchedulerDPAttnAdapter.shutdown_vote_pending), and only where a cross-DP gather carries it: more than one DP rank and SGLANG_SCHEDULER_SKIP_ALL_GATHER off. The adapter marks the vote sent after the gather.
  • The module docstring of rust_server/shutdown.py replaces the TODO with the mechanism.
  • Known limitation, documented on DPShutdownVote: a paused engine runs no gather, so a rank that honors a ShutdownReq while paused leaves without voting; the other DP groups stop when their launcher stops them.

Unchanged: the HTTP drain. Rank 0 still sends the ShutdownReq only once its drain completes.

Tests

  • test/registered/unit/managers/scheduler_components/test_dp_attn.py (base-a-test-cpu): new TestDPShutdownVote covers a peer's vote stopping this rank, this rank's own vote not reporting a peer, no vote keeping it running, the vote marked sent only after the gather, a vote pending only where a cross-DP gather carries it (two DP ranks; not one, not with the gather skipped), the scheduler wiring, and the local and fallback rows. TestSyncRowExchangeGloo (real gloo processes, worlds 2-8) now has one non-root rank vote in the last round only and asserts every rank sees it in that round only.
  • test/registered/unit/managers/test_disagg_idle_step_counters.py: new TestPrefillOverlapShutdownVote asserts the overlap prefill loop runs exactly one selection after shutdown while a vote is pending, and none without one. This file also takes ruff-format's blank-line layout around the new class.
  • test/registered/unit/disaggregation/test_prefill_result_polling.py: its fake DP adapter gains shutdown_vote_pending, which the prefill loop now reads.

Checklist

  • The three test files above pass locally on this branch: 62 passed, 110 subtests.
  • pre-commit passes on the changed files.
  • No new collective, flag, or environment variable.

CI States

Latest PR Test (Base): ❌ Run #42242905961
Latest PR Test (Extra): ❌ Run #42242906027
Latest PR Test (AMD ROCm 10): ❌ Run #42242905838

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@metamergebot metamergebot added the run-ci CI: run the baseline test suite on this PR label Nov 4, 2026
@metamergebot

Copy link
Copy Markdown
Collaborator Author

/rerun-group rust-server

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/core/test_srt_endpoint.py

🚀 1-gpu-h100 (4 tests): ✅ View workflow run

cd test/ && python3 registered/vlm/test_rust_native_mm_e2e.py
cd test/ && python3 registered/vlm/test_rust_native_mm_mmmu.py
cd test/ && python3 registered/e2e/rust_server/test_mm_transport_1node_cpu.py
cd test/ && python3 registered/e2e/rust_server/test_mm_transport_1node_cuda_ipc.py

🚀 2-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/rust_server/test_mm_transport_1node_tp2_shm.py
cd test/ && python3 registered/e2e/rust_server/test_mm_transport_2node_cpu.py

…h the MLP-sync gather

With attention DP and a group-local control broadcast, a ShutdownReq
reaches one DP group only: the Rust server's rank-0 listener sends it to
its own group. Those ranks set gracefully_exit, finish the step and leave;
the other groups start the next step and block in the MLP-sync gather,
which then fails against the departed ranks (gloo "Connection closed by
peer"), so the engine exits non-zero on every stop. rust_server/shutdown.py
carried this as a TODO.

Every attention-DP rank already gathers once per scheduler step, so the
shutdown rides that gather instead of a new collective:

- MLPSyncBatchInfo gains an is_shutting_down column; the gather yields
  any_shutting_down (max over DP ranks).
- SchedulerDPAttnAdapter.prepare_mlp_sync_batch passes the rank's
  gracefully_exit into the gather (DPShutdownVote) and, when a peer DP
  rank honored a ShutdownReq, calls Scheduler.handle_peer_shutdown, which
  sets gracefully_exit, so every rank leaves at the same step boundary,
  before any further gather.
- The overlap prefill loop launches nothing new on shutdown, so it runs
  one more batch selection while this rank's own vote is unsent
  (shutdown_vote_pending), and only where a cross-DP gather carries it:
  more than one DP rank and the gather not skipped.

A paused engine runs no gather, so a rank that honors a ShutdownReq while
paused leaves without voting; the other DP groups stop when their launcher
stops them. Documented on DPShutdownVote.

Tests: test_dp_attn.py covers peer vs own vs no shutdown on the adapter,
the vote going out with the gather, where a vote is pending, the
scheduler wiring, the local and fallback rows, and the real gloo exchange
across worlds 2-8; test_disagg_idle_step_counters.py covers the overlap
prefill loop's extra selection (and takes ruff-format's blank-line
layout); test_prefill_result_polling.py's fake adapter gains
shutdown_vote_pending.

Co-authored-by: shodoco <shodoco@users.noreply.github.com>
@metamergebot

Copy link
Copy Markdown
Collaborator Author

/rerun-group rust-server

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

🚀 1-gpu-5090 (5 tests): ✅ View workflow run

cd test/ && python3 registered/core/test_srt_endpoint.py  # base-b
cd test/ && python3 registered/vlm/test_rust_native_mm_e2e.py  # base-b
cd test/ && python3 registered/vlm/test_rust_native_mm_mmmu.py  # base-b
cd test/ && python3 registered/e2e/rust_server/test_mm_transport_1node_cpu.py  # base-b
cd test/ && python3 registered/e2e/rust_server/test_mm_transport_1node_cuda_ipc.py  # base-b

🚀 2-gpu-h100 (2 tests): ❌ View workflow run

cd test/ && python3 registered/e2e/rust_server/test_mm_transport_1node_tp2_shm.py  # base-b
cd test/ && python3 registered/e2e/rust_server/test_mm_transport_2node_cpu.py  # base-b

@metamergebot metamergebot removed the run-ci CI: run the baseline test suite on this PR label Nov 6, 2026
@metamergebot

Copy link
Copy Markdown
Collaborator Author

This PR is the bottom of a two-PR stack; the merge request for the stack, with the CI evidence, is on sgl#48492. Please merge this PR first, then sgl#48492.

metamergebot added a commit that referenced this pull request Nov 6, 2026
Brings in main through the stack's base (#48447), which now carries
#48399's rename of the force-shutdown switch. This change follows that
rename in the merge:

- python/sglang/srt/rust_server/server.py: the force-shutdown read that
  this change moves to the top of install_sigterm_drain now reads
  envs.SGLANG_FORCE_SHUTDOWN.get(); get_bool_env_var("SGL_FORCE_SHUTDOWN")
  no longer exists on main.
- test/registered/unit/entrypoints/test_rust_server_shutdown.py: the
  drain tests' helper sets SGLANG_FORCE_SHUTDOWN.
- rust/sglang-server (drain.rs, lib.rs, runtime.rs): the three new doc
  comments name SGLANG_FORCE_SHUTDOWN, as the surrounding docs do on main.
@metamergebot

Copy link
Copy Markdown
Collaborator Author

Bottom of the two-PR stack, now at 4163f1ba (main merged in for sgl#48399). The merge request for the stack, with the CI evidence, is on sgl#48492. Please merge this PR first, then sgl#48492.

@hanming-lu
hanming-lu merged commit 8b4a677 into main Nov 6, 2026
110 of 128 checks passed
@hanming-lu
hanming-lu deleted the oss-sync-a35b6e939f branch November 6, 2026 18:52
metamergebot added a commit that referenced this pull request Nov 6, 2026
#48447, this branch's base, merged to main as 8b4a677 (squashed). Its
content was already here, so the four files where the squash met this
branch's own edits on top of it (scheduler.py, dp_attn.py,
rust_server/shutdown.py, test_dp_attn.py) keep this branch's version;
the diff against main is unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants