Repository navigation
[Fix] dp-attention: Carry a rank's ShutdownReq to its DP peers through the MLP-sync gather - #48447
Conversation
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
|
/rerun-group rust-server |
|
🚀 🚀 🚀 |
…h the MLP-sync gather With attention DP and a group-local control broadcast, a ShutdownReq reaches one DP group only: the Rust server's rank-0 listener sends it to its own group. Those ranks set gracefully_exit, finish the step and leave; the other groups start the next step and block in the MLP-sync gather, which then fails against the departed ranks (gloo "Connection closed by peer"), so the engine exits non-zero on every stop. rust_server/shutdown.py carried this as a TODO. Every attention-DP rank already gathers once per scheduler step, so the shutdown rides that gather instead of a new collective: - MLPSyncBatchInfo gains an is_shutting_down column; the gather yields any_shutting_down (max over DP ranks). - SchedulerDPAttnAdapter.prepare_mlp_sync_batch passes the rank's gracefully_exit into the gather (DPShutdownVote) and, when a peer DP rank honored a ShutdownReq, calls Scheduler.handle_peer_shutdown, which sets gracefully_exit, so every rank leaves at the same step boundary, before any further gather. - The overlap prefill loop launches nothing new on shutdown, so it runs one more batch selection while this rank's own vote is unsent (shutdown_vote_pending), and only where a cross-DP gather carries it: more than one DP rank and the gather not skipped. A paused engine runs no gather, so a rank that honors a ShutdownReq while paused leaves without voting; the other DP groups stop when their launcher stops them. Documented on DPShutdownVote. Tests: test_dp_attn.py covers peer vs own vs no shutdown on the adapter, the vote going out with the gather, where a vote is pending, the scheduler wiring, the local and fallback rows, and the real gloo exchange across worlds 2-8; test_disagg_idle_step_counters.py covers the overlap prefill loop's extra selection (and takes ruff-format's blank-line layout); test_prefill_result_polling.py's fake adapter gains shutdown_vote_pending. Co-authored-by: shodoco <shodoco@users.noreply.github.com>
8626284 to
42cb56d
Compare
|
/rerun-group rust-server |
|
🚀 🚀 |
|
This PR is the bottom of a two-PR stack; the merge request for the stack, with the CI evidence, is on sgl#48492. Please merge this PR first, then sgl#48492. |
Brings in main through the stack's base (#48447), which now carries #48399's rename of the force-shutdown switch. This change follows that rename in the merge: - python/sglang/srt/rust_server/server.py: the force-shutdown read that this change moves to the top of install_sigterm_drain now reads envs.SGLANG_FORCE_SHUTDOWN.get(); get_bool_env_var("SGL_FORCE_SHUTDOWN") no longer exists on main. - test/registered/unit/entrypoints/test_rust_server_shutdown.py: the drain tests' helper sets SGLANG_FORCE_SHUTDOWN. - rust/sglang-server (drain.rs, lib.rs, runtime.rs): the three new doc comments name SGLANG_FORCE_SHUTDOWN, as the surrounding docs do on main.
|
Bottom of the two-PR stack, now at |
#48447, this branch's base, merged to main as 8b4a677 (squashed). Its content was already here, so the four files where the squash met this branch's own edits on top of it (scheduler.py, dp_attn.py, rust_server/shutdown.py, test_dp_attn.py) keep this branch's version; the diff against main is unchanged.
Motivation
With attention DP and a group-local control broadcast (which the Rust server forces on with
enable_dp_attention_local_control_broadcast), aShutdownReqreaches one DP group only: the Rust server's rank-0 listener sends it to its own group. Those ranks setgracefully_exit, finish the step and leave the event loop. The other DP groups start the next step and block in the MLP-sync gather (exchange_sync_rows), which then fails against the departed ranks with glooConnection closed by peer, so an engine with more than one DP group exits non-zero on every stop.python/sglang/srt/rust_server/shutdown.pycarried this as a TODO. The Python server's control broadcast spans the whole TP group, so it does not hit this.Modifications
Every attention-DP rank already gathers once per scheduler step, so the shutdown rides that gather instead of adding a collective:
MLPSyncBatchInfogains one column,is_shutting_down; the gather yieldsany_shutting_down(max over DP ranks). The fallback row carries 0.SchedulerDPAttnAdapter.prepare_mlp_sync_batchpasses the rank'sgracefully_exitinto the gather (DPShutdownVote) and, when a peer DP rank honored aShutdownReq, callsScheduler.handle_peer_shutdown, which setsgracefully_exitand logsScheduler exiting with its DP peers. Every rank then leaves at the top of its next step, before any further gather.Scheduler.init_dp_attn_adapterwires both callbacks.gracefully_exitis set, so a prefill rank that honored theShutdownReqwould leave before its vote reached a gather. It now runs one more batch selection while the rank's own vote is unsent (SchedulerDPAttnAdapter.shutdown_vote_pending), and only where a cross-DP gather carries it: more than one DP rank andSGLANG_SCHEDULER_SKIP_ALL_GATHERoff. The adapter marks the vote sent after the gather.rust_server/shutdown.pyreplaces the TODO with the mechanism.DPShutdownVote: a paused engine runs no gather, so a rank that honors aShutdownReqwhile paused leaves without voting; the other DP groups stop when their launcher stops them.Unchanged: the HTTP drain. Rank 0 still sends the
ShutdownReqonly once its drain completes.Tests
test/registered/unit/managers/scheduler_components/test_dp_attn.py(base-a-test-cpu): newTestDPShutdownVotecovers a peer's vote stopping this rank, this rank's own vote not reporting a peer, no vote keeping it running, the vote marked sent only after the gather, a vote pending only where a cross-DP gather carries it (two DP ranks; not one, not with the gather skipped), the scheduler wiring, and the local and fallback rows.TestSyncRowExchangeGloo(real gloo processes, worlds 2-8) now has one non-root rank vote in the last round only and asserts every rank sees it in that round only.test/registered/unit/managers/test_disagg_idle_step_counters.py: newTestPrefillOverlapShutdownVoteasserts the overlap prefill loop runs exactly one selection after shutdown while a vote is pending, and none without one. This file also takes ruff-format's blank-line layout around the new class.test/registered/unit/disaggregation/test_prefill_result_polling.py: its fake DP adapter gainsshutdown_vote_pending, which the prefill loop now reads.Checklist
CI States
Latest PR Test (Base): ❌ Run #42242905961
Latest PR Test (Extra): ❌ Run #42242906027
Latest PR Test (AMD ROCm 10): ❌ Run #42242905838
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: