[wanzhenchn]

Re-submits #2154 with the root-cause fix for the hang #2315 reverted it over, and with answers to each concern #2315.

1. The hang

#2315's diagnosis was correct, and this PR implements it rather than working around it.

47d77d3 turned is_deferred_out off on a P/D producer. _dp_draft_lockstep_active() was reading that same flag to mean "pipeline parallelism is off". Under DP attention a producer processing an output-less middle prefill chunk therefore skipped the aligning propose() while its idle DP peers entered the draft collectives and waited in DPMetadata.make's all_reduce. One rank short, the all_reduce never completed — 142 warmup requests sent, zero returned after 210 s.

fix(k3-disagg): keep P/D producers in the DP draft lockstep makes the predicate ask is_pipeline_parallel directly. A P/D producer keeps its drafter loaded and still owes its peers the collectives propose() carries, so it stays in the lockstep; only PP ranks are excluded, and now by name. The two flags had been coextensive until #2154 gave is_deferred_out a second reason to be false; using one to test the other was the latent coupling, and naming the condition removes that class of bug rather than special-casing the producer.

A second commit narrows the trigger itself. fix(k3-disagg): limit the producer's deferred-output opt-out to recurrent state: the opt-out exists so the consumer applies the handoff token T0 exactly once, but only a recurrent state notices a second application — a paged write lands at T0's own position, so repeating it is idempotent. Family.has_recurrent_state states this per family, and V4, MLA and MHA producers keep the deferred-output path they used before P/D handoff existed. The V4 configuration that hung is no longer on the modified path at all.

Coverage against recurrence:

• test_draft_lockstep_reads_pipeline_parallel_not_deferred_output asserts _dp_draft_lockstep_active reads exactly is_pipeline_parallel and nothing else, so a future stand-in for the flag fails here.
• test_only_recurrent_state_producers_give_up_deferred_output sweeps every attention family against a producer config: V4, MLA and MHA keep deferred output, the hybrids and pure GDN give it up.
• test_every_family_is_covered fails if a family is added without stating its answer, so the sweep cannot go stale.

2. The fixes the rollback withdrew

#2315 noted that it was a full rollback, that removing listener readiness and save fencing can reintroduce failures outside Kimi-K3 including other Mooncake P/D and offload users, and that those fixes need to be reintroduced. That is what this PR does, and it reintroduces them behind the draft-lockstep fix rather than in front of it: the Mooncake listener bind/readiness handoff, the CUDA-stream fence before asynchronous offload saves, the offload load-start tracking that publishes restored prefixes, the asymmetric-only DCP guard on slot-region transfer, port 0 surviving endpoint parsing, the full first-decode draft width, and the Kimi-K3 P/D state handoff itself.

3. Validation

Full unit suite in the ROCm container, tests/plugin excluded (the image has no sglang.srt.layers):

Tree	Result
This branch	16 failed, 8355 passed, 207 skipped, 3 xfailed, 4 errors
main baseline, same environment	16 failed, 8298 passed, 207 skipped, 3 xfailed, 4 errors

The failure sets are identical: test_fused_compress_ragged (4), test_mega_mxfp4_method (1), test_mla_flydsl_gather_dispatch (2 failed + 4 errors), test_scheduler_metrics (7). test_accumulation.py differs only in which parameters fail, in both trees — it is flaky. test_scheduler_metrics passes 34/34 in isolation on both trees and fails only in full-suite context, on main as well. No regression is attributable to this branch; the passed-count difference is the tests this branch adds.

[github-actions]

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

• ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

• ✅ Run after the PR is approved and Pre Checkin passes
• ✅ Run immediately when an approval review is submitted
• ✅ Can be requested before approval with labels

Label	Tests
ci:full	Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom	Run native ATOM model accuracy tests
ci:vllm	Run ATOM vLLM OOT model accuracy tests
ci:sglang	Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2332 --add-label <label>

[ZhangLirong-amd]

The fix for the original V4 DPA8 + DSpark hang looks correct. _dp_draft_lockstep_active() now checks is_pipeline_parallel directly, so disabling deferred output no longer causes a producer to skip draft collectives while its DP peers enter them. Keeping deferred output
enabled for V4 also restores its previous behavior at that decision point.

However, disabling Scheduler.use_spec for all P/D producers introduces two additional cases that need attention:

1. EAGLE / serial MTP loses successor-token metadata during chunked prefill.
   Setting self.use_spec=False also makes self.drafter_needs_next_token=False, although the runner still retains and invokes the drafter. Consequently, batch.next_token_ids is absent. For a non-DP middle chunk, EagleProposer.compute_draft_kv() returns without writing draft
   KV; for a DP alignment pass, propose_draft_token_ids() cannot replace the dummy anchor with the actual successor token. Please separate “the producer does not perform speculative verification” from “the drafter still needs successor-token metadata.” This does not affect
   DSpark through this particular path, since its context-KV write does not use successor tokens.

2. DSpark dynamic q-bucketing can expand a producer’s target-only decode beyond its scheduled width.
   The scheduler now gives the producer a q=1 decode batch, but _dspark_apply_q_bucket() still derives full_q from self.drafter.mtp_k + 1. Executing the existing decision logic with mtp_k=3, ell=1, and buckets 1,2,4 selects q=2 for a batch scheduled with only one token per
   request. This can make the rebuilt token layout and attention metadata exceed what the scheduler reserved. Please skip this transformation for batches without speculative verification, or otherwise ensure it cannot increase any request’s scheduled width.

I checked the lockstep behavior and the two cases above with CPU-only source/control-flow checks; these are not end-to-end GPU reproductions.

The passing V4/DSpark accuracy jobs are useful, but the P/D benchmark and Mooncake RDMA smoke jobs are currently skipped. Before re-landing, I would also like to see the current PR head pass the original regression configuration: 1P1D, DPA8, DSpark, chunked prefill, and
concurrency 128, including warmup and sustained request progress. The original hang fix looks sound, but that distributed regression coverage is still needed for the full re-land