Repository navigation
[Fix] [AMD] dsa: One decode graph variant on every attention-DP rank, one verify graph - #48302
llying-001 wants to merge 9 commits into
Conversation
…verify graph On HIP, DSA models capture a dense (k-only indexer) and a sparse decode graph per batch size, and each rank picked its variant from its own longest sequence. Under DP attention the ranks replay in lockstep and the collectives captured inside the graphs pair up across ranks; some register their graph's input buffers, so a rank replaying the other variant reads its peer's stale buffers (garbled output on GLM-5.2 with in-graph aiter all-gather / reduce-scatter, MI355X PD decode TP4 + DP4 attention). - Target-verify captures hold one DSA graph: the dense variant only skips the decode/idle indexer logits, so both verify graphs ran the same kernels (and select() synced seq_lens to the host on every verify replay). - The MLP sync row gains an upper bound of the rank's longest decode sequence (host lengths only); the maximum over ranks rides ScheduleBatch / ForwardBatch as dp_decode_graph_max_seq_len and the DSA, sequence-length bucket and DSv4.1 candidate variants select from it in decode/idle. No new collective, no device read. Without the sync (no DP attention) the per-rank rule is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
…-off score 52,830, upstream PR sgl-project/sglang#48302
This comment has been minimized.
This comment has been minimized.
|
/tag-and-rerun-ci |
Read the gathered decode bound before the forward mode, so graph-variant selection without the MLP sync stays exactly as on main and does not need a forward mode. Size the bound's step width from the speculative algorithm, so plain decode never reads the spec config. Give the hand-built MLP sync results in two unit tests the new field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt-dp-consistency
|
/rerun-failed-ci |
|
The only failing test in this PR's
Its failure makes the fail-fast gate skip the remaining base-b shard(s) of this run ( |
|
/rerun-failed-ci |
…nt-dp-consistency
|
Correction to my comment above: this failure is not intermittent. It is a deterministic regression on |
…nt-dp-consistency # Conflicts: # python/sglang/srt/managers/scheduler_components/dp_attn.py
…nt-dp-consistency # Conflicts: # python/sglang/srt/managers/scheduler_components/dp_attn.py
|
@HaiShaw @hnyls2002 could you review this when you have time? cc @ch-wan for the DP-attention side. It fixes garbled output on HIP DSA models (GLM-5.x, DeepSeek-V3.2) under DP attention: each rank picked its dense/sparse decode graph from its own longest sequence, so ranks could replay different graphs in the same step while the collectives captured in them pair up across ranks. The fix gathers an upper bound of each rank's longest decode sequence in the existing MLP-sync row and selects the variant from the max, so every rank replays the same graph; without DP attention the per-rank rule is unchanged. Details and accuracy/speed numbers are in the description (0 garbled in 1,800 prompts vs 20-53 per 200-400 in the controls; +1.3% throughput, +5.5% with the MoE gather/combine in-graph that this makes correct). Status:
Codeowner approval is needed for |
… source text The bound's trip from the ScheduleBatch to the runner was asserted by regexes over inspect.getsource. Call the code instead: ScheduleBatch.copy(), ForwardBatch.init_new() on a decode batch, and TBO filter_batch for both children.
|
/rerun-failed-ci |
|
@HaiShaw @hnyls2002 could you take a look when you have time? (codeowners of
|
|
@RolaoDenthu could you review this one when you have time? It keeps DSA decode graph variants consistent across attention-DP ranks (the garbled-output fix for GLM-5.2 on MI355X). |
| # Decode only: a DSpark target verify stages its metadata for the variant of | ||
| # its own committed lengths (select_for_max_seq_len) before the replay. | ||
| dp_max_seq_len = dp_decode_max_seq_len(forward_batch) | ||
| if dp_max_seq_len is not None: | ||
| return self.select_for_max_seq_len(dp_max_seq_len) |
There was a problem hiding this comment.
This hunk changes NVIDIA SM100 decode selection under DP attention, which the [AMD] title and the test plan don't cover. It also half-applies in a DSpark verify runner: IDLE ranks now take the bound while TARGET_VERIFY ranks keep local lengths, and the early-staging reason doesn't hold there, since _wants_early_verify_metadata() requires not attn_dp_enabled. Suggest dropping this hunk and doing DSV4.1 (decode + verify, bound + verify_extra_tokens) in a follow-up with a B200 run from the owners; or keep it, retitle without [AMD], and add that run here.
Motivation
On HIP, DSA models (GLM-5.x, DeepSeek-V3.2) capture two decode graphs per batch size: dense (the indexer only writes K when every sequence fits in
index_topk) and sparse (DsaGraphVariants). Each rank picks its variant from its own longest sequence, and an idle rank always picks sparse.Under DP attention the ranks replay their graphs in lockstep, and the collectives captured inside them pair up across ranks. Some of these collectives register their graph's input buffers (aiter all-gather / reduce-scatter / custom all-reduce). When two ranks replay different variants in the same step, a rank reads its peer's buffers from the other graph: stale data. On GLM-5.2-MXFP4, MI355X, PD decode TP4 + DP4 attention with the MoE gather / combine inside the graphs, 26 of 200 GSM8K prompts came back garbled. Pinning every rank to one variant removed all of them, which confirmed the cause. The low-rate version (first token right, then noise:
Theutzer1-case55500The $500 + 800 = 1300 ...) also shows up with only the custom all-reduce in the graph.Two further observations from the code:
_should_skip_logits_computation()); a target-verify capture runs the full indexer in both, so MTP deployments capture the verify graphs twice for nothing.select()has no host lengths and syncsseq_lens.max()to the host before every replay.Modifications
graph_variants.py:create_attention_graph_variants()takes the capture forward mode and builds the dense / sparse pair only for decode captures (decode_cuda_graph_runner.pypasses it). Target-verify captures hold one graph.dp_attn.MLPSyncBatchInfo) gains a column: an upper bound of the rank's longest decode sequence, read on the host only (the exactseq_lens_cpumax when the mirror exists, else committed lengths plus two steps' width). The maximum over ranks ridesScheduleBatch/ForwardBatchasdp_decode_graph_max_seq_len, following the existingdp_prefill_cuda_graph_max_prefix_len.dp_decode_max_seq_len()returns that bound in decode / idle, andDsaGraphVariants,SeqLenBucketGraphVariantsandDsv41CandidateGraphVariantsselect from it.test/registered/unit/layers/test_graph_variants_dp_consistency.py(13 tests, CPU). The bound's path to the runner is checked by callingScheduleBatch.copy(),ForwardBatch.init_new()and the TBOfilter_batch().Scope and behavior changes:
index_topkand skips a.item().Dsv41CandidateGraphVariantson SM100+ also selects decode variants from the global bound. Since the bound is at least every rank's maximum, a rank only moves to a candidate with a larger limit, which is the more general one; correctness is unchanged, and a short-context rank may use a more general (possibly slower) candidate. I could not run this on NVIDIA hardware, so a check from the DSV4.1 owners would help.Why sync a bound rather than always replay the general graph under DP attention: the +1.3 % below comes mostly from dropping the
.item()before verify, not from the dense variant. ForSeqLenBucketGraphVariants(aiter's 16 / 128 / max-partition buckets), always taking the max-partition fallback costs more on short contexts. The bound is one more int64 per rank in the existing all-gather. The trade-off is that a short-context rank follows the longest rank's variant.Accuracy Tests
The same change on a v0.5.22-based build (GLM-5.2-MXFP4, MI355X), PD 1 prefill (TP1 x PP4) + 1 decode (TP4 / DP4 attention) across two nodes over RDMA, real MTP acceptance. Controls first, then the fix. The prefill device pool was capped so that evictions run continuously, the condition in which the low-rate failure appeared.
On current
main(this branch merged with main at983455ae, head4dae7a44): the stocklmsysorg/sglang-rocm:v0.5.23-rocm720-mi35x-20261109image running the branch's Python, GLM-5.2-MXFP4 on one MI355X node, aggregated TP4 / DP4 attention (the PD decode shape). Two servers side by side, each under 400 GSM8K prompts (64 in flight) and 64 long-context prompts of about 14k tokens (32 in flight) at once, so the ranks hold sequences on both sides ofindex_topk:The same audit build of the runner on the no-MTP server, in two runs:
index_topkEvery replay had a bound, idle ranks included. This checks the mechanism and output on
main; I did not rerun the failure reproduction (the PD controls above) onmain.Speed Tests and Profiling
Same two nodes as the v0.5.22-based accuracy runs, AgentX trace replay, concurrency 104, 1 hour, simulated acceptance 2.99:
.item()is gone).Checklist
ruff format,ruff check,isortpass on the changed files).CI States
Latest PR Test (Base): ✅ Run #43080525038
Latest PR Test (Extra): ❌ Run #43080524836
Latest PR Test (AMD ROCm 10): ❌ Run #43080525019
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: