Skip to content

[Fix] [AMD] dsa: One decode graph variant on every attention-DP rank, one verify graph - #48302

Open
llying-001 wants to merge 9 commits into
sgl-project:mainfrom
llying-001:fix/dsa-graph-variant-dp-consistency
Open

llying-001 wants to merge 9 commits into
sgl-project:mainfrom
llying-001:fix/dsa-graph-variant-dp-consistency

Conversation

@llying-001

@llying-001 llying-001 commented Nov 4, 2026 •

Copy link
Copy Markdown

Motivation

On HIP, DSA models (GLM-5.x, DeepSeek-V3.2) capture two decode graphs per batch size: dense (the indexer only writes K when every sequence fits in index_topk) and sparse (DsaGraphVariants). Each rank picks its variant from its own longest sequence, and an idle rank always picks sparse.

Under DP attention the ranks replay their graphs in lockstep, and the collectives captured inside them pair up across ranks. Some of these collectives register their graph's input buffers (aiter all-gather / reduce-scatter / custom all-reduce). When two ranks replay different variants in the same step, a rank reads its peer's buffers from the other graph: stale data. On GLM-5.2-MXFP4, MI355X, PD decode TP4 + DP4 attention with the MoE gather / combine inside the graphs, 26 of 200 GSM8K prompts came back garbled. Pinning every rank to one variant removed all of them, which confirmed the cause. The low-rate version (first token right, then noise: Theutzer1-case55500The $500 + 800 = 1300 ...) also shows up with only the custom all-reduce in the graph.

Two further observations from the code:

  • The dense variant only changes the decode / idle indexer (_should_skip_logits_computation()); a target-verify capture runs the full indexer in both, so MTP deployments capture the verify graphs twice for nothing.
  • On the verify path select() has no host lengths and syncs seq_lens.max() to the host before every replay.

Modifications

  • graph_variants.py: create_attention_graph_variants() takes the capture forward mode and builds the dense / sparse pair only for decode captures (decode_cuda_graph_runner.py passes it). Target-verify captures hold one graph.
  • The MLP sync row (dp_attn.MLPSyncBatchInfo) gains a column: an upper bound of the rank's longest decode sequence, read on the host only (the exact seq_lens_cpu max when the mirror exists, else committed lengths plus two steps' width). The maximum over ranks rides ScheduleBatch / ForwardBatch as dp_decode_graph_max_seq_len, following the existing dp_prefill_cuda_graph_max_prefix_len.
  • dp_decode_max_seq_len() returns that bound in decode / idle, and DsaGraphVariants, SeqLenBucketGraphVariants and Dsv41CandidateGraphVariants select from it.
  • No new collective, no device read.
  • Unit tests: test/registered/unit/layers/test_graph_variants_dp_consistency.py (13 tests, CPU). The bound's path to the runner is checked by calling ScheduleBatch.copy(), ForwardBatch.init_new() and the TBO filter_batch().

Scope and behavior changes:

  • Covered: decode and idle steps. A DSV4.1 DSpark target verify keeps its own lengths, since its metadata is staged for that variant before the replay, and the NVIDIA megakernel's verify / draft variants are untouched. Neither applies to GLM-5.2 on MI355X; extending the rule to them would be a follow-up.
  • Without the MLP sync the per-rank rule is unchanged. With the sync but a single DP rank (a gathered-buffer MoE backend), the bound is that rank's own maximum: exactly the old value when the host mirror exists, otherwise committed lengths plus two steps, which is slightly more conservative near index_topk and skips a .item().
  • NVIDIA: under DP attention, Dsv41CandidateGraphVariants on SM100+ also selects decode variants from the global bound. Since the bound is at least every rank's maximum, a rank only moves to a candidate with a larger limit, which is the more general one; correctness is unchanged, and a short-context rank may use a more general (possibly slower) candidate. I could not run this on NVIDIA hardware, so a check from the DSV4.1 owners would help.

Why sync a bound rather than always replay the general graph under DP attention: the +1.3 % below comes mostly from dropping the .item() before verify, not from the dense variant. For SeqLenBucketGraphVariants (aiter's 16 / 128 / max-partition buckets), always taking the max-partition fallback costs more on short contexts. The bound is one more int64 per rank in the existing all-gather. The trade-off is that a short-context rank follows the longest rank's variant.

Accuracy Tests

The same change on a v0.5.22-based build (GLM-5.2-MXFP4, MI355X), PD 1 prefill (TP1 x PP4) + 1 decode (TP4 / DP4 attention) across two nodes over RDMA, real MTP acceptance. Controls first, then the fix. The prefill device pool was capped so that evictions run continuously, the condition in which the low-rate failure appeared.

Arm Prompts Bad start Garbled
Control, image as is (2 runs) 400 6 20
Fix (3 runs) 600 0 0
Control, MoE gather / combine in-graph 200 13 53
Fix, MoE gather / combine in-graph (2 runs) 400 0 0
Control, no MTP (decode-mode graphs only) 200 3 33
Fix, no MTP (2 runs) 800 0 0
  • An audit build of the runner (one device read per replay, mechanism check only) over 21,504 replays: the ranks picked the same variant on every replay (the per-rank rule would have disagreed on 1,644), the bound was never below the true maximum, and dense was still used (3.7 %).
  • Uncapped production config, 1,319 GSM8K prompts: 0.966 / 0.969 (the same as stock SGLang with the official FP8 checkpoint on the same prompts, 0.967), 0 garbled, 0 bad start. lm-eval GSM8K strict / flexible 0.971 / 0.973 and 0.967 / 0.970. Long context (16k, 100k needle and long-prefix GSM8K) at the control's level.

On current main (this branch merged with main at 983455ae, head 4dae7a44): the stock lmsysorg/sglang-rocm:v0.5.23-rocm720-mi35x-20261109 image running the branch's Python, GLM-5.2-MXFP4 on one MI355X node, aggregated TP4 / DP4 attention (the PD decode shape). Two servers side by side, each under 400 GSM8K prompts (64 in flight) and 64 long-context prompts of about 14k tokens (32 in flight) at once, so the ranks hold sequences on both sides of index_topk:

Server GSM8K Bad start / garbled Needle / long-prefix GSM8K
No MTP (DSA dense / sparse decode graphs) 392/400 (0.980) 0 / 0 32/32, 31/32
EAGLE 3/1/4 (one target-verify graph; no DSA dual-graph at capture) 393/400 (0.983) 0 / 0 32/32, 31/32

The same audit build of the runner on the no-MTP server, in two runs:

Load Replays aligned across 4 ranks Rank disagreements (per-rank rule would have) Dense Bound below device max Dense with a row over index_topk GSM8K
GSM8K + long context (above) 17,920 0 (468, 2.6 %) 7 0 0 see above
GSM8K only, 600 prompts 19,712 0 (448, 2.3 %) 686 (3.5 %), the same replays on every rank 0 0 587/600 (0.978), 0 bad start, 0 garbled

Every replay had a bound, idle ranks included. This checks the mechanism and output on main; I did not rerun the failure reproduction (the PD controls above) on main.

Speed Tests and Profiling

Same two nodes as the v0.5.22-based accuracy runs, AgentX trace replay, concurrency 104, 1 hour, simulated acceptance 2.99:

  • Unchanged otherwise: 50,797 -> 51,477 tok/s/GPU (+1.3 %, single runs, about +-1 % run to run). The GPU idle before the verify graph drops from 0.59 to 0.03 ms per step (the .item() is gone).
  • With the MoE gather / combine inside the graphs, which this makes correct: 53,609 tok/s/GPU (+5.5 %).

Checklist

  • Format your code (ruff format, ruff check, isort pass on the changed files).
  • Add unit tests.
  • Update documentation (no user-facing option).
  • Provide accuracy and speed benchmark results.

CI States

Latest PR Test (Base): ✅ Run #43080525038
Latest PR Test (Extra): ❌ Run #43080524836
Latest PR Test (AMD ROCm 10): ❌ Run #43080525019

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

…verify graph

On HIP, DSA models capture a dense (k-only indexer) and a sparse decode graph per
batch size, and each rank picked its variant from its own longest sequence. Under DP
attention the ranks replay in lockstep and the collectives captured inside the graphs
pair up across ranks; some register their graph's input buffers, so a rank replaying
the other variant reads its peer's stale buffers (garbled output on GLM-5.2 with
in-graph aiter all-gather / reduce-scatter, MI355X PD decode TP4 + DP4 attention).

- Target-verify captures hold one DSA graph: the dense variant only skips the
  decode/idle indexer logits, so both verify graphs ran the same kernels (and select()
  synced seq_lens to the host on every verify replay).
- The MLP sync row gains an upper bound of the rank's longest decode sequence (host
  lengths only); the maximum over ranks rides ScheduleBatch / ForwardBatch as
  dp_decode_graph_max_seq_len and the DSA, sequence-length bucket and DSv4.1
  candidate variants select from it in decode/idle. No new collective, no device read.
  Without the sync (no DP attention) the per-rank rule is unchanged.

Co-authored-by: Cursor <cursoragent@cursor.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

llying-001 added a commit to AMD-AGI/Infera that referenced this pull request Nov 4, 2026
@github-actions

This comment has been minimized.

@llying-001 llying-001 changed the title [Fix] [AMD] DSA decode graphs: one variant on every attention-DP rank, one verify graph [Fix] [AMD] dsa: One decode graph variant on every attention-DP rank, one verify graph Nov 4, 2026
@bingxche

bingxche commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Nov 4, 2026
@llying-001 llying-001 changed the title [Fix] [AMD] dsa: One decode graph variant on every attention-DP rank, one verify graph [Fix] dsa: keep decode CUDA graph variants consistent across DP ranks Nov 5, 2026
@llying-001 llying-001 changed the title [Fix] dsa: keep decode CUDA graph variants consistent across DP ranks [Fix] [AMD] dsa: One decode graph variant on every attention-DP rank, one verify graph Nov 5, 2026
llying-001 and others added 2 commits November 5, 2026 05:23
Read the gathered decode bound before the forward mode, so graph-variant
selection without the MLP sync stays exactly as on main and does not need a
forward mode. Size the bound's step width from the speculative algorithm, so
plain decode never reads the spec config. Give the hand-built MLP sync
results in two unit tests the new field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@llying-001

Copy link
Copy Markdown
Author

/rerun-failed-ci

@llying-001

Copy link
Copy Markdown
Author

The only failing test in this PR's PR Test Base run is test/registered/attention/unittests/mla/test_flashinfer.py, case test_runner_mode_eagle_draft_cuda_graph_runner_cases (runner_eagle_draft_decode_mla_cuda_graph_chain): AssertionError: Tensor-likes are not close! (job). It is not caused by this PR:

Its failure makes the fail-fast gate skip the remaining base-b shard(s) of this run (base-b-test-2-gpu-large (2)); every other job passed. Could a maintainer review with this in mind, or rerun once the test is stable on main? Thanks!

@llying-001

Copy link
Copy Markdown
Author

/rerun-failed-ci

@llying-001

Copy link
Copy Markdown
Author

Correction to my comment above: this failure is not intermittent. It is a deterministic regression on main from #48005, tracked in #48677 and fixed by #48734 (a08a6158e0, merged 2026-11-05 23:03 UTC); the 23:06 scheduled run passes because it includes that fix. This branch was last merged with main just before #48734 landed, so I have merged main again to pick up the fix.

…nt-dp-consistency

# Conflicts:
#	python/sglang/srt/managers/scheduler_components/dp_attn.py
…nt-dp-consistency

# Conflicts:
#	python/sglang/srt/managers/scheduler_components/dp_attn.py
@llying-001

Copy link
Copy Markdown
Author

@HaiShaw @hnyls2002 could you review this when you have time? cc @ch-wan for the DP-attention side.

It fixes garbled output on HIP DSA models (GLM-5.x, DeepSeek-V3.2) under DP attention: each rank picked its dense/sparse decode graph from its own longest sequence, so ranks could replay different graphs in the same step while the collectives captured in them pair up across ranks. The fix gathers an upper bound of each rank's longest decode sequence in the existing MLP-sync row and selects the variant from the max, so every rank replays the same graph; without DP attention the per-rank rule is unchanged. Details and accuracy/speed numbers are in the description (0 garbled in 1,800 prompts vs 20-53 per 200-400 in the controls; +1.3% throughput, +5.5% with the MoE gather/combine in-graph that this makes correct).

Status:

Codeowner approval is needed for layers/attention (graph_variants.py) and managers / model_executor. Happy to adjust anything. Thanks!

… source text

The bound's trip from the ScheduleBatch to the runner was asserted by regexes
over inspect.getsource. Call the code instead: ScheduleBatch.copy(),
ForwardBatch.init_new() on a decode batch, and TBO filter_batch for both
children.
@llying-001

Copy link
Copy Markdown
Author

/rerun-failed-ci

@llying-001

Copy link
Copy Markdown
Author

@HaiShaw @hnyls2002 could you take a look when you have time? (codeowners of layers/attention and of managers / model_executor.) Status on 4dae7a4:

  • Merged current main without conflicts. PR Test (Base) is green. On PR Test (AMD), stage-b-test-1-gpu-small-amd-mi35x fails in test_kimi_k3_kda_decode.py::test_pre_rename_names_still_select, which fails the same way on main (983455a); the other failures are the opt-in pr-gate checks.
  • Re-validated on main with GLM-5.2-MXFP4 (TP4 / DP4 attention, MI355X): over 37,632 audited replays the ranks always replayed the same decode graph (the per-rank rule would have disagreed on 916), the bound never fell below the device maximum, and GSM8K had no garbled output. Details are in the description.
  • The source-text test is replaced by behavior tests of the bound's path: ScheduleBatch.copy(), ForwardBatch.init_new() and the TBO filter_batch().

@llying-001

Copy link
Copy Markdown
Author

@RolaoDenthu could you review this one when you have time? It keeps DSA decode graph variants consistent across attention-DP ranks (the garbled-output fix for GLM-5.2 on MI355X). PR Test (Base) is green, and the description has the re-validation on current main.

Comment on lines +147 to +151
# Decode only: a DSpark target verify stages its metadata for the variant of
# its own committed lengths (select_for_max_seq_len) before the replay.
dp_max_seq_len = dp_decode_max_seq_len(forward_batch)
if dp_max_seq_len is not None:
return self.select_for_max_seq_len(dp_max_seq_len)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hunk changes NVIDIA SM100 decode selection under DP attention, which the [AMD] title and the test plan don't cover. It also half-applies in a DSpark verify runner: IDLE ranks now take the bound while TARGET_VERIFY ranks keep local lengths, and the early-staging reason doesn't hold there, since _wants_early_verify_metadata() requires not attn_dp_enabled. Suggest dropping this hunk and doing DSV4.1 (decode + verify, bound + verify_extra_tokens) in a follow-up with a B200 run from the owners; or keep it, retitle without [AMD], and add that run here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants