Skip to content

[AMD] [GLM-5.3-Flash Day 0] Enable speculative decoding (MTP) on ROCm - #39778

Merged
hnyls2002 merged 4 commits into
sgl-project:mainfrom
Jacob0226:jacob/glm53-mtp-rocm
Sep 22, 2026
Merged

hnyls2002 merged 4 commits into
sgl-project:mainfrom
Jacob0226:jacob/glm53-mtp-rocm

Conversation

@Jacob0226

@Jacob0226 Jacob0226 commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

#39341 brought the k-pool indexer's paged path to gfx950, but the speculative entry point still refuses ROCm: _forward_cuda_target_verify asserts CUDA-only, and update_kpool_write_plan returns before filling the write plan. Opening both to is_hip() is the whole change.

before after
history visible to the first speculated token 0 tokens the whole prompt
indexer scratch destination request slot 0, every request the request's own slot
GSM8K 2.00% 97.0%
draft tokens accepted per step, of 6 1.33 4.01

Scope

The two gates above, on the GLM-5.3-Flash k-pool path. CUDA takes the branch it already took. ROCm without --speculative-algorithm is unchanged, because ordinary decode keeps using its own fallback.

Both gates must open together: fill the plan without removing the assert and the code never runs; remove the assert without filling the plan and plan is not None still passes, so target-verify runs on an all-zero plan instead of failing.

Test plan

Base rocm/sgl-dev:v0.5.19-rocm720-mi35x-20260914, sglang main plus the eight open Day-0 PRs (#39338, #39339, #39340, #39341, #38545, #38546, #38547, #39317). This PR is the only variable. zai-org/GLM-5.3-Flash, MI355X TP4, EAGLE with 5 steps and 6 draft tokens, decode CUDA graphs on. GSM8K, 200 examples, temperature 1.0, one scoring pass each.

Build GSM8K accepted per step
baseline, MTP off 99.5% —
baseline, MTP on 2.00% 1.33
this PR, MTP on 97.0% 4.01

B200 reference for the same model and MTP settings: 97.50%, 3.25–4.12 accepted per step. MTP off staying at 99.5% is the check that ordinary decoding still uses its fallback.

Dependency

#39341 brought the k-pool indexer's paged path to gfx950; this PR opens the speculative entry point to it.


CI States

Latest PR Test (Base): ⏳ Run #35777645861
Latest PR Test (Extra): ❌ Run #35777645410
Latest PR Test (AMD ROCm 10): ⏳ Run #35777645802

Speculative decoding on MI355X produced fluent but arithmetically wrong text
and scored 2.00% on GSM8K, against 98% for the same model with MTP off and
97.50% for the same model and MTP on a B200. Draft acceptance sat at 1.33 of 6
tokens, near the 1.0 floor that means nothing is ever accepted.

Both symptoms came from one gate. update_kpool_write_plan returns immediately
when the platform is not CUDA, so on ROCm the k-pool write plan kept the zeros
it was allocated with. Plain decode survives that because
_get_kpool_decode_metadata recomputes pool_seqlens inline whenever
pool_seqlens_per_q is None, and decode never allocates that buffer. The verify
path does allocate it, so it took the all-zero plan instead of the fallback.
With group_lengths all zero the fused top-k selects no pooled history, and the
first draft position ends up attending to nothing at all.

Instrumenting the indexer on a 20-token prompt showed pool_seqlens as
[0,0,0,0,0,0] with zero valid indices in row 0, against [5,5,5,5,6,6] and 20
valid indices covering the whole sequence once the plan is populated. A B200
running the same prompt produces the second pattern, which is what identified
the plan as the divergence.

plan.req and plan.write_start were zero for the same reason, so every request
also wrote its k-pool tail into request slot 0.

The assert in _forward_cuda_target_verify is opened in the same commit because
neither half is useful alone: opening it without a populated plan gives the
2.00% server above, and populating the plan while the assert still fires means
the code never runs.

GSM8K over 200 examples, MI355X TP4, zai-org/GLM-5.3-Flash: 2.00% -> 97.0%,
accept length 1.33 -> 4.01. Without MTP the same build is unchanged at 99.5%.
@Jacob0226 Jacob0226 changed the title [AMD] Enable MTP for GLM-5.3-Flash on ROCm [AMD] [GLM-5.3-Flash Day 0] Enable speculative decoding (MTP) on ROCm Sep 16, 2026
1am9trash pushed a commit that referenced this pull request Sep 17, 2026
Squashed from #39778, which carries the same change
against main and remains open there for review.

Co-Authored-By: Jacob Chang <jacchang@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@Jacob0226
Jacob0226 marked this pull request as ready for review September 18, 2026 07:05
@1am9trash 1am9trash added the amd label Sep 18, 2026

@1am9trash 1am9trash left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Only remove some gate on amd code path.
No any logic change.

Only blocker: #39341

@1am9trash 1am9trash added the run-ci CI: run the baseline test suite on this PR label Sep 18, 2026
@1am9trash

Copy link
Copy Markdown
Collaborator

It's just a small gate removal.
I think we don't need to run the whole CI for it.

@hnyls2002
hnyls2002 merged commit 077c319 into sgl-project:main Sep 22, 2026
70 of 103 checks passed
georgewhewell added a commit to hellas-ai/nix-strix-halo that referenced this pull request Oct 9, 2026
0013 opened target_verify to HIP but update_kpool_write_plan still returned
early off CUDA, so MTP verify ran on an all-zero plan: our NEXTN arm accepted
only 1.5-1.95 of 4 drafts. Upstream sgl-project/sglang#39778 opens both gates
together (MI355X TP4: GSM8K 2% -> 97%, 1.33 -> 4.01 of 6 drafts accepted).
Ordinary decode keeps its own path (the verify-only plan fields stay unset).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

amd run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants