Repository navigation
[AMD] [GLM-5.3-Flash Day 0] Enable speculative decoding (MTP) on ROCm - #39778
Merged
Merged
Conversation
Speculative decoding on MI355X produced fluent but arithmetically wrong text and scored 2.00% on GSM8K, against 98% for the same model with MTP off and 97.50% for the same model and MTP on a B200. Draft acceptance sat at 1.33 of 6 tokens, near the 1.0 floor that means nothing is ever accepted. Both symptoms came from one gate. update_kpool_write_plan returns immediately when the platform is not CUDA, so on ROCm the k-pool write plan kept the zeros it was allocated with. Plain decode survives that because _get_kpool_decode_metadata recomputes pool_seqlens inline whenever pool_seqlens_per_q is None, and decode never allocates that buffer. The verify path does allocate it, so it took the all-zero plan instead of the fallback. With group_lengths all zero the fused top-k selects no pooled history, and the first draft position ends up attending to nothing at all. Instrumenting the indexer on a 20-token prompt showed pool_seqlens as [0,0,0,0,0,0] with zero valid indices in row 0, against [5,5,5,5,6,6] and 20 valid indices covering the whole sequence once the plan is populated. A B200 running the same prompt produces the second pattern, which is what identified the plan as the divergence. plan.req and plan.write_start were zero for the same reason, so every request also wrote its k-pool tail into request slot 0. The assert in _forward_cuda_target_verify is opened in the same commit because neither half is useful alone: opening it without a populated plan gives the 2.00% server above, and populating the plan while the assert still fires means the code never runs. GSM8K over 200 examples, MI355X TP4, zai-org/GLM-5.3-Flash: 2.00% -> 97.0%, accept length 1.33 -> 4.01. Without MTP the same build is unchanged at 99.5%.
1am9trash
pushed a commit
that referenced
this pull request
Sep 17, 2026
Squashed from #39778, which carries the same change against main and remains open there for review. Co-Authored-By: Jacob Chang <jacchang@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Jacob0226
marked this pull request as ready for review
September 18, 2026 07:05
Jacob0226
requested review from
1am9trash,
Fridge003,
HaiShaw,
YAMY1234,
hubertlu-tw,
kkHuang-amd and
rainj-me
as code owners
September 18, 2026 07:05
Collaborator
|
It's just a small gate removal. |
3 of 4 tasks
georgewhewell
added a commit
to hellas-ai/nix-strix-halo
that referenced
this pull request
Oct 9, 2026
0013 opened target_verify to HIP but update_kpool_write_plan still returned early off CUDA, so MTP verify ran on an all-zero plan: our NEXTN arm accepted only 1.5-1.95 of 4 drafts. Upstream sgl-project/sglang#39778 opens both gates together (MI355X TP4: GSM8K 2% -> 97%, 1.33 -> 4.01 of 6 drafts accepted). Ordinary decode keeps its own path (the verify-only plan fields stay unset).
This was referenced Nov 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#39341 brought the k-pool indexer's paged path to gfx950, but the speculative entry point still refuses ROCm:
_forward_cuda_target_verifyasserts CUDA-only, andupdate_kpool_write_planreturns before filling the write plan. Opening both tois_hip()is the whole change.Scope
The two gates above, on the GLM-5.3-Flash k-pool path. CUDA takes the branch it already took. ROCm without
--speculative-algorithmis unchanged, because ordinary decode keeps using its own fallback.Both gates must open together: fill the plan without removing the assert and the code never runs; remove the assert without filling the plan and
plan is not Nonestill passes, so target-verify runs on an all-zero plan instead of failing.Test plan
Base
rocm/sgl-dev:v0.5.19-rocm720-mi35x-20260914, sglangmainplus the eight open Day-0 PRs (#39338, #39339, #39340, #39341, #38545, #38546, #38547, #39317). This PR is the only variable. zai-org/GLM-5.3-Flash, MI355X TP4, EAGLE with 5 steps and 6 draft tokens, decode CUDA graphs on. GSM8K, 200 examples, temperature 1.0, one scoring pass each.B200 reference for the same model and MTP settings: 97.50%, 3.25–4.12 accepted per step. MTP off staying at 99.5% is the check that ordinary decoding still uses its fallback.
Dependency
#39341 brought the k-pool indexer's paged path to gfx950; this PR opens the speculative entry point to it.
CI States
Latest PR Test (Base): ⏳ Run #35777645861
Latest PR Test (Extra): ❌ Run #35777645410
Latest PR Test (AMD ROCm 10): ⏳ Run #35777645802