Repository navigation
[ROCm][Perf][Qwen3.8] Keep dense mono K2 on during MTP verify. - #64720
Open
abrahamzewoudie wants to merge 2 commits into
Open
abrahamzewoudie wants to merge 2 commits into
abrahamzewoudie wants to merge 2 commits into
Conversation
An MTP verify step is wider than a plain decode, so the mono graph turned off. Run K2 inside the row cap and load Quark MTP weights as BF16. Co-authored-by: Cursor <cursoragent@cursor.com>
abrahamzewoudie
requested review from
AndreasKaratzas,
DarkLight1337,
Harry-Chen,
dllehr-amd,
hongxiayang,
khluu,
tjtanaa,
vadiklyutiy and
ywang96
as code owners
October 30, 2026 10:18
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #61573. This branch is that mono-kernel branch plus one commit. Review the latest commit for the MTP change. The other files are #61573 and are not new here.
An MTP verify step is wider than a plain decode, so the mono graph turned off. K1 stays on vLLM's GDN core, which can roll a rejected draft back. K2 stays on the fused MXFP4 feed-forward while the step fits the K2 cap. Quark MXFP4 checkpoints store
mtp.*as BF16 underquantization_config.exclude, so those draft weights load unquantized.Serve with
speculative-config: {"method": "mtp", "num_speculative_tokens": 3}andVLLM_ROCM_MONO_DECODE=1. k=3 is the best combined geomean at TP1 and TP2. VL is highest at k=2.MI355X, 20-config sweep per TP, versus the best Argos mono kernel with no speculative decoding. The k=1 column is from the earlier k=1 summary.
TP1. Mono kernel + MTP vs mono kernel
TP2. Mono kernel + MTP vs mono kernel
Made with Cursor