Repository navigation
Conversation
alexnails
requested review from
Fridge003,
HaiShaw,
Qiaolin-Yu,
hebiao064,
hnyls2002,
ispobock,
kpham-sgl,
merrymercy,
pyc96,
wisclmy0611 and
zijiexia
as code owners
October 17, 2026 05:09
This was referenced Oct 17, 2026
This was referenced Oct 17, 2026
5 tasks done
…rozen-kv-mtp # Conflicts: # docs/src/snippets/autoregressive/gemma4-deployment.jsx # python/sglang/srt/layers/attention/xpu_backend.py # python/sglang/srt/speculative/frozen_kv_mtp_worker_v2.py
…router main detects sliding-window pools with layer_router and translates ids through RunnerIds, so the pool-binding test mocks those instead of SWAKVPool, and also checks that only the frozen draft takes the target pool's window id space.
This was referenced Nov 9, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (2/6), merge in order: #44851 → #44852 → #44853 → #44854 → #44863 → #44869
Motivation
Frozen-KV MTP on XPU fails under the synchronous scheduler and uses the draft's dummy KV pool when determining sliding-window translation. It also leaves recurrent assistant forwards eager despite the shared graph runner supporting XPU.
Modifications
pp_proxy_tensorsthrough target prefill and verification.Depends on #44851. No CPU-offload changes or weight quantization.
Accuracy Tests
Five attention backend unit tests pass, including a pool-ownership regression that fails before the fix. The XPU MTP serving fixture checks repeated decode, per-request verification statistics, and nonzero draft capture. Six greedy prompts match ordinary decoding, including 1,016- and 1,814-token prompts crossing the sliding-window boundary.
GSM8K via
sgl-eval==0.1.3: 82.5%/200, zero errors, two truncated responses. This is a smoke check, not proof of accuracy improvement or equivalence.Speed Tests and Profiling
Same physical Intel Arc Pro B70, TP1 BF16, unchanged target weights, context 8,192, KV capacity 32,768, max running requests 8; MTP top-k 1, three draft steps/four verification tokens. OpenAI chat benchmark, fixed 256 output tokens, two repeats; 8 requests at concurrency 1 and 32 at concurrency 8.
Capture alone improves TPOT by 1.81x/1.58x over corrected eager MTP. The total 2.61x/1.99x over ordinary decoding also includes speculative decoding's benefit.
Checklist
CI States
Latest PR Test (Base): 🚫 Run #42143737121
Latest PR Test (Extra): 🚫 Run #42143737119
Latest PR Test (AMD ROCm 10): 🚫 Run #42143737107
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: