Skip to content

[Perf] [Intel] gemma4: Enable frozen-KV MTP graph capture - #44852

Open
alexnails wants to merge 3 commits into
alexnails/gemma4-text-onlyfrom
alexnails/gemma4-xpu-frozen-kv-mtp
Open

alexnails wants to merge 3 commits into
alexnails/gemma4-text-onlyfrom
alexnails/gemma4-xpu-frozen-kv-mtp

Conversation

@alexnails

@alexnails alexnails commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Stack (2/6), merge in order: #44851 → #44852 → #44853 → #44854 → #44863 → #44869

Motivation

Frozen-KV MTP on XPU fails under the synchronous scheduler and uses the draft's dummy KV pool when determining sliding-window translation. It also leaves recurrent assistant forwards eager despite the shared graph runner supporting XPU.

Modifications

  • Accept and forward pp_proxy_tensors through target prefill and verification.
  • Bind frozen draft attention to the target allocator's KV pool and skip write-slot metadata for this read-only draft.
  • Enable frozen-KV draft graph capture on XPU, with a serving regression and an E2B/BMG MTP recipe.

Depends on #44851. No CPU-offload changes or weight quantization.

Accuracy Tests

Five attention backend unit tests pass, including a pool-ownership regression that fails before the fix. The XPU MTP serving fixture checks repeated decode, per-request verification statistics, and nonzero draft capture. Six greedy prompts match ordinary decoding, including 1,016- and 1,814-token prompts crossing the sliding-window boundary.

GSM8K via sgl-eval==0.1.3: 82.5%/200, zero errors, two truncated responses. This is a smoke check, not proof of accuracy improvement or equivalence.

Speed Tests and Profiling

Same physical Intel Arc Pro B70, TP1 BF16, unchanged target weights, context 8,192, KV capacity 32,768, max running requests 8; MTP top-k 1, three draft steps/four verification tokens. OpenAI chat benchmark, fixed 256 output tokens, two repeats; 8 requests at concurrency 1 and 32 at concurrency 8.

Concurrency Ordinary TPOT Corrected eager MTP Captured MTP
1 11.168 ms 7.742 ms 4.282 ms
8 11.774 ms 9.372 ms 5.914 ms

Capture alone improves TPOT by 1.81x/1.58x over corrected eager MTP. The total 2.61x/1.99x over ordinary decoding also includes speculative decoding's benefit.

Checklist

  • Pre-commit checks and attention unit tests.
  • XPU serving regressions and greedy parity.
  • Accuracy and serving benchmarks.
  • Cookbook and generated-command checks.

CI States

Latest PR Test (Base): 🚫 Run #42143737121
Latest PR Test (Extra): 🚫 Run #42143737119
Latest PR Test (AMD ROCm 10): 🚫 Run #42143737107

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

…rozen-kv-mtp

# Conflicts:
#	docs/src/snippets/autoregressive/gemma4-deployment.jsx
#	python/sglang/srt/layers/attention/xpu_backend.py
#	python/sglang/srt/speculative/frozen_kv_mtp_worker_v2.py
…router

main detects sliding-window pools with layer_router and translates ids
through RunnerIds, so the pool-binding test mocks those instead of
SWAKVPool, and also checks that only the frozen draft takes the target
pool's window id space.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants