Repository navigation
[Perf] [Intel] kv-canary: Add an on-device XPU fallback for plan/verify/write - #48358
Closed
dayanandav wants to merge 1 commit into
Closed
dayanandav wants to merge 1 commit into
dayanandav wants to merge 1 commit into
Conversation
…fy/write Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dayanandav
requested review from
BBuf,
DarkSharpness,
HaiShaw,
HydraQYH,
celve and
yuan-luo
as code owners
November 4, 2026 08:32
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On XPU the kv-canary plan/verify/write ops fall back to the Python references (
*_ref.py), which copyreq_to_tokento host and loop per token. Decode drops to 2-3 tok/s with the canary on, and all 11 kv_canary e2e tests time out on XPU.Modifications
kernels/ops/kv_canary/_tensor_fallback.py(new, XPU only): vectorized on-device plan entries, verify and write, with no host sync. Plan reuses the existing Triton offsets kernel.plan/api.py,verify.py,write.py: dispatch to it whendevice.type == "xpu". CUDA keeps the JIT kernels; other devices keep the references.test/kv_canary/e2e_base.py: use--attention-backend intel_xpuon XPU (the default Triton backend is too slow for the ~7k-token workload).test_tensor_fallback.py(new, XPU CI): 120 differential tests, byte-equal against the Python references (plan, verify, write; MHA/SWA, all real-KV hash modes, duplicate slots, ring overflow).Accuracy Tests
Byte-equal to the references: 120/120 pass. kv_canary e2e on Arc Pro B60, 11/11 pass (baseline MHA/SWA, perturb_real_kv_used x3, perturb_real_kv_unused_cache x3, perturb_req_to_token x2, pr_25015). Perturbation tests still report the expected violations.
Speed Tests and Profiling
Qwen3-0.6B, 8 requests, canary on, Arc Pro B60:
With only the attention-backend change the tests still time out (2.96 tok/s); the reference ops were ~99% of each decode step.
🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ❌ Run #41800276397
Latest PR Test (Extra): ❌ Run #41800276334
Latest PR Test (AMD ROCm 10): ❌ Run #41800276398
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: