Skip to content

[Perf] [Intel] kv-canary: Add an on-device XPU fallback for plan/verify/write - #48358

Closed
dayanandav wants to merge 1 commit into
sgl-project:mainfrom
dayanandav:xpu-kv-canary-tensor-fallback
Closed

dayanandav wants to merge 1 commit into
sgl-project:mainfrom
dayanandav:xpu-kv-canary-tensor-fallback

Conversation

@dayanandav

@dayanandav dayanandav commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

On XPU the kv-canary plan/verify/write ops fall back to the Python references (*_ref.py), which copy req_to_token to host and loop per token. Decode drops to 2-3 tok/s with the canary on, and all 11 kv_canary e2e tests time out on XPU.

Modifications

  • kernels/ops/kv_canary/_tensor_fallback.py (new, XPU only): vectorized on-device plan entries, verify and write, with no host sync. Plan reuses the existing Triton offsets kernel.
  • plan/api.py, verify.py, write.py: dispatch to it when device.type == "xpu". CUDA keeps the JIT kernels; other devices keep the references.
  • test/kv_canary/e2e_base.py: use --attention-backend intel_xpu on XPU (the default Triton backend is too slow for the ~7k-token workload).
  • test_tensor_fallback.py (new, XPU CI): 120 differential tests, byte-equal against the Python references (plan, verify, write; MHA/SWA, all real-KV hash modes, duplicate slots, ring overflow).

Accuracy Tests

Byte-equal to the references: 120/120 pass. kv_canary e2e on Arc Pro B60, 11/11 pass (baseline MHA/SWA, perturb_real_kv_used x3, perturb_real_kv_unused_cache x3, perturb_req_to_token x2, pr_25015). Perturbation tests still report the expected violations.

Speed Tests and Profiling

Qwen3-0.6B, 8 requests, canary on, Arc Pro B60:

Decode
Python reference 2-3 tok/s
This PR ~220 tok/s

With only the attention-backend change the tests still time out (2.96 tok/s); the reference ops were ~99% of each decode step.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #41800276397
Latest PR Test (Extra): ❌ Run #41800276334
Latest PR Test (AMD ROCm 10): ❌ Run #41800276398

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

…fy/write

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@dayanandav

Copy link
Copy Markdown
Contributor Author

Superseded by #48372, #48373 and #48374 (regrouped into 3 PRs).

@dayanandav dayanandav closed this Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant