Repository navigation
Conversation
…a4-xpu-intermediate-prefill-gemm
…a4-xpu-intermediate-prefill-gemm
…a4-xpu-intermediate-prefill-gemm
This was referenced Nov 9, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Gemma4 E2B's BF16 XPU prefill GEMM fast path is restricted to exactly 4096 rows, leaving useful intermediate shapes on the ordinary projection/activation path. This admits 256, 1024, 2048 and 4095 rows when the loaded native library explicitly reports support.
Stacked on #44925, with the existing 4096-row path preserved. Companion native PR: sgl-project/sgl-kernel-xpu#561 (depends on sgl-kernel-xpu#559).
Modifications
Query native capability independently for
(256, 1024, 2048, 4095, 4096)at each projection width. Dispatch the fused GEMM only for reported row counts. Missing/false queries and older 4096-only wheels preserve the original 4096-row standalone-lookup/native paths. Unadmitted shapes use the parent model path. Existing hardware, dtype, TP, weight, backend, hook, delegated-method and gradient guards remain; add a 32-bit addressing bound.Activation and PLE retain their existing 4096-row dispatch. Extend the existing integration fixture with admitted shapes, compile/replay, ordinary-path fallback checks, and explicit legacy-wheel capability coverage.
Accuracy Tests
Speed Tests and Profiling
Intel Arc Pro B70, TP1, unchanged BF16 target/assistant snapshots, same full decode graphs and NEXTN settings, prefix cache disabled in both arms. Compare parent
a3c379de38againsta2dfdf0eb0, using the same 89 native binaries in both arms; the parent only dispatches this GEMM at 4096 rows.24 samples per row count. Unchanged 26/61/128/512-row paths have mean changes between -1.13% and +1.05%; no short-prompt gain is claimed.
Decode blocks use 128/640/1536 requests, each generating 128 tokens, with about 71/71/86 seconds per arm. TPOT is recomputed as
(mean E2E - mean TTFT)/127; output throughput is accepted output tokens divided by measured wall duration. Treat decode as roughly flat. One A/B, no repeated-control drift estimate for this full comparison or significance claim. Both servers retired; source/native hashes rechecked before and after each arm.Evidence archive SHA256:
539e5d8081610493ff860981ec80dce50214116dd4eea522706fecba7667c5eb.Checklist
Draft while parent/native integration and numerical variability receive review.
CI States
Latest PR Test (Base): ❌ Run #42150942862
Latest PR Test (Extra): ❌ Run #42150942995
Latest PR Test (AMD ROCm 10): ❌ Run #42150942860
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: