Repository navigation
Conversation
5 tasks done
…4-xpu-prefill-gemm-narrow
…4-xpu-prefill-gemm-narrow
…4-xpu-prefill-gemm-narrow
This was referenced Nov 9, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enable Gemma4's existing native BF16 prefill GEMM for its 15 narrow MLP layers when the installed binary confirms support. On the measured B70 workload this lowers uncached long-prefill latency by 0.79% for one prompt and 0.88% for four queued prompts, relative to the wide-only parent.
Stacked on #44910, with native dependency sgl-kernel-xpu #559. Tested parent
8b1ce6cebc8d739f3c97b11d2cb86d43fc5ad9bc; childea813bd72eb6a14b988311d5b778371bdf848050.Changes
Query
(4096, 2 * intermediate_size, 1536)during construction and reuse the existing native route for either supported width. The new binary enables 20 wide and 15 narrow callbacks. The older wide-only binary retains 20 native callbacks and Torch projection plus lookup for the narrow layers. The model continues to share one 128 KiB BF16 table across all 35 MLPs. Existing eligibility, hooks, delegated methods, backend overrides and M128 behavior retain their guards.The native dependency changes host shape selection; its complete packaged BMG device section is byte-identical to its parent. Weights and BF16 rounding boundaries are unchanged.
Validation
Serving performance
One same-device A/B/B/A, with four fresh servers, unchanged BF16 target and assistant snapshots, identical new native sidecar in both arms, and explicit global/request seeds. Each arm has 24 uncached samples per workload, plus decode and compatibility checks.
Both candidate arms are below both controls for each workload. All 480 timed completions were validated; all 120 corresponding cases have identical text/token IDs across all four arms. Sixteen complete 16-token compatibility cases also match text, IDs and every returned logprob field across all arms.
Concurrency-8 fallback checks complete 64 x 128-token requests in both warm and measured blocks per arm, with zero errors and matching zero-cache vectors. Mean reported TPOT is 0.20% slower within 0.51% control drift; no decode gain is claimed. Generated text matches across all four arms for 58/64 cases; the unchanged control repeats 60/64 and the candidate 61/64. These differences do not establish quality causality.
All four arms retain expected hashes and actual mappings for 87 original featureful libraries plus two sidecar libraries, immutable source checks and independent retirement audits. This is exact production-source sidecar serving, not a released full-wheel result. Raw means, outputs, logprobs, cache and provenance were independently audited. Final serving evidence SHA256:
75493bcb7cd575919c5244d34b760e0c13c8682ae793b5ebc57a23268fb5fa6f; unit/source evidence:22750c1f8510bf5c971905ceb43eeefeff8043a309bb591c723654348a102ec1.This draft inherits #44908/#44910's unresolved broader quality/determinism gate. Short paired compatibility checks do not resolve it; the stack is not ready to merge on these results alone.
CI States
Latest PR Test (Base): ❌ Run #42150940895
Latest PR Test (Extra): ❌ Run #42150940885
Latest PR Test (AMD ROCm 10): ❌ Run #42150940771
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: