Skip to content

[Perf] [XPU] gemma4: Use native GEMM for narrow BF16 prefill - #44919

Draft
alexnails wants to merge 4 commits into
alexnails/gemma4-xpu-prefill-gemmfrom
alexnails/gemma4-xpu-prefill-gemm-narrow
Draft

alexnails wants to merge 4 commits into
alexnails/gemma4-xpu-prefill-gemmfrom
alexnails/gemma4-xpu-prefill-gemm-narrow

Conversation

@alexnails

@alexnails alexnails commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Enable Gemma4's existing native BF16 prefill GEMM for its 15 narrow MLP layers when the installed binary confirms support. On the measured B70 workload this lowers uncached long-prefill latency by 0.79% for one prompt and 0.88% for four queued prompts, relative to the wide-only parent.

Stacked on #44910, with native dependency sgl-kernel-xpu #559. Tested parent 8b1ce6cebc8d739f3c97b11d2cb86d43fc5ad9bc; child ea813bd72eb6a14b988311d5b778371bdf848050.

Changes

Query (4096, 2 * intermediate_size, 1536) during construction and reuse the existing native route for either supported width. The new binary enables 20 wide and 15 narrow callbacks. The older wide-only binary retains 20 native callbacks and Torch projection plus lookup for the narrow layers. The model continues to share one 128 KiB BF16 table across all 35 MLPs. Existing eligibility, hooks, delegated methods, backend overrides and M128 behavior retain their guards.

The native dependency changes host shape selection; its complete packaged BMG device section is byte-identical to its parent. Weights and BF16 rounding boundaries are unchanged.

Validation

  • Precommit passed; 11 actual-environment XPU tests passed, including 14 subtests covering both widths, fullgraph/changed-operand poisoned replay and fallbacks.
  • Separate synthetic real-MLP owner profiles prove new-binary 35 native/zero lookup kernels versus actual older-binary 20 native/15 lookups. All 35 output hashes match; these are unit checks, separate from serving.
  • Separate real-server CPU/XPU preflights prove parent 20 wide native/15 lookups and child 20 wide plus 15 narrow native/zero lookups, with exact CPU shapes. Both preflights retire before timing.
  • Native dependency separately validates all 15 trained narrow layers at three scales against parent, Torch BF16 projection plus lookup and FP32 references; three packaged native capability/compile/replay tests pass.

Serving performance

One same-device A/B/B/A, with four fresh servers, unchanged BF16 target and assistant snapshots, identical new native sidecar in both arms, and explicit global/request seeds. Each arm has 24 uncached samples per workload, plus decode and compatibility checks.

Workload Control A / candidate B / candidate B / control A, ms Lower mean latency Control drift
1 x 4096 tokens, 1 output 187.681 / 186.477 / 185.769 / 187.533 0.79% -0.079%
4 queued x 4096 tokens, 1 output each 739.032 / 732.376 / 731.913 / 738.331 0.88% -0.095%

Both candidate arms are below both controls for each workload. All 480 timed completions were validated; all 120 corresponding cases have identical text/token IDs across all four arms. Sixteen complete 16-token compatibility cases also match text, IDs and every returned logprob field across all arms.

Concurrency-8 fallback checks complete 64 x 128-token requests in both warm and measured blocks per arm, with zero errors and matching zero-cache vectors. Mean reported TPOT is 0.20% slower within 0.51% control drift; no decode gain is claimed. Generated text matches across all four arms for 58/64 cases; the unchanged control repeats 60/64 and the candidate 61/64. These differences do not establish quality causality.

All four arms retain expected hashes and actual mappings for 87 original featureful libraries plus two sidecar libraries, immutable source checks and independent retirement audits. This is exact production-source sidecar serving, not a released full-wheel result. Raw means, outputs, logprobs, cache and provenance were independently audited. Final serving evidence SHA256: 75493bcb7cd575919c5244d34b760e0c13c8682ae793b5ebc57a23268fb5fa6f; unit/source evidence: 22750c1f8510bf5c971905ceb43eeefeff8043a309bb591c723654348a102ec1.

This draft inherits #44908/#44910's unresolved broader quality/determinism gate. Short paired compatibility checks do not resolve it; the stack is not ready to merge on these results alone.


CI States

Latest PR Test (Base): ❌ Run #42150940895
Latest PR Test (Extra): ❌ Run #42150940885
Latest PR Test (AMD ROCm 10): ❌ Run #42150940771

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants