Skip to content

[Perf] [Intel] gemma4: Extend BF16 prefill GEMM to intermediate rows - #44936

Draft
alexnails wants to merge 4 commits into
alexnails/gemma4-xpu-ple-add-scalefrom
alexnails/gemma4-xpu-intermediate-prefill-gemm
Draft

alexnails wants to merge 4 commits into
alexnails/gemma4-xpu-ple-add-scalefrom
alexnails/gemma4-xpu-intermediate-prefill-gemm

Conversation

@alexnails

@alexnails alexnails commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Gemma4 E2B's BF16 XPU prefill GEMM fast path is restricted to exactly 4096 rows, leaving useful intermediate shapes on the ordinary projection/activation path. This admits 256, 1024, 2048 and 4095 rows when the loaded native library explicitly reports support.

Stacked on #44925, with the existing 4096-row path preserved. Companion native PR: sgl-project/sgl-kernel-xpu#561 (depends on sgl-kernel-xpu#559).

Modifications

Query native capability independently for (256, 1024, 2048, 4095, 4096) at each projection width. Dispatch the fused GEMM only for reported row counts. Missing/false queries and older 4096-only wheels preserve the original 4096-row standalone-lookup/native paths. Unadmitted shapes use the parent model path. Existing hardware, dtype, TP, weight, backend, hook, delegated-method and gradient guards remain; add a 32-bit addressing bound.

Activation and PLE retain their existing 4096-row dispatch. Extend the existing integration fixture with admitted shapes, compile/replay, ordinary-path fallback checks, and explicit legacy-wheel capability coverage.

Accuracy Tests

  • All three existing XPU model integration files pass: activation lookup, PLE add/scale, and prefill GEMM. The GEMM fixture covers both widths, changed live operands, compile/replay, hooks/backend fallbacks, and missing/false/wide-only/4096-only capability queries.
  • Companion final native binary: 59 pytest cases pass. Across 35 trained layers × 5 admitted row counts × 3 input scales, all 525 operator comparisons match Torch projection plus native activation exactly in BF16 words.
  • Full serving A/B: all 216 one-token outputs match. Full 16-token text/token-ID/logprob records match exactly in 23/28 cases. Differences occur at M256 index2 (top-five logprobs) and the four M512 cases; an identical-baseline rerun independently reproduces logprob differences in these same five cases. Two M512 text/token-ID sequences differ in this A/B; none differ in that shorter baseline-repeat compatibility check, so the cause of output variation remains open. M512 retains the parent dispatch. Longer decode text matches are 124/128, 585/640 and 1198/1536 at concurrency1/8/32. Thus this evidence does not establish universal output parity. No new model accuracy score was run.

Speed Tests and Profiling

Intel Arc Pro B70, TP1, unchanged BF16 target/assistant snapshots, same full decode graphs and NEXTN settings, prefix cache disabled in both arms. Compare parent a3c379de38 against a2dfdf0eb0, using the same 89 native binaries in both arms; the parent only dispatches this GEMM at 4096 rows.

Prefill rows Parent mean first-token request, ms Candidate, ms Reduction
256 39.724 38.034 4.25%
1024 54.182 51.643 4.69%
2048 96.482 91.352 5.32%
4095 192.406 183.093 4.84%
4096 180.857 181.110 -0.14%

24 samples per row count. Unchanged 26/61/128/512-row paths have mean changes between -1.13% and +1.05%; no short-prompt gain is claimed.

Request concurrency Parent / candidate TPOT, ms Parent / candidate output tokens/s Throughput change
1 4.046 / 4.038 230.33 / 230.74 +0.18%
8 6.554 / 6.550 1154.81 / 1153.99 -0.07%
32 13.561 / 13.440 2278.44 / 2300.03 +0.95%

Decode blocks use 128/640/1536 requests, each generating 128 tokens, with about 71/71/86 seconds per arm. TPOT is recomputed as (mean E2E - mean TTFT)/127; output throughput is accepted output tokens divided by measured wall duration. Treat decode as roughly flat. One A/B, no repeated-control drift estimate for this full comparison or significance claim. Both servers retired; source/native hashes rechecked before and after each arm.

Evidence archive SHA256: 539e5d8081610493ff860981ec80dce50214116dd4eea522706fecba7667c5eb.

Checklist

  • Pre-commit passes.
  • Extend existing model integration coverage; run all three relevant fixtures.
  • Provide scoped numerical checks and serving measurements.
  • Independent whole-model accuracy evaluation and upstream CI.

Draft while parent/native integration and numerical variability receive review.


CI States

Latest PR Test (Base): ❌ Run #42150942862
Latest PR Test (Extra): ❌ Run #42150942995
Latest PR Test (AMD ROCm 10): ❌ Run #42150942860

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants