Skip to content

[Perf] [XPU] gemma4: Accelerate BF16 long-prefill activation - #44908

Draft
alexnails wants to merge 5 commits into
alexnails/xpu-bf16-gelu-mul-lutfrom
alexnails/gemma4-xpu-prefill-lut-standalone
Draft

alexnails wants to merge 5 commits into
alexnails/xpu-bf16-gelu-mul-lutfrom
alexnails/gemma4-xpu-prefill-lut-standalone

Conversation

@alexnails

@alexnails alexnails commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Gemma4 E2B's 4,096-token XPU prefill chunks spend substantial device time in GELU-and-multiply after their gate/up projections. This adds a Triton activation that looks up the native BF16 GELU result for each BF16 gate value, then multiplies by the up projection with the existing rounding sequence. The original gate/up and down GEMMs remain unchanged.

The Gemma4 integration enables this path for unquantized BF16 E2B at TP1 on B70, for both intermediate widths (6,144 and 12,288), only at 4,096 rows. A single nonpersistent 128 KiB table is initialized after weight loading through the existing native activation and shared by all 35 MLPs. Other shapes, forced activation backends, replaced activation methods, hooks, and unsupported configurations use the original path. No new sgl-kernel-xpu GEMM API is required.

Stacked on #44905 (which builds on #44890). This GELU activation table is separate from that PR's softcap comparison-rank table. Model weights remain BF16 and unchanged.

Validation on Arc Pro B70, Torch 2.13.0+xpu:

  • Nine GPU tests pass: public activation, owner/load/reload behavior, real wide/narrow MLPs, fullgraph compilation, captured replay with changed inputs, and fallback behavior. Exhaustive BF16 gate patterns include nonfinite values; NaNs are compared by classification.
  • A regression test replaces the delegated activation methods before their first call. It fails on the previous implementation for both intermediate widths and passes on the submitted source.
  • A CPU/XPU serving trace confirms exactly 35 lookup kernels for one 4,096-token prefill. Startup checks confirm one shared table and 35 eligible MLPs.

Final serving comparison uses unchanged BF16 target and assistant snapshots, the original featureful native wheel, no experimental native-GEMM overlay, and an ABBA sequence (24 uncached samples per workload per block; one output token):

Workload Control A0 / A3 Candidate B1 / B2 Lower latency vs mean controls
1 × 4,096-token prompt 198.038 / 198.386 ms 191.812 / 190.272 ms 3.62%
4 queued × 4,096-token prompts 779.464 / 780.909 ms 754.352 / 751.347 ms 3.50%

Both candidate blocks beat both controls. Control drift was 0.18–0.19%. Four queued requests execute four 4,096-row prefill chunks. These measurements include HTTP/request handling and measure one-output request latency.

All 16 separate compatibility cases match output IDs, text, and every returned logprob field across all four blocks, including nonempty top-5 logprobs for all 16 output tokens. Canonical c8 decode TPOT was 6.234 / 6.295 / 6.310 / 6.309 ms in ABBA order: candidate mean is 0.49% slower than mean controls, within 1.20% control drift, and 0.10% faster than the final control. All 64 requests per block completed without errors and produced 128 tokens. No decode improvement is claimed.

Draft status: a causal model-quality effect remains unresolved. The following evidence is preserved rather than treating aggregate score differences as a quality pass:

  • Full paired 200-question long-context GSM8K with four concurrent requests: control 168/200, candidate 160/200; 10 control-only and two candidate-only correct answers. Zero errors; two/four truncations. This neutral-padding workload is not ordinary GSM8K.
  • Full paired 200-question run with serialized requests: control 164/200, candidate 163/200; six control-only and five candidate-only correct answers; zero errors and three truncations each. All 400 ordered prefill batches, single-request decode batches, request bodies and prompt-token counts matched, but only 101/200 generations were identical. Both requests used seed 42; the servers' global seeds were uncontrolled and differed (309648033 versus 357593183). That is a protocol difference, not an established cause.
  • A diagnostic on the actual serving tensors compared lookup with the saved standard native activation at every eligible layer call across the same 200 questions: 6,300 ordered comparisons across all 35 layers, totaling 249,141,657,600 elements, with zero non-NaN bit or NaN-class mismatches. All request maps and 895,494 uncached prompt tokens reconciled. The observer intentionally adds work; no latency result comes from this diagnostic.
  • A bounded unchanged-baseline repeat pinned global and request seeds to 42, serialized exact stored request bodies, flushed cache before each request, and verified identical uncached prefill/decode shapes. Two of four cases changed text/token counts within the same baseline server. The diagnostic stopped after eight requests as specified. This demonstrates baseline nonrepeatability under this protocol; it does not establish feature parity or explain the cause.

Source is frozen at ebc9d5476aa68dceae837acf39b85ca25c8bbf0d. Evidence archives, independently verified locally: final source/tests/ABBA 540cda67f3025997d61bc8e2b0192dac8d45d9c873e485491021ef09aea366dd (96 files); paired concurrent quality 6e6908b4609d4f5ac4e576ebcfeb33e5d5af4f400ac94d031632899f021050c4 (34 files); same-input oracle cba522973e782bf4d088b3c7d4990d6c9d5af7f759ee978eca317aaf42c1dd67 (19 files); serialized quality e3efbfb804d5829296c19d59b14b047d5f789e95669242409b3d8365d7bf0ead (30 files); baseline repeat 0a0a35dea12a92b7f2950672b7da9461fe12033aa06178c6c8d2382a71186a8b (18 files). No full-feature native GEMM wheel or experimental GEMM API is required by this integration.


CI States

Latest PR Test (Base): ❌ Run #42150936992
Latest PR Test (Extra): ❌ Run #42150936885
Latest PR Test (AMD ROCm 10): ❌ Run #42150936931

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

main renamed BaseFusedOp.dispatch_forward to _resolve_forward_method and
removed GeluTanhAndMulOp, so the prefill LUT checks the renamed resolver
and builds its table with sgl_kernel's gelu_tanh_and_mul, the kernel
GeluAndMul.forward_xpu runs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants