Skip to content

[Perf] [XPU] activation: Add native-BF16 GELU lookup - #44905

Open
alexnails wants to merge 3 commits into
alexnails/gemma4-xpu-greedy-lutfrom
alexnails/xpu-bf16-gelu-mul-lut
Open

alexnails wants to merge 3 commits into
alexnails/gemma4-xpu-greedy-lutfrom
alexnails/xpu-bf16-gelu-mul-lut

Conversation

@alexnails

@alexnails alexnails commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Add sglang.kernels.ops.activation.bf16_gelu_tanh_mul_lut, an inference-only Triton XPU operator for BF16 [gate, up] matrices. It indexes a caller-owned 65,536-entry BF16 table by the gate's raw bits, then multiplies the looked-up native GELU result by up with FP32 opmath and BF16 output rounding. This retains the native BF16 activation intermediate and leaves GEMMs and model weights unchanged.

The namespace entry is lazy, and the wrapper supports fullgraph compilation and XPU graph replay. Guards enforce contiguous BF16 inputs/table on one XPU, inference-only use, the table size, and the 32-bit addressing limit; empty outputs are supported. The caller owns table construction and lifetime. A following Gemma4 integration supplies a table populated through the existing native activation.

Stacked on #44890 (797dbef49803cd37f4da51383b0e9188e427f439). This GELU value table is separate from that PR's softcap comparison-rank table.

Validation on Arc Pro B70 with Torch 2.13.0+xpu:

  • Three registered GPU tests pass: fullgraph and changed-input/poisoned captured replay at offsets 0/1/32; exhaustive 65,536 BF16 gate patterns with finite/nonfinite up values; and an allocation-free addressing-bound check. Finite values, signed zeros, and infinities match the native activation exactly; NaNs match by classification.
  • Two independent trained-projection microbenchmarks at M4096 show 17.97–18.04% lower activation latency for I12288 and 18.63–18.79% for I6144 versus the native activation. Physical projection banks are 805 MB / 402 MB, and retained-output poisoning and changed-input replay are checked. These are activation-only results.
  • The following model integration has separately passed nine GPU tests and a serving dispatch/profile check; those changes and serving claims are outside this three-file PR. Its model integration is published as draft [Perf] [XPU] gemma4: Accelerate BF16 long-prefill activation #44908 and is not ready to land: a paired 200-question long-context check scored 168/200 for control and 160/200 for candidate (zero request errors; 10 control-only and two candidate-only correct). A subsequent diagnostic compared lookup and the saved standard native activation on identical serving tensors for all 35 layers across the same 200 questions: 6,300 comparisons / 249,141,657,600 elements, with zero bit or NaN-class mismatches. The full serialized pair verified matching prefill batch shapes and scored 164/200 versus 163/200, with 101/200 identical generations. The two servers used different uncontrolled global seeds despite request seed 42; greedy source paths do not show an active RNG consumer, so this is a disclosed protocol difference, not an established cause. A bounded unchanged-baseline repeat with both global and request seeds fixed at 42 changed two of four completions within the same server (eight requests total). This establishes baseline variation under that protocol; the feature's quality effect remains unresolved. The model integration remains a draft. No model-quality claim is made for this kernel PR.

Reproducible local evidence: activation archive SHA256 1ff500da6df7651c695fae0f562394ab58463fa95d5817dd351a79e2ff355ae8; final-source validation/serving evidence SHA256 540cda67f3025997d61bc8e2b0192dac8d45d9c873e485491021ef09aea366dd (96 files independently verified). Source commit: c6b2000ff43c503cfcf530b5cb9a77e36e797012.

Additional diagnostic evidence: same-input serving oracle SHA256 cba522973e782bf4d088b3c7d4990d6c9d5af7f759ee978eca317aaf42c1dd67 (19 files / all 6,300 ordered checks independently verified); serialized quality evidence SHA256 e3efbfb804d5829296c19d59b14b047d5f789e95669242409b3d8365d7bf0ead (30 files independently verified).


CI States

Latest PR Test (Base): ❌ Run #42143750050
Latest PR Test (Extra): ❌ Run #42143749979
Latest PR Test (AMD ROCm 10): ❌ Run #42143750085

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@alexnails alexnails changed the title [XPU] Add native-BF16 GELU lookup activation [Perf] [XPU] activation: Add native-BF16 GELU lookup Oct 17, 2026
@alexnails alexnails added the run-ci CI: run the baseline test suite on this PR label Oct 17, 2026
@AgainstEntropy AgainstEntropy removed the run-ci CI: run the baseline test suite on this PR label Nov 6, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 16, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 17, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 18, 2026
airMeng added a commit to airMeng/sglang that referenced this pull request Nov 18, 2026
Mechanical compatibility with main CI suite renaming in sgl-project#50874.
Retain the original one-GPU extra and four-GPU nightly assignments.
Sources: sgl-project#44905, sgl-project#48374, sgl-project#43891.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants