Skip to content

[Perf] gemm: Extend BF16 prefill to validated intermediate rows - #561

Draft
alexnails wants to merge 9 commits into
sgl-project:mainfrom
alexnails:alexnails/xpu-bf16-prefill-variable-rows
Draft

alexnails wants to merge 9 commits into
sgl-project:mainfrom
alexnails:alexnails/xpu-bf16-prefill-variable-rows

Conversation

@alexnails

@alexnails alexnails commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Allow the existing optimized BF16 gate/up GEMM to serve M=256, 1024, 2048 and 4095, in addition to 4096, for K=1536 and N=12288/24576. Other shapes retain their previous dispatch. This changes the capability/dispatch predicate and tests; the kernel body, tile layout and BF16 weights are unchanged.

Depends on #559. The incremental change is two files at 06bd1b41f5d3fc3c35a50fd932fa21faf633f848; incremental diff. This draft targets main because the parent branch is in the fork; its upstream diff includes the parent until #559 lands.

Validation on Intel Arc Pro B70:

  • Final compiled native target: 59 pytest cases passed, including capability boundaries, both widths, compile and live graph replay at all five admitted row counts.
  • Final binary: 525 trained-weight comparisons (35 Gemma4 E2B layers × 5 row counts × 3 input scales), zero differing BF16 words against Torch projection plus native activation.
  • An experimental broader dispatch was screened before restricting the admitted rows; M128/512 remain outside the optimized capability.

Companion SGLang PR #44936 queries capability per row count and preserves older-wheel behavior. Its matched serving A/B used the same native binaries in both arms: mean first-token request latency improved 4.25% at M256, 4.69% at M1024, 5.32% at M2048 and 4.84% at M4095. Decode throughput changed +0.18% / -0.07% / +0.95% at request concurrency 1/8/32; treat those small changes as roughly flat.

Serving recipe: BF16 Gemma4 E2B + BF16 assistant, B70 TP1, identical full decode graphs/NEXTN settings, prefix cache disabled, 24 first-token samples per row size; decode blocks 128/640/1536 requests with 128 output tokens each (about 71/71/86 seconds). Single A/B, no statistical significance claim. All 216 first-token outputs match. Longer response/logprob checks are not universally exact; an identical-baseline rerun also differs in all four M512 cases. No new accuracy score or universal numerical parity is claimed. The 525 exact checks concern this operator and their tested inputs.

The target was built and loaded alongside the existing featureful package through an isolated sidecar; this is not full-wheel installation validation. Pre-commit passes. Draft pending parent integration and review.

Evidence archive SHA256: 539e5d8081610493ff860981ec80dce50214116dd4eea522706fecba7667c5eb (raw serving records, source/binary hashes, tests, baseline rerun and retirement records).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant