Skip to content

[Perf] gemm: Reuse BF16 prefill GEMM for narrow gate/up - #559

Draft
alexnails wants to merge 8 commits into
sgl-project:mainfrom
alexnails:alexnails/xe20-gate-prefill-narrow
Draft

alexnails wants to merge 8 commits into
sgl-project:mainfrom
alexnails:alexnails/xe20-gate-prefill-narrow

Conversation

@alexnails

Copy link
Copy Markdown
Collaborator

Change

Enable the existing M-fast BF16 gate/up prefill specialization for (M, N, K) = (4096, 12288, 1536), covering Gemma4's 15 narrow MLP layers. The shared binary-capability/dispatch predicate currently admits only the wide shape. Older callers and the M128 route retain their behavior.

This is a two-file child of #558, tested at 891b15b5e143be53192523bc427594a5868ca088 against parent c743cdf3d1b012296333da87701718f506eca9d7. The upstream base is main because the unmerged parent branch lives in the fork; the incremental change is the shape predicate and corresponding query/compiled-replay tests.

No kernel body, tile, weight format, schema or API changes. The complete packaged BMG device section is byte-identical to the parent (28,317 bytes, SHA256 62b25e75dbd0d434fc63d4c177261e32049f38cef8095697540a7787bf86d7ae). All 22 shared MoE headers are unchanged. The host query and dispatcher use the same predicate.

Validation

  • Precommit passed. All three changed package tests passed: binary capability and fullgraph/changed-operand replay for both supported widths, including offsets 0/1/32 and poisoned replay with changed inputs, weights and table.
  • Separate parent and child processes checked all 15 trained narrow layers at M4096 over three input scales. All 45 input/weight/raw gate/raw up/fused-output hash sets match the parent exactly. Outputs also match Torch BF16 projection plus the published BF16 lookup exactly in these cases. Full FP32 RMS/max error metrics match the Torch reference.
  • Actual CPU/XPU traces prove the parent uses BN32 and the child BN128, with exact operator shapes. Misaligned offset1 capture with changed inputs/weights/table passes an independent exactly representable projection oracle, including signed zeros and GEMM's initial positive-zero accumulation.
  • The packaged wheel preserves 1,030 other entries; RECORD and actual loaded native-library provenance are retained.

Performance

Intel Arc Pro B70, BF16, M4096/N12288/K1536, gradients disabled and inference mode disabled. Control is actual Torch BF16 F.linear plus the published standalone BF16 GELU lookup.

Each fresh process rotates 16 distinct physical 36 MiB weight allocations (576 MiB), retains 32 disjoint outputs across the two backends, validates poisoned/current-input replay for every bank, and measures six mirrored rounds. Both runs win all six rounds; source, package and validation hashes remain unchanged before/after.

Fresh-process seed Torch + lookup, us Native, us Lower latency
2026101741 1365.816 1315.799 3.66%
2026101742 1366.490 1315.116 3.76%

These are operator microbenchmarks. SGLang integration/serving validation is separate; no serving or model-quality gain is claimed.

Evidence archives: numerical/source/dispatch 5192e75dbc5aae4ffa0ca2542b77e4bc364be80d93b2c3b4ac924bf03b3050a0 (35 payloads), targeted tests bd4218217fa56336f9fd080c44339a480671cafa85f66537eaea86fee5a1588c (5 payloads), and both cold runs 9cce3fa8993d5f1ccfbb25ffb63a63bacdf2e71f9b025ddbe62942ad49b49ff3 (24 payloads). Payload hashes and raw-round means were independently verified.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant