Skip to content

[Perf] [Intel] gemma4: Use exact softcap comparison ranks for greedy verification - #44890

Open
alexnails wants to merge 5 commits into
alexnails/gemma4-xpu-greedy-logitsfrom
alexnails/gemma4-xpu-greedy-lut
Open

alexnails wants to merge 5 commits into
alexnails/gemma4-xpu-greedy-logitsfrom
alexnails/gemma4-xpu-greedy-lut

Conversation

@alexnails

@alexnails alexnails commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #44884; this PR contains only the exact comparison-rank lookup follow-up.

Motivation

The fused greedy-verification logits reduction still computes FP32 softcap for every vocabulary element. BF16 has only 65,536 bit patterns, so build exact comparison ranks through the existing native FP32 softcap once and reduce those ranks instead. This preserves transformed equality classes, including softcap saturation, signed zero, and NaNs. Weights, BF16 logits, and the head GEMM remain unchanged.

Modifications

  • Extend the XPU reduction with an optional immutable 256 KB comparison table and unsigned-rank/first-index reductions.
  • Keep the table in a nonpersistent LogitsProcessor buffer, keyed by generation device and softcap. Construct it during eager warmup; unmatched first capture or compile keeps the existing tanh reduction.
  • Carry the immutable table reference with deferred logits through graph replay. Slice token rows without slicing the table. Current-request eligibility and the ordinary-logits fallback remain unchanged.

The table records exact FP32 output ordering and equality for the actual native softcap operation; it does not approximate or change model precision. Scope remains no-gather Gemma4 BF16 FrozenKV target verification, validated at TP1.

Accuracy Tests

  • Exhaustive native FP32 softcap order/equality proof across all 65,536 BF16 bit patterns for softcap None, zero, and 30; device lookup checked for every pattern.
  • Six XPU kernel tests passed, including signed zero, saturation ties, first NaN/global index, chunk boundaries, public lazy dispatch, Meta shapes, and graph replay with changed inputs.
  • Two real LogitsProcessor lifetime/capture tests passed, covering separate owners, cap changes, nonpersistent state, immutable captured references, and no table construction inside first capture.
  • 75 CPU namespace/policy tests and 15 subtests passed. Rank-aware Meta smoke and pre-commit passed.
  • Fifteen controlled same-GPU serving cases matched the parent exactly, including returned token logprobs and repetition-penalty/bias/min-new-token fallbacks.
  • Official GSM8K200, temperature 0, seed 42, eight threads: parent 159/200 (79.5%), candidate 164/200 (82.0%), zero errors and two truncations each. Earlier parent serving scored 165/200; these are compatibility results, with no quality-improvement claim.
  • Candidate official GSM8K200 temperature 1: 165/200 (82.5%), zero errors, two truncations. This uses the ordinary-logits fallback, not the changed greedy reduction.

Exact operator and controlled-serving parity do not establish arbitrary batch-independent model bitwise equivalence.

Speed Tests and Profiling

Same-GPU ABBA versus parent 7aba: Gemma4 E2B BF16, TP1, MTP steps 3/top-k 1/draft tokens 4, full graph backend; actual target-verify capture sizes end at 48, overlap disabled, page size 64, max running 64, max total tokens 32768, context 8192, static memory fraction 0.70. Each server switch warmed 128 requests at c32 with 256 output tokens. Timing uses seed 42 and 256 outputs, 64 requests at c1 and 128 at c8/c32. All 12 runs completed without errors, with identical input/output token counts.

Mean TPOT, ms, control/candidate/candidate/control:

Concurrency Control A Candidate A Candidate B Control B Lower time vs mean controls Lower time vs final control
1 4.129629 4.120205 4.115793 4.093881 -0.152% -0.589%
8 6.017597 5.979372 5.986314 5.960932 0.107% -0.368%
32 10.882142 10.804637 10.778815 10.925135 1.026% 1.221%

C1/c8 are near flat and slightly slower versus final controls. At c32, both candidate repeats beat both controls; final control was 0.395% slower than initial, and candidate mean was 0.831% lower than initial control. On 55/128 requests with identical texts across all four c32 runs, mean streaming-time improvement was 1.014%. Controls differed on 38/128 texts; candidates differed on 48/128. This subset does not remove between-run drift.

Additional bounded c64 ABBA uses 256 requests/run with 256 output tokens, a deterministic repeated copy of the 200 canonical prompts, and 256-request c64 warmup per server switch:

Concurrency Control A Candidate A Candidate B Control B Lower time vs mean controls Lower time vs final control
64 20.931916 20.349956 20.459503 20.564288 1.655% 0.776%

Control improved 1.756% between repeats, so the final-control comparison is conservative. Both candidate repeats beat final control. The 109/256 identical-text subset improved 1.753% in mean streaming time; controls differed on 92/256 texts and candidates on 107/256. These figures retain between-run drift limitations.

C64 is mixed eager/captured execution: startup logs resolve target-verify capture sizes through 48, every logged above 48-request decode batch used eager execution, and smaller decode batches also used captured execution. All four runs succeeded without errors and had identical input/output token counts.

A real c32 serving trace contains 16 rank partial/final reductions, confirming actual dispatch. Profile timing alone is not an end-to-end gain measurement.

Bounded eager head-plus-finalization microbenchmark, four rotating original BF16 weight banks (3.2 GB), compared to the parent tanh reduction:

Rows Parent us Rank-table us Lower time
4 1371.28 1370.04 0.09%
32 1524.96 1500.50 1.60%
128 1971.09 1822.13 7.56%
256 2576.16 2217.53 13.92%

These are microbenchmarks, not serving gains. Fixed 4096 chunks use 16 B shared memory and report zero spill bytes; the table adds 256 KB model-owned GPU storage and indexed read traffic. An 8192-chunk candidate offered no consistent improvement. No achieved-occupancy claim is inferred from compiler resource reports.

Checklist

  • Format code with pre-commit.
  • Add kernel and runtime lifetime tests.
  • Document exact semantics, state ownership, capture fallback, and scope.
  • Finish and archive paired serving benchmarks and official quality results.
  • Follow SGLang code style.

CI States

Latest PR Test (Base): ❌ Run #42143750620
Latest PR Test (Extra): ❌ Run #42143750296
Latest PR Test (AMD ROCm 10): ❌ Run #42143750299

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

Same int32 row * STRIDE overflow as the softcap path: past 8192 rows of a
262144-entry vocabulary the rank kernel addressed past 2**31 elements.
@AgainstEntropy

Copy link
Copy Markdown
Collaborator

_rank_partial has the same int32 row * STRIDE overflow I described on #44884 (#44884 (comment)): with a 262144-entry vocabulary, any input with more than 8192 rows overflows.

I pushed two commits to this branch:

With the fixes, the rank path gives the same tokens as the softcap path at 8193 and 12000 rows. test_softcap_argmax.py and test_deferred_verify_logits_xpu.py pass, except their XPU-graph capture tests, which fail in our container only because of its old oneAPI. The PRs stacked above this one will get the fix the next time they are merged up.

@AgainstEntropy AgainstEntropy removed the run-ci CI: run the baseline test suite on this PR label Nov 6, 2026
…a4-xpu-greedy-lut

# Conflicts:
#	python/sglang/srt/layers/logits_processor.py

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants