Repository navigation
Conversation
alexnails
requested review from
BBuf,
DarkSharpness,
Edwardf0t1,
Fridge003,
HaiShaw,
HydraQYH,
Qiaolin-Yu,
Ying1123,
celve,
ch-wan,
hnyls2002,
ispobock,
merrymercy and
yuan-luo
as code owners
October 17, 2026 11:24
This was referenced Oct 17, 2026
…a4-xpu-greedy-lut
Same int32 row * STRIDE overflow as the softcap path: past 8192 rows of a 262144-entry vocabulary the rank kernel addressed past 2**31 elements.
Collaborator
|
I pushed two commits to this branch:
With the fixes, the rank path gives the same tokens as the softcap path at 8193 and 12000 rows. |
…a4-xpu-greedy-lut # Conflicts: # python/sglang/srt/layers/logits_processor.py
…a4-xpu-greedy-lut
This was referenced Nov 9, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #44884; this PR contains only the exact comparison-rank lookup follow-up.
Motivation
The fused greedy-verification logits reduction still computes FP32 softcap for every vocabulary element. BF16 has only 65,536 bit patterns, so build exact comparison ranks through the existing native FP32 softcap once and reduce those ranks instead. This preserves transformed equality classes, including softcap saturation, signed zero, and NaNs. Weights, BF16 logits, and the head GEMM remain unchanged.
Modifications
The table records exact FP32 output ordering and equality for the actual native softcap operation; it does not approximate or change model precision. Scope remains no-gather Gemma4 BF16 FrozenKV target verification, validated at TP1.
Accuracy Tests
Exact operator and controlled-serving parity do not establish arbitrary batch-independent model bitwise equivalence.
Speed Tests and Profiling
Same-GPU ABBA versus parent 7aba: Gemma4 E2B BF16, TP1, MTP steps 3/top-k 1/draft tokens 4, full graph backend; actual target-verify capture sizes end at 48, overlap disabled, page size 64, max running 64, max total tokens 32768, context 8192, static memory fraction 0.70. Each server switch warmed 128 requests at c32 with 256 output tokens. Timing uses seed 42 and 256 outputs, 64 requests at c1 and 128 at c8/c32. All 12 runs completed without errors, with identical input/output token counts.
Mean TPOT, ms, control/candidate/candidate/control:
C1/c8 are near flat and slightly slower versus final controls. At c32, both candidate repeats beat both controls; final control was 0.395% slower than initial, and candidate mean was 0.831% lower than initial control. On 55/128 requests with identical texts across all four c32 runs, mean streaming-time improvement was 1.014%. Controls differed on 38/128 texts; candidates differed on 48/128. This subset does not remove between-run drift.
Additional bounded c64 ABBA uses 256 requests/run with 256 output tokens, a deterministic repeated copy of the 200 canonical prompts, and 256-request c64 warmup per server switch:
Control improved 1.756% between repeats, so the final-control comparison is conservative. Both candidate repeats beat final control. The 109/256 identical-text subset improved 1.753% in mean streaming time; controls differed on 92/256 texts and candidates on 107/256. These figures retain between-run drift limitations.
C64 is mixed eager/captured execution: startup logs resolve target-verify capture sizes through 48, every logged above 48-request decode batch used eager execution, and smaller decode batches also used captured execution. All four runs succeeded without errors and had identical input/output token counts.
A real c32 serving trace contains 16 rank partial/final reductions, confirming actual dispatch. Profile timing alone is not an end-to-end gain measurement.
Bounded eager head-plus-finalization microbenchmark, four rotating original BF16 weight banks (3.2 GB), compared to the parent tanh reduction:
These are microbenchmarks, not serving gains. Fixed 4096 chunks use 16 B shared memory and report zero spill bytes; the table adds 256 KB model-owned GPU storage and indexed read traffic. An 8192-chunk candidate offered no consistent improvement. No achieved-occupancy claim is inferred from compiler resource reports.
Checklist
CI States
Latest PR Test (Base): ❌ Run #42143750620
Latest PR Test (Extra): ❌ Run #42143750296
Latest PR Test (AMD ROCm 10): ❌ Run #42143750299
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: