Repository navigation
Fix overflow in legacy and v1 DSA top-k selectors - #37625
Conversation
zRzRzRzRzRzRzR
left a comment
There was a problem hiding this comment.
Issue received, following and test this.
|
This PR seems to have only fixed its copy in the kpool JIT kernel. There are several other copies of the same code affect all dsa model. |
I was working on addressing issues with GLM 5.3 Flash in my SGlang build that lead to crashes directly, but was keeping the breadth smaller to minimize review burden. It sounds like you desire the full and complete fix, so let me evaluate that. Thank you for the feedback. |
1436d19 to
7ad39a0
Compare
|
@zRzRzRzRzRzRzR Updated the branch in response to the scope issue you identified.
The branch is rebased onto SGLang |
|
The fix in this PR is correct, but it costs quite a bit of performance as it stands. Model output quality before vs. after the fix doesn't seem to differ much on long text (at least when measured end-to-end, e.g. by benchmark scores such as GQPA, need more than 32K context) — an interesting finding. Maybe there's a better way to fix this? cc @JustinTong0323 |
Kernel level there is some perf regression but not obvious in e2e, so I think it shall be ok. |
|
@DarkSharpness to review :) |
|
@ormandj could you please rebase to the latest main branch? |
|
Real-model data point for the legacy (v1) part, since it just got a review: we hit this truncation on DeepSeek-V4-Flash with real long prompts, not only synthetic inputs. Where. On How often it triggers. We captured the indexer scores (fp16) for 22 prompts of about 128K tokens (RULER vt / cwe / qa_1 and InfiniteBench longbook_choice_eng), on all 21 CSA layers and 1,536 rows per prompt, for 569,856 (row, layer) pairs in total. Replaying stage 1 bit-exactly from those scores:
Whether the selection is actually wrong. We counted a row as wrong when a non-selected candidate scores strictly higher than the lowest selected one (fp16 rounding is monotone, so this proves the top-k is inexact):
On GPU (H100, that tree's kernel, 50 repeats).
So on real DSV4 inputs the legacy path is exact whenever it does not truncate, and truncation happens only on repetitive long inputs, where it nearly always returns a wrong set. We have not measured the end-to-end quality impact. |
759070d to
1eebe42
Compare
|
@DarkSharpness, replying to your review:
The v2 changes, including the coarse-bound and all-equal paths, are removed from this PR. The diff now covers the four legacy/v1 selectors so those fixes can be reviewed and merged independently. I will submit the v2 changes in a separate PR. |
|
@zRzRzRzRzRzRzR, replying to your selector-scope comment:
The patch covers all four legacy/v1 copies: KPool JIT, DeepSeek-V4 JIT v1, legacy CUDA AOT and DeepSeek-V4 AOT. The separate QSA fast_topk copy on newer main is outside this patch. |
|
@DarkSharpness, replying to your rebase request:
Rebased onto main |
|
@AranKomat, replying to your AMD test-coverage question:
The suite compares selected-value multisets and allows index swaps among equal scores. It includes oversized and exact-capacity bins, ties, offsets and changed inputs. The retained ROCm AOT cases have not been rerun on gfx942; the current GPU results are from SM120. |
|
Updated the code and PR body. The diff now covers the four legacy/v1 top-k selectors. The v2 changes are removed so the legacy/v1 fix can be reviewed and merged independently; I will submit v2 in a separate PR. Normal bins retain their existing path, while overflow handling uses exact FP32 cutoff refinement and a final row scan. Existing ROCm tests are retained. The body lists current SM120 test results and the remaining selector/platform coverage. |
1eebe42 to
9963cfb
Compare
KPool already rescans overflowing bins on main; keep its implementation and add regression coverage for it, the JIT v1 selector and the AOT selectors. Co-authored-by: Benjamin Oldenburg <benjamin.oldenburg@ordis.co.th>
Main renamed fast_kpool_topk_transform_fused's pool_size to kpool (sgl-project#42671). The overflow cases pass against main's exact rescan.
Each refinement pass also emits the values the previous key byte placed strictly above the cutoff, so an oversized coarse bin takes five input passes instead of six. Use the JIT device macro, drop the redundant fp16 include and trailing barrier, and shorten the comments.
9963cfb to
e28fa20
Compare
|
/tag-and-rerun-ci |
|
/tag-and-rerun-ci extra |
|
The 26 failed checks on
The PR's tests passed in |
…k-274, wanted_stack LANDED marks, CAVEATS docs-only ledger items (A) Header: STACK line 260->274; three wave lines (+spec-mamba-2026-10-28 (pr45491+pr45435+pr45472+pr45670), +admission-2026-10-28(pr46272*+ pr43484*+pr46496*+pr46643*+pr45607*+pr45789*), +parsers-2026-10-28 (pr44124+pr45631+pr45855+pr46167)); tip line -> 2714457; updated 261-274 headline block (entry-119 SUPERSEDED by upstream sgl-project#45435/sgl-project#45472; classic-pool mamba admission gate w/ upstream ImportError CAUGHT; UNCONDITIONAL latch reset = TTFT watch; coder-parser union; logits trilogy as CONTRACT UNIT; 14 picks, 0 integrator conflicts). (B) wanted_stack.txt: FAM-A5 + FAM-B234 blocks deleted -> LANDED records (chain shas, interlock audit, supersession + behavior-change pointers into entries 261-270); SECTION-B sgl-project#44124/trilogy blocks -> LANDED; sgl-project#37958/sgl-project#45559 block -> DONE (verdict now in CAVEATS). (C) DOCS-ONLY LEDGER ITEMS: CAVEATS gains the sgl-project#37958/sgl-project#45559 VERIFIED- INERT bullet (two independent hard gates, coarse-barrier fallback, do-not-re-litigate-at-merge-sgl-project#5 line, revisit condition = dropping NEXTN or HiCache) and the dated C0b/C0c corrections (sgl-project#44030 obsolete rationale, sgl-project#44858 REVERSED by sgl-project#45289 [ported-pending wt-moe], sgl-project#44115 permanent N/A, 10-28 open-status re-verification list, no-pin-moves note: 0.7.2/0.4.9 floor unchanged) with the C0a tick (sgl-project#37625/sgl-project#37941/sgl-project#39727 still outstanding; only sgl-project#43927/sgl-project#44849 partially satisfied). C0b/C0c/C8 marked DONE in wanted_stack; C0a stays open.
Motivation
Legacy DSA top-k selection can drop candidates when a coarse radix bin reaches its fixed buffer capacity, selecting lower-scoring tokens before full scores are compared. This patch covers the DeepSeek-V4 JIT v1, legacy CUDA AOT and DeepSeek-V4 AOT selectors. KPool JIT already rescans overflowing bins on main (#39273), so its kernel is unchanged here and only gains regression coverage.
Modifications
Keep normal-bin selection. For a coarse bin larger than the stash, find the exact FP32 cutoff one key byte per row scan; each scan also emits the scores the previous byte placed strictly above the cutoff, and the last scan emits the remaining ones and the cutoff ties (six scans of the row including the coarse histogram, down from seven). Preserve page mapping,
-1padding and synchronization around shared-memory reuse. The JIT header takes its CUDA/HIP types fromsgl_kernel/utils.cuh. Retain the additive ROCm tests. Co-authored with Benjamin Oldenburg (@bold84).Add a CUDA test for
topk_transform_paged(JIT v1) covering threshold bins on both sides of the 8192-entry stash, strictly higher values ahead of an oversized bin, cutoff ties, clusters that separate only in one of the lower three FP32 key bytes, runtimetopkvalues, short rows, score rows padded past their lengths, a noncontiguous score view, and both raw and page-mapped outputs. Add KPool regressions at its 4096-entry stash for oversized and exact-capacity bins, cutoff ties, clusters that separate in the lower three key bytes, padded rows and page-table expansion, registered for CUDA and AMD.The v2 changes are removed from this PR so the legacy/v1 fix can be reviewed and merged independently. I will submit the v2 changes in a separate PR.
Accuracy Tests
On RTX PRO 6000 Blackwell (SM120), a combined build passed 12 KPool JIT tests/38 subtests and 7 focused legacy CUDA AOT cases, including oversized coarse bins. One ROCm-only case was skipped. Those results predate the rebase onto main's KPool implementation.
On RTX PRO 6000 Blackwell (SM120), torch 2.14.1+cu130, Triton 3.8.0, at this branch,
test_topk_v1.pypassed (15 tests) andtest_dsa_kpool_topk_transform.pypassed against main's KPool kernel (12 tests, 38 subtests). Against main's unpatchedtopk_v1.cuh, 9 of the 15 v1 tests fail (the overflow cases). The AOT header was compiled for sm_120 in a standalone harness and matched a CPU reference on random rows, single-bin rows above the stash, and strictly higher values plus cutoff ties. The AOT test matrix and ROCm tests have not been run.Speed Tests and Profiling
Rows whose coarse threshold bin exceeds the stash take six row scans; other rows are unchanged. No speed benchmark was run.
Checklist
Developed with AI assistance.
CI States
Latest PR Test (Base): ✅ Run #38096800282
Latest PR Test (Extra): ❌ Run #38096800233
Latest PR Test (AMD ROCm 10): ❌ Run #38096800271