Skip to content

Fix overflow in legacy and v1 DSA top-k selectors - #37625

Merged
BBuf merged 4 commits into
sgl-project:mainfrom
ormandj:pr/kpool-topk-fix-main
Oct 15, 2026
Merged

BBuf merged 4 commits into
sgl-project:mainfrom
ormandj:pr/kpool-topk-fix-main

Conversation

@ormandj

@ormandj ormandj commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Legacy DSA top-k selection can drop candidates when a coarse radix bin reaches its fixed buffer capacity, selecting lower-scoring tokens before full scores are compared. This patch covers the DeepSeek-V4 JIT v1, legacy CUDA AOT and DeepSeek-V4 AOT selectors. KPool JIT already rescans overflowing bins on main (#39273), so its kernel is unchanged here and only gains regression coverage.

Modifications

Keep normal-bin selection. For a coarse bin larger than the stash, find the exact FP32 cutoff one key byte per row scan; each scan also emits the scores the previous byte placed strictly above the cutoff, and the last scan emits the remaining ones and the cutoff ties (six scans of the row including the coarse histogram, down from seven). Preserve page mapping, -1 padding and synchronization around shared-memory reuse. The JIT header takes its CUDA/HIP types from sgl_kernel/utils.cuh. Retain the additive ROCm tests. Co-authored with Benjamin Oldenburg (@bold84).

Add a CUDA test for topk_transform_paged (JIT v1) covering threshold bins on both sides of the 8192-entry stash, strictly higher values ahead of an oversized bin, cutoff ties, clusters that separate only in one of the lower three FP32 key bytes, runtime topk values, short rows, score rows padded past their lengths, a noncontiguous score view, and both raw and page-mapped outputs. Add KPool regressions at its 4096-entry stash for oversized and exact-capacity bins, cutoff ties, clusters that separate in the lower three key bytes, padded rows and page-table expansion, registered for CUDA and AMD.

The v2 changes are removed from this PR so the legacy/v1 fix can be reviewed and merged independently. I will submit the v2 changes in a separate PR.

Accuracy Tests

On RTX PRO 6000 Blackwell (SM120), a combined build passed 12 KPool JIT tests/38 subtests and 7 focused legacy CUDA AOT cases, including oversized coarse bins. One ROCm-only case was skipped. Those results predate the rebase onto main's KPool implementation.

On RTX PRO 6000 Blackwell (SM120), torch 2.14.1+cu130, Triton 3.8.0, at this branch, test_topk_v1.py passed (15 tests) and test_dsa_kpool_topk_transform.py passed against main's KPool kernel (12 tests, 38 subtests). Against main's unpatched topk_v1.cuh, 9 of the 15 v1 tests fail (the overflow cases). The AOT header was compiled for sm_120 in a standalone harness and matched a CPU reference on random rows, single-bin rows above the stash, and strictly higher values plus cutoff ties. The AOT test matrix and ROCm tests have not been run.

Speed Tests and Profiling

Rows whose coarse threshold bin exceeds the stash take six row scans; other rows are unchanged. No speed benchmark was run.

Checklist

  • Format changed code.
  • Run regression tests described above.

Developed with AI assistance.


CI States

Latest PR Test (Base): ✅ Run #38096800282
Latest PR Test (Extra): ❌ Run #38096800233
Latest PR Test (AMD ROCm 10): ❌ Run #38096800271

@zRzRzRzRzRzRzR zRzRzRzRzRzRzR left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue received, following and test this.

@zRzRzRzRzRzRzR

Copy link
Copy Markdown
Contributor

This PR seems to have only fixed its copy in the kpool JIT kernel. There are several other copies of the same code affect all dsa model.

@ormandj

ormandj commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor Author

This PR seems to have only fixed its copy in the kpool JIT kernel. There are several other copies of the same code affect all dsa model.

I was working on addressing issues with GLM 5.3 Flash in my SGlang build that lead to crashes directly, but was keeping the breadth smaller to minimize review burden. It sounds like you desire the full and complete fix, so let me evaluate that. Thank you for the feedback.

@ormandj
ormandj force-pushed the pr/kpool-topk-fix-main branch from 1436d19 to 7ad39a0 Compare September 4, 2026 01:08
@ormandj ormandj changed the title Fix kpool top-k selection for oversized radix bins Fix oversized DSA radix bins across top-k kernels Sep 4, 2026
@ormandj

ormandj commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor Author

@zRzRzRzRzRzRzR Updated the branch in response to the scope issue you identified.

  • The kpool JIT, DeepSeek-V4 JIT, legacy AOT, and DeepSeek-V4 AOT copies now use matching package-local shared selectors; the JIT and AOT packages remain independently buildable.
  • DeepSeek-V4 v2 register, streaming, and clustered selection now uses the same full-row FP32 fallback when its tie buffer overflows.
  • The legacy fallback now includes the exact-capacity boundary after an isolated GPU test failed at exactly 4,096 threshold-bin candidates.
  • Cumulative histogram reads are barrier-separated from storage reuse, and all four legacy wrappers synchronize before consuming the selected indices.
  • Coverage now exercises every affected entry point, including oversized and exact-capacity bins, negative values, input permutations, higher values already emitted, full-key ties, low-byte separation, offsets, idle rows, the AOT decode path, and raw-index and page-table cluster staging.
  • Existing AOT and v2 comparisons now require exact selected-value multisets, while accepting index swaps among equal scores. Stale v2 dispatch cases were replaced with cases for the current 32,768 small-batch cluster floor and 512-row cluster limit; 262,144-token coverage remains at batch 40.

The branch is rebased onto SGLang main at 9ed2721c6d. A downstream integration candidate carrying the selection changes passed the kpool JIT, focused DeepSeek-V4 JIT, and source-built SM120 AOT runs on one Blackwell (sm_120) GPU. CUDA and ROCm CI for this exact commit are pending; ROCm was not compiled or run locally.

@zRzRzRzRzRzRzR

zRzRzRzRzRzRzR commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

The fix in this PR is correct, but it costs quite a bit of performance as it stands.

Model output quality before vs. after the fix doesn't seem to differ much on long text (at least when measured end-to-end, e.g. by benchmark scores such as GQPA, need more than 32K context) — an interesting finding.

Maybe there's a better way to fix this? cc @JustinTong0323

@JustinTong0323

Copy link
Copy Markdown
Collaborator

The fix in this PR is correct, but it costs quite a bit of performance as it stands.

Model output quality before vs. after the fix doesn't seem to differ much on long text (at least when measured end-to-end, e.g. by benchmark scores such as GQPA, need more than 32K context) — an interesting finding.

Maybe there's a better way to fix this? cc @JustinTong0323

Kernel level there is some perf regression but not obvious in e2e, so I think it shall be ok.

@JustinTong0323

Copy link
Copy Markdown
Collaborator

@DarkSharpness to review :)

@DarkSharpness

Copy link
Copy Markdown
Collaborator

@ormandj could you please rebase to the latest main branch?

@Wendy-Hamlet

Copy link
Copy Markdown

Real-model data point for the legacy (v1) part, since it just got a review: we hit this truncation on DeepSeek-V4-Flash with real long prompts, not only synthetic inputs.

Where. On c9ceab34cf (2026-07-06), DSV4 prefill with raw indices went through topk_transform_512 in topk_v1.cuh. Its threshold-bin candidates go into a shared-memory buffer of SMEM_INPUT_SIZE = 8192; anything past pos < SMEM_INPUT_SIZE is dropped, and which ones are dropped depends on atomicAdd order. The same code is still on main (3ed56a341b). There, the default DSV4 prefill path now uses v2 after #39648, so this concerns SGLANG_OPT_USE_TOPK_V2=0 and releases that predate #39648.

How often it triggers. We captured the indexer scores (fp16) for 22 prompts of about 128K tokens (RULER vt / cwe / qa_1 and InfiniteBench longbook_choice_eng), on all 21 CSA layers and 1,536 rows per prompt, for 569,856 (row, layer) pairs in total. Replaying stage 1 bit-exactly from those scores:

  • 448 pairs had a threshold bin larger than 8192 while slots were still open. The largest bin held 28,050 candidates.
  • All 448 are in the 4 RULER vt prompts (variable tracking, highly repetitive text), mostly at L8 with some at L4 and L30.
  • The other 18 prompts peak at 5,391 candidates, so they never truncate.

Whether the selection is actually wrong. We counted a row as wrong when a non-selected candidate scores strictly higher than the lowest selected one (fp16 rounding is monotone, so this proves the top-k is inexact):

  • 444 of the 448 truncated rows are wrong; per prompt, the median number of misplaced candidates per row ranges from 4 to 241.
  • Rows that were not truncated have 0 wrong selections (all window rows of the 4 vt prompts, plus one cwe and one longbook prompt).

On GPU (H100, that tree's kernel, 50 repeats).

  • Synthetic inputs with an oversized bin: 64/64 rows are inexact, and in a few rows the selected set also changes between repeats.
  • Replaying real vt score rows: every truncated row is inexact (L4 21, L8 76, L30 5 rows).

So on real DSV4 inputs the legacy path is exact whenever it does not truncate, and truncation happens only on repetitive long inputs, where it nearly always returns a wrong set. We have not measured the end-to-end quality impact.

@ormandj
ormandj force-pushed the pr/kpool-topk-fix-main branch from 759070d to 1eebe42 Compare September 30, 2026 00:45
@ormandj ormandj changed the title Fix incorrect sparse-attention top-k selection when candidate bins overflow Fix overflow in legacy and v1 DSA top-k selectors Sep 30, 2026
@ormandj

ormandj commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

@DarkSharpness, replying to your review:

The legacy topk part LGTM.

The v2 changes, including the coarse-bound and all-equal paths, are removed from this PR. The diff now covers the four legacy/v1 selectors so those fixes can be reviewed and merged independently. I will submit the v2 changes in a separate PR.

@ormandj

ormandj commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

@zRzRzRzRzRzRzR, replying to your selector-scope comment:

There are several other copies of the same code affect all dsa model.

The patch covers all four legacy/v1 copies: KPool JIT, DeepSeek-V4 JIT v1, legacy CUDA AOT and DeepSeek-V4 AOT. The separate QSA fast_topk copy on newer main is outside this patch.

@ormandj

ormandj commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

@DarkSharpness, replying to your rebase request:

@ormandj could you please rebase to the latest main branch?

Rebased onto main c7be3e935b5.

@ormandj

ormandj commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

@AranKomat, replying to your AMD test-coverage question:

Are there particular AMD boundary/graph cases you would like added to the existing regression suite?

The suite compares selected-value multisets and allows index swaps among equal scores. It includes oversized and exact-capacity bins, ties, offsets and changed inputs. The retained ROCm AOT cases have not been rerun on gfx942; the current GPU results are from SM120.

@ormandj

ormandj commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Updated the code and PR body. The diff now covers the four legacy/v1 top-k selectors. The v2 changes are removed so the legacy/v1 fix can be reviewed and merged independently; I will submit v2 in a separate PR. Normal bins retain their existing path, while overflow handling uses exact FP32 cutoff refinement and a final row scan. Existing ROCm tests are retained. The body lists current SM120 test results and the remaining selector/platform coverage.

@DarkSharpness DarkSharpness left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some final comments. Please also fix lint.

@ormandj
ormandj force-pushed the pr/kpool-topk-fix-main branch from 1eebe42 to 9963cfb Compare October 7, 2026 05:43
ormandj and others added 4 commits October 10, 2026 01:21
KPool already rescans overflowing bins on main; keep its implementation and add regression coverage for it, the JIT v1 selector and the AOT selectors.

Co-authored-by: Benjamin Oldenburg <benjamin.oldenburg@ordis.co.th>
Main renamed fast_kpool_topk_transform_fused's pool_size to kpool (sgl-project#42671). The overflow cases pass against main's exact rescan.
Each refinement pass also emits the values the previous key byte placed strictly above the cutoff, so an oversized coarse bin takes five input passes instead of six. Use the JIT device macro, drop the redundant fp16 include and trailing barrier, and shorten the comments.
@ormandj
ormandj force-pushed the pr/kpool-topk-fix-main branch from 9963cfb to e28fa20 Compare October 10, 2026 23:57
@DarkSharpness

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 11, 2026
@JustinTong0323

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci extra

@github-actions github-actions Bot added the run-ci-extra CI: also run the extra suite (requires run-ci) label Oct 12, 2026
@ormandj ormandj mentioned this pull request Oct 13, 2026
7 of 12 tasks
@ormandj

ormandj commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

The 26 failed checks on e28fa20528 are outside this PR's files:

  • The NVIDIA extra suites stopped by fail-fast after the extra-a-test-1-gpu-small / extra-a-test-1-gpu-small (1) job, where test/registered/scheduler/test_priority_scheduling.py failed an abort status assertion.
  • AMD: kv_canary/test_self_e2e_pr_25015.py, model_loading/test_load_weights_from_remote_instance.py, attention/test_kda_kernels.py, observability/test_tracing.py, kernels/ops/attention/test_fused_kda_conv_recurrent_verify.py, kernels/ops/attention/dsv4/test_compact_attention_hip.py and amd/disaggregation/test_disaggregation_pp.py.
  • NPU: npu/basic_function/HiCache/test_npu_hicache_mha.py and npu/basic_function/pd_disaggregation/test_npu_pd_disaggregation.py.
  • Intel CPU: unit/server_args/test_server_args.py.

The PR's tests passed in jit-kernel-unit-test-1-gpu-large: test_topk_v1.py and test_dsa_kpool_topk_transform.py.

@BBuf
BBuf merged commit c5bf3aa into sgl-project:main Oct 15, 2026
352 of 425 checks passed
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
…k-274, wanted_stack LANDED marks, CAVEATS docs-only ledger items

(A) Header: STACK line 260->274; three wave lines (+spec-mamba-2026-10-28
(pr45491+pr45435+pr45472+pr45670), +admission-2026-10-28(pr46272*+
pr43484*+pr46496*+pr46643*+pr45607*+pr45789*), +parsers-2026-10-28
(pr44124+pr45631+pr45855+pr46167)); tip line -> 2714457; updated
261-274 headline block (entry-119 SUPERSEDED by upstream sgl-project#45435/sgl-project#45472;
classic-pool mamba admission gate w/ upstream ImportError CAUGHT;
UNCONDITIONAL latch reset = TTFT watch; coder-parser union; logits
trilogy as CONTRACT UNIT; 14 picks, 0 integrator conflicts).

(B) wanted_stack.txt: FAM-A5 + FAM-B234 blocks deleted -> LANDED records
(chain shas, interlock audit, supersession + behavior-change pointers
into entries 261-270); SECTION-B sgl-project#44124/trilogy blocks -> LANDED;
sgl-project#37958/sgl-project#45559 block -> DONE (verdict now in CAVEATS).

(C) DOCS-ONLY LEDGER ITEMS: CAVEATS gains the sgl-project#37958/sgl-project#45559 VERIFIED-
INERT bullet (two independent hard gates, coarse-barrier fallback,
do-not-re-litigate-at-merge-sgl-project#5 line, revisit condition = dropping NEXTN
or HiCache) and the dated C0b/C0c corrections (sgl-project#44030 obsolete rationale,
sgl-project#44858 REVERSED by sgl-project#45289 [ported-pending wt-moe], sgl-project#44115 permanent N/A,
10-28 open-status re-verification list, no-pin-moves note: 0.7.2/0.4.9
floor unchanged) with the C0a tick (sgl-project#37625/sgl-project#37941/sgl-project#39727 still
outstanding; only sgl-project#43927/sgl-project#44849 partially satisfied). C0b/C0c/C8 marked
DONE in wanted_stack; C0a stays open.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek high priority jit-kernel run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) sgl-kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants