Skip to content

dsv4.1: Top-k kernels and candidate selection - #39648

Merged
hnyls2002 merged 18 commits into
mainfrom
dsv4.1-topk
Sep 16, 2026
Merged

hnyls2002 merged 18 commits into
mainfrom
dsv4.1-topk

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Sep 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Add the DeepSeek-V4.1 sparse-indexer kernels that sit beside Top-k v2: a bf16 small-row top-k, the two-level indexer's block-max keys and sorted block table, and a cluster-occupancy probe.
  • Rework Top-k v2's edge handling and cluster dispatch (details below); the existing suite still passes and two new regression cases pin the fixed -inf tie behaviour.

New kernels

  • topk_bf16_small (topk_transform_bf16_small): bf16 top-k for rows of at most 16384 scores, two radix passes over the raw bf16 bytes, fused with the page-table transform; adapted from DeepSelect's init-window select.
  • block_amax (amax8_varlen): level-one keys of the two-level indexer, the max of each 8-score block with the row's newest block forced to +inf.
  • candidate_block_table (sort_candidate_blocks): counting sort of a row's selected blocks over a bitmap plus the page transform, in place.
  • jit/utils/occupancy.py (get_max_active_clusters): asks cudaOccupancyMaxActiveClusters how many clusters of a given width fit at a given occupancy; sizes the Top-k v2 persistent pool at JIT time.
  • attention/dsv4/candidate_table.py hosts the two indexer helpers; topk.py keeps only the top-k entry points.

Top-k v2 changes

  • Padding uses NaN instead of -FLT_MAX, so inactive entries never displace valid -inf scores (test_topk_v2_negative_infinity_ties, test_topk_v2_ragged_negative_infinity); the padding bin is taken from extract_coarse_bin(padding_value()), so the histogram correction does not depend on the platform's NaN encoding.
  • coarse_bin_lower_bound steps the midpoint up on odd-significand bins (fp32 -> fp16 rounds to nearest even) and returns +/-inf at the key-space edges.
  • Cluster path: TopKCluster<N> for N = 8 and 16, inputs interleaved across ranks, rank 0 owns the histogram and threshold behind release/acquire cluster barriers; the persistent-pool size and the cluster-16 small-batch cap come from the occupancy probe, with per-arch defaults as fallback; topk_plan_cluster carries per-arch candidate tables.
  • SUPPORT_CLUSTER limits the cluster path to sm90/sm100; sm120 and ROCm take the streaming path.
  • plan_topk_v2(static_threshold): a negative value selects the heuristic (was 0); no in-tree caller passes the argument.
  • The [Bug] Nvidia compiler crashes with segmentation fault when trying to serve DeepkSeek v4 #32830 mitigation (__builtin_assume in the small-batch cluster epilogue) is dropped: the restructured epilogue compiles with nvcc 13.1 through 13.4 for sm_90a, while the previous code still segfaults cicc without it.
  • TieHandleSmem no longer over-aligns its counters; topk_v2_plan_is_written replaces the test kit's use of the private JIT module.

Verification

  • test_topk_v2.py: 288 cases pass on SM90 and SM100.
  • The new kernels were checked against torch references (block max with +inf newest block, sorted block table with page mapping, bf16 top-k); they have no registered tests of their own and are covered by the model's end-to-end tests.
  • bench_topk.py seeds each configuration and adds page_size to the paged sweep; not re-run here.

CI States

Latest PR Test (Base): 🚫 Run #35064790417
Latest PR Test (Extra): ❌ Run #35064790112
Latest PR Test (AMD ROCm 10): 🚫 Run #35064790372

@hnyls2002
hnyls2002 added this pull request to stack #39649 September 15, 2026 21:30
@hnyls2002
hnyls2002 removed this pull request from stack #39649 September 15, 2026 21:50
@hnyls2002
hnyls2002 added this pull request to stack #39654 September 15, 2026 21:56
@hnyls2002
hnyls2002 removed this pull request from stack #39654 September 15, 2026 22:10
@hnyls2002
hnyls2002 added this pull request to stack #39658 September 15, 2026 22:10
@hnyls2002
hnyls2002 removed this pull request from stack #39658 September 15, 2026 22:31
@hnyls2002
hnyls2002 added this pull request to stack #39667 September 15, 2026 22:32
@hnyls2002
hnyls2002 removed this pull request from stack #39667 September 15, 2026 22:43
@hnyls2002
hnyls2002 added this pull request to stack #39669 September 15, 2026 22:43
@hnyls2002
hnyls2002 removed this pull request from stack #39669 September 15, 2026 22:56
@hnyls2002
hnyls2002 added this pull request to stack #39672 September 15, 2026 22:58
@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/rerun-test test_topk_v2.py

@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test_topk_v2.py:

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/kernels/ops/attention/test_topk_v2.py

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Sep 16, 2026
@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/rerun-test test/registered/kernels/ops/attention/test_dsa_indexer.py test/registered/kernel/attention/test_dsa_metadata_replay.py test/registered/kernels/ops/attention/test_dsv32_indexer_fusion.py test/registered/unit/layers/test_dsv4_nonpaged_indexer.py

@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/kernels/ops/attention/test_dsa_indexer.py test/registered/kernel/attention/test_dsa_metadata_replay.py test/registered/kernels/ops/attention/test_dsv32_indexer_fusion.py test/registered/unit/layers/test_dsv4_nonpaged_indexer.py:

🚀 1-gpu-h100 (3 tests): ✅ View workflow run

cd test/ && python3 registered/kernels/ops/attention/test_dsa_indexer.py
cd test/ && python3 registered/kernel/attention/test_dsa_metadata_replay.py
cd test/ && python3 registered/kernels/ops/attention/test_dsv32_indexer_fusion.py

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/layers/test_dsv4_nonpaged_indexer.py

@hnyls2002
hnyls2002 merged commit faaff1e into main Sep 16, 2026
91 of 116 checks passed
@hnyls2002
hnyls2002 deleted the dsv4.1-topk branch September 16, 2026 06:54
EricKing626 added a commit to EricKing626/sglang that referenced this pull request Sep 16, 2026
Ports the packed-row (row_starts / row_to_batch) support onto upstream's
new dsv4.1 top-k design from sgl-project#39648:

* page_table / page_bits left TopKProblem and moved into PageTransform, so
  the row -> table-row mapping now happens in get_transform() instead of
  patching problem.page_table.
* The 16-byte head residue is now expressed the way upstream's ragged kernel
  expresses it: masked head columns are written as impl::padding_value() and
  counted through the new TopKProblem::input_start, instead of -FLT_MAX.
* topk_main_kernel keeps row_seq_len (the row's real length) for the trivial
  and cluster dispatch decisions, since the residue only widens the read
  window.
* topk.py / test_topk_v2.py docstrings take upstream's rewritten paragraphs
  and keep the packed-row paragraphs on top.
xiaofei-zheng added a commit to xiaofei-zheng/sglang that referenced this pull request Sep 16, 2026
The coarse bin comes from the top bits of the fp16 cast, so one bin spans a
quarter-binade. The collect pass stages threshold-bin candidates under
`if (pos < kMaxNumTie)` and phase 4 clips `min(count_eq, kMaxNumTie)`, so an
overflowing bin leaves `handle_tie` ranking an arrival-order subset and the
kernel returns a wrong selected-value multiset with no diagnostic.

Found while profiling GLM-5.2 decode on MI355X: device-side counters over an
agentic serving run enter that path on ~0.04% of rows across all 8 ranks, with
a largest observed bin of 4139 against the 2048 cap and none of those rows
bit-identical. sgl-project#35257 reports a related problem from the CUDA side. sgl-project#39648
restructured this kernel but left all four truncation sites unchanged.

Refine the overflowing bin on the exact key instead: up to three radix rounds
over the order-preserving key from `extract_exact_bin`, 12 / 12 / 8 bits wide,
exiting once the refined set fits the staging buffer or once all 32 bits are
consumed, where the survivors are bit-identical and truncating them is exact.
Each round emits its own "above" set while building the next round's histogram,
so the refinement costs at most four input passes. The histogram overlays
`tie_values`, which this path re-derives rather than trusting, so LDS is
unchanged at 26928 B. The fast path branches on `count_eq > kMaxNumTie` and is
otherwise untouched.

Also pin the ROCm `__launch_bounds__` occupancy argument to the physical wave
floor: HIP reads it as waves per SIMD where CUDA reads it as blocks per SM, so
`kOccupancy` asked ROCm for 2 waves/SIMD, below the 4 a 1024-thread block
already forces. The CUDA spelling stays because
`topk_small_batch_cluster_kernel` shadows `kOccupancy` with a template
parameter.

Testing. Pre-existing cases draw from `torch.randn` and tolerate
MAX_PERMIT_ERROR = 5, so they never reach the overflow. Added narrow,
narrow_bin, tiny and two_values distributions that collapse a row into one
coarse bin, plus overflow with DUAL_OUTPUT and overflow on an unaligned ragged
window. On the kernel on main those 14 cases fail while the 2 all_equal
controls pass, which pins the fault to truncation rather than to the binning.
All 304 pass here.

MI355X: the hot path is unchanged (5.324 vs 5.484 us at 6x8192, 12.094 vs
12.267 at 4x32768); rows that overflow now pay for work previously skipped by
returning a wrong answer (narrow_bin 17.051 -> 21.684, two_values 36.194 ->
118.811 at 4x32768). 57 -> 79 VGPRs, zero spills.

Known gap: the CUDA-only `TopKCluster` path still truncates. Refining there
needs cluster-wide rather than block-local histogram and emit counters.

Co-authored-by: Cursor <cursoragent@cursor.com>
Leoyzen added a commit to Leoyzen/sglang that referenced this pull request Sep 17, 2026
Re-port of sgl-project#37625 onto post-sgl-project#39648 code. The original PR
was authored against a pre-sgl-project#39648 base whose Smem spelled the tie buffer
'tie.values'/'tie.handle' and whose Phase-4 locals were 'above_count'/
'equal_count'; sgl-project#39648 renamed those to 'tie_values'/'tie_handle' and
'count_gt'/'count_eq'. The overflow fallback is adapted to the current
names, and 'handle_oversized_bin' now takes the already-published v_lo/v_hi
boundaries instead of a raw threshold_bin, because the current find_threshold
publishes boundaries rather than the bin index.

The fallback's membership test is written as 'val >= v_lo && val < v_hi'
rather than the exclusion form: the ragged entry point masks the <= 3 columns
ahead of a window with NaN padding, which the exclusion form would admit.

Co-authored-by: David Orman <ormandj@corenode.com>
Co-authored-by: Benjamin Oldenburg <benjamin.oldenburg@ordis.th>
Leoyzen added a commit to Leoyzen/sglang that referenced this pull request Sep 17, 2026
Re-port of sgl-project#37625 onto post-sgl-project#39648 code. The original PR
was authored against a pre-sgl-project#39648 base whose Smem spelled the tie buffer
'tie.values'/'tie.handle' and whose Phase-4 locals were 'above_count'/
'equal_count'; sgl-project#39648 renamed those to 'tie_values'/'tie_handle' and
'count_gt'/'count_eq'. The overflow fallback is adapted to the current
names, and 'handle_oversized_bin' now takes the already-published v_lo/v_hi
boundaries instead of a raw threshold_bin, because the current find_threshold
publishes boundaries rather than the bin index.

The fallback's membership test is written as 'val >= v_lo && val < v_hi'
rather than the exclusion form: the ragged entry point masks the <= 3 columns
ahead of a window with NaN padding, which the exclusion form would admit.

Co-authored-by: David Orman <ormandj@corenode.com>
Co-authored-by: Benjamin Oldenburg <benjamin.oldenburg@ordis.th>
Leoyzen added a commit to Leoyzen/sglang that referenced this pull request Sep 18, 2026
Re-port of sgl-project#37625 onto post-sgl-project#39648 code. The original PR
was authored against a pre-sgl-project#39648 base whose Smem spelled the tie buffer
'tie.values'/'tie.handle' and whose Phase-4 locals were 'above_count'/
'equal_count'; sgl-project#39648 renamed those to 'tie_values'/'tie_handle' and
'count_gt'/'count_eq'. The overflow fallback is adapted to the current
names, and 'handle_oversized_bin' now takes the already-published v_lo/v_hi
boundaries instead of a raw threshold_bin, because the current find_threshold
publishes boundaries rather than the bin index.

The fallback's membership test is written as 'val >= v_lo && val < v_hi'
rather than the exclusion form: the ragged entry point masks the <= 3 columns
ahead of a window with NaN padding, which the exclusion form would admit.

Co-authored-by: David Orman <ormandj@corenode.com>
Co-authored-by: Benjamin Oldenburg <benjamin.oldenburg@ordis.th>
xiaofei-zheng added a commit to xiaofei-zheng/sglang that referenced this pull request Sep 21, 2026
The coarse bin comes from the top bits of the fp16 cast, so one bin spans a
quarter-binade. The collect pass stages threshold-bin candidates under
`if (pos < kMaxNumTie)` and phase 4 clips `min(count_eq, kMaxNumTie)`, so an
overflowing bin leaves `handle_tie` ranking an arrival-order subset and the
kernel returns a wrong selected-value multiset with no diagnostic.

Found while profiling GLM-5.2 decode on MI355X: device-side counters over an
agentic serving run enter that path on ~0.04% of rows across all 8 ranks, with
a largest observed bin of 4139 against the 2048 cap and none of those rows
bit-identical. sgl-project#35257 reports a related problem from the CUDA side. sgl-project#39648
restructured this kernel but left all four truncation sites unchanged.

Refine the overflowing bin on the exact key instead: up to three radix rounds
over the order-preserving key from `extract_exact_bin`, 12 / 12 / 8 bits wide,
exiting once the refined set fits the staging buffer or once all 32 bits are
consumed, where the survivors are bit-identical and truncating them is exact.
Each round emits its own "above" set while building the next round's histogram,
so the refinement costs at most four input passes. The histogram overlays
`tie_values`, which this path re-derives rather than trusting, so LDS is
unchanged at 26928 B. The fast path branches on `count_eq > kMaxNumTie` and is
otherwise untouched.

Also pin the ROCm `__launch_bounds__` occupancy argument to the physical wave
floor: HIP reads it as waves per SIMD where CUDA reads it as blocks per SM, so
`kOccupancy` asked ROCm for 2 waves/SIMD, below the 4 a 1024-thread block
already forces. The CUDA spelling stays because
`topk_small_batch_cluster_kernel` shadows `kOccupancy` with a template
parameter.

Testing. Pre-existing cases draw from `torch.randn` and tolerate
MAX_PERMIT_ERROR = 5, so they never reach the overflow. Added narrow,
narrow_bin, tiny and two_values distributions that collapse a row into one
coarse bin, plus overflow with DUAL_OUTPUT and overflow on an unaligned ragged
window. On the kernel on main those 14 cases fail while the 2 all_equal
controls pass, which pins the fault to truncation rather than to the binning.
All 304 pass here.

MI355X: the hot path is unchanged (5.324 vs 5.484 us at 6x8192, 12.094 vs
12.267 at 4x32768); rows that overflow now pay for work previously skipped by
returning a wrong answer (narrow_bin 17.051 -> 21.684, two_values 36.194 ->
118.811 at 4x32768). 57 -> 79 VGPRs, zero spills.

Known gap: the CUDA-only `TopKCluster` path still truncates. Refining there
needs cluster-wide rather than block-local histogram and emit counters.

Co-authored-by: Cursor <cursoragent@cursor.com>
xiaofei-zheng added a commit to xiaofei-zheng/sglang that referenced this pull request Sep 27, 2026
The coarse bin comes from the top bits of the fp16 cast, so one bin spans a
quarter-binade. The collect pass stages threshold-bin candidates under
`if (pos < kMaxNumTie)` and phase 4 clips `min(count_eq, kMaxNumTie)`, so an
overflowing bin leaves `handle_tie` ranking an arrival-order subset and the
kernel returns a wrong selected-value multiset with no diagnostic.

Found while profiling GLM-5.2 decode on MI355X: device-side counters over an
agentic serving run enter that path on ~0.04% of rows across all 8 ranks, with
a largest observed bin of 4139 against the 2048 cap and none of those rows
bit-identical. sgl-project#35257 reports a related problem from the CUDA side. sgl-project#39648
restructured this kernel but left all four truncation sites unchanged.

Refine the overflowing bin on the exact key instead: up to three radix rounds
over the order-preserving key from `extract_exact_bin`, 12 / 12 / 8 bits wide,
exiting once the refined set fits the staging buffer or once all 32 bits are
consumed, where the survivors are bit-identical and truncating them is exact.
Each round emits its own "above" set while building the next round's histogram,
so the refinement costs at most four input passes. The histogram overlays
`tie_values`, which this path re-derives rather than trusting, so LDS is
unchanged at 26928 B. The fast path branches on `count_eq > kMaxNumTie` and is
otherwise untouched.

Also pin the ROCm `__launch_bounds__` occupancy argument to the physical wave
floor: HIP reads it as waves per SIMD where CUDA reads it as blocks per SM, so
`kOccupancy` asked ROCm for 2 waves/SIMD, below the 4 a 1024-thread block
already forces. The CUDA spelling stays because
`topk_small_batch_cluster_kernel` shadows `kOccupancy` with a template
parameter.

Testing. Pre-existing cases draw from `torch.randn` and tolerate
MAX_PERMIT_ERROR = 5, so they never reach the overflow. Added narrow,
narrow_bin, tiny and two_values distributions that collapse a row into one
coarse bin, plus overflow with DUAL_OUTPUT and overflow on an unaligned ragged
window. On the kernel on main those 14 cases fail while the 2 all_equal
controls pass, which pins the fault to truncation rather than to the binning.
All 304 pass here.

MI355X: the hot path is unchanged (5.324 vs 5.484 us at 6x8192, 12.094 vs
12.267 at 4x32768); rows that overflow now pay for work previously skipped by
returning a wrong answer (narrow_bin 17.051 -> 21.684, two_values 36.194 ->
118.811 at 4x32768). 57 -> 79 VGPRs, zero spills.

Known gap: the CUDA-only `TopKCluster` path still truncates. Refining there
needs cluster-wide rather than block-local histogram and emit counters.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jit-kernel run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants