Repository navigation
dsv4.1: Top-k kernels and candidate selection - #39648
Merged
Merged
Conversation
hnyls2002
requested review from
BBuf,
DarkSharpness,
HaiShaw,
HydraQYH,
celve and
yuan-luo
September 15, 2026 21:27
This was referenced Sep 15, 2026
This was referenced Sep 15, 2026
hnyls2002
force-pushed
the
dsv4.1-topk
branch
from
September 15, 2026 22:22
243b7e2 to
64087a9
Compare
hnyls2002
force-pushed
the
dsv4.1-kernels
branch
from
September 15, 2026 23:37
474d4bd to
afc6a1f
Compare
hnyls2002
force-pushed
the
dsv4.1-topk
branch
from
September 15, 2026 23:37
64087a9 to
14e300f
Compare
hnyls2002
force-pushed
the
dsv4.1-kernels
branch
from
September 16, 2026 00:27
afc6a1f to
8edf025
Compare
hnyls2002
force-pushed
the
dsv4.1-topk
branch
from
September 16, 2026 00:27
14e300f to
8b5f7c1
Compare
hnyls2002
force-pushed
the
dsv4.1-kernels
branch
from
September 16, 2026 01:12
8edf025 to
1cdc9f1
Compare
Collaborator
Author
|
/rerun-test test_topk_v2.py |
Contributor
|
Results for 🚀 |
Collaborator
Author
|
/tag-and-rerun-ci |
Collaborator
Author
|
/rerun-test test/registered/kernels/ops/attention/test_dsa_indexer.py test/registered/kernel/attention/test_dsa_metadata_replay.py test/registered/kernels/ops/attention/test_dsv32_indexer_fusion.py test/registered/unit/layers/test_dsv4_nonpaged_indexer.py |
Contributor
|
Results for 🚀 🚀 |
EricKing626
added a commit
to EricKing626/sglang
that referenced
this pull request
Sep 16, 2026
Ports the packed-row (row_starts / row_to_batch) support onto upstream's new dsv4.1 top-k design from sgl-project#39648: * page_table / page_bits left TopKProblem and moved into PageTransform, so the row -> table-row mapping now happens in get_transform() instead of patching problem.page_table. * The 16-byte head residue is now expressed the way upstream's ragged kernel expresses it: masked head columns are written as impl::padding_value() and counted through the new TopKProblem::input_start, instead of -FLT_MAX. * topk_main_kernel keeps row_seq_len (the row's real length) for the trivial and cluster dispatch decisions, since the residue only widens the read window. * topk.py / test_topk_v2.py docstrings take upstream's rewritten paragraphs and keep the packed-row paragraphs on top.
4 tasks done
xiaofei-zheng
added a commit
to xiaofei-zheng/sglang
that referenced
this pull request
Sep 16, 2026
The coarse bin comes from the top bits of the fp16 cast, so one bin spans a quarter-binade. The collect pass stages threshold-bin candidates under `if (pos < kMaxNumTie)` and phase 4 clips `min(count_eq, kMaxNumTie)`, so an overflowing bin leaves `handle_tie` ranking an arrival-order subset and the kernel returns a wrong selected-value multiset with no diagnostic. Found while profiling GLM-5.2 decode on MI355X: device-side counters over an agentic serving run enter that path on ~0.04% of rows across all 8 ranks, with a largest observed bin of 4139 against the 2048 cap and none of those rows bit-identical. sgl-project#35257 reports a related problem from the CUDA side. sgl-project#39648 restructured this kernel but left all four truncation sites unchanged. Refine the overflowing bin on the exact key instead: up to three radix rounds over the order-preserving key from `extract_exact_bin`, 12 / 12 / 8 bits wide, exiting once the refined set fits the staging buffer or once all 32 bits are consumed, where the survivors are bit-identical and truncating them is exact. Each round emits its own "above" set while building the next round's histogram, so the refinement costs at most four input passes. The histogram overlays `tie_values`, which this path re-derives rather than trusting, so LDS is unchanged at 26928 B. The fast path branches on `count_eq > kMaxNumTie` and is otherwise untouched. Also pin the ROCm `__launch_bounds__` occupancy argument to the physical wave floor: HIP reads it as waves per SIMD where CUDA reads it as blocks per SM, so `kOccupancy` asked ROCm for 2 waves/SIMD, below the 4 a 1024-thread block already forces. The CUDA spelling stays because `topk_small_batch_cluster_kernel` shadows `kOccupancy` with a template parameter. Testing. Pre-existing cases draw from `torch.randn` and tolerate MAX_PERMIT_ERROR = 5, so they never reach the overflow. Added narrow, narrow_bin, tiny and two_values distributions that collapse a row into one coarse bin, plus overflow with DUAL_OUTPUT and overflow on an unaligned ragged window. On the kernel on main those 14 cases fail while the 2 all_equal controls pass, which pins the fault to truncation rather than to the binning. All 304 pass here. MI355X: the hot path is unchanged (5.324 vs 5.484 us at 6x8192, 12.094 vs 12.267 at 4x32768); rows that overflow now pay for work previously skipped by returning a wrong answer (narrow_bin 17.051 -> 21.684, two_values 36.194 -> 118.811 at 4x32768). 57 -> 79 VGPRs, zero spills. Known gap: the CUDA-only `TopKCluster` path still truncates. Refining there needs cluster-wide rather than block-local histogram and emit counters. Co-authored-by: Cursor <cursoragent@cursor.com>
Leoyzen
added a commit
to Leoyzen/sglang
that referenced
this pull request
Sep 17, 2026
Re-port of sgl-project#37625 onto post-sgl-project#39648 code. The original PR was authored against a pre-sgl-project#39648 base whose Smem spelled the tie buffer 'tie.values'/'tie.handle' and whose Phase-4 locals were 'above_count'/ 'equal_count'; sgl-project#39648 renamed those to 'tie_values'/'tie_handle' and 'count_gt'/'count_eq'. The overflow fallback is adapted to the current names, and 'handle_oversized_bin' now takes the already-published v_lo/v_hi boundaries instead of a raw threshold_bin, because the current find_threshold publishes boundaries rather than the bin index. The fallback's membership test is written as 'val >= v_lo && val < v_hi' rather than the exclusion form: the ragged entry point masks the <= 3 columns ahead of a window with NaN padding, which the exclusion form would admit. Co-authored-by: David Orman <ormandj@corenode.com> Co-authored-by: Benjamin Oldenburg <benjamin.oldenburg@ordis.th>
Leoyzen
added a commit
to Leoyzen/sglang
that referenced
this pull request
Sep 17, 2026
Re-port of sgl-project#37625 onto post-sgl-project#39648 code. The original PR was authored against a pre-sgl-project#39648 base whose Smem spelled the tie buffer 'tie.values'/'tie.handle' and whose Phase-4 locals were 'above_count'/ 'equal_count'; sgl-project#39648 renamed those to 'tie_values'/'tie_handle' and 'count_gt'/'count_eq'. The overflow fallback is adapted to the current names, and 'handle_oversized_bin' now takes the already-published v_lo/v_hi boundaries instead of a raw threshold_bin, because the current find_threshold publishes boundaries rather than the bin index. The fallback's membership test is written as 'val >= v_lo && val < v_hi' rather than the exclusion form: the ragged entry point masks the <= 3 columns ahead of a window with NaN padding, which the exclusion form would admit. Co-authored-by: David Orman <ormandj@corenode.com> Co-authored-by: Benjamin Oldenburg <benjamin.oldenburg@ordis.th>
This was referenced Sep 17, 2026
Leoyzen
added a commit
to Leoyzen/sglang
that referenced
this pull request
Sep 18, 2026
Re-port of sgl-project#37625 onto post-sgl-project#39648 code. The original PR was authored against a pre-sgl-project#39648 base whose Smem spelled the tie buffer 'tie.values'/'tie.handle' and whose Phase-4 locals were 'above_count'/ 'equal_count'; sgl-project#39648 renamed those to 'tie_values'/'tie_handle' and 'count_gt'/'count_eq'. The overflow fallback is adapted to the current names, and 'handle_oversized_bin' now takes the already-published v_lo/v_hi boundaries instead of a raw threshold_bin, because the current find_threshold publishes boundaries rather than the bin index. The fallback's membership test is written as 'val >= v_lo && val < v_hi' rather than the exclusion form: the ragged entry point masks the <= 3 columns ahead of a window with NaN padding, which the exclusion form would admit. Co-authored-by: David Orman <ormandj@corenode.com> Co-authored-by: Benjamin Oldenburg <benjamin.oldenburg@ordis.th>
xiaofei-zheng
added a commit
to xiaofei-zheng/sglang
that referenced
this pull request
Sep 21, 2026
The coarse bin comes from the top bits of the fp16 cast, so one bin spans a quarter-binade. The collect pass stages threshold-bin candidates under `if (pos < kMaxNumTie)` and phase 4 clips `min(count_eq, kMaxNumTie)`, so an overflowing bin leaves `handle_tie` ranking an arrival-order subset and the kernel returns a wrong selected-value multiset with no diagnostic. Found while profiling GLM-5.2 decode on MI355X: device-side counters over an agentic serving run enter that path on ~0.04% of rows across all 8 ranks, with a largest observed bin of 4139 against the 2048 cap and none of those rows bit-identical. sgl-project#35257 reports a related problem from the CUDA side. sgl-project#39648 restructured this kernel but left all four truncation sites unchanged. Refine the overflowing bin on the exact key instead: up to three radix rounds over the order-preserving key from `extract_exact_bin`, 12 / 12 / 8 bits wide, exiting once the refined set fits the staging buffer or once all 32 bits are consumed, where the survivors are bit-identical and truncating them is exact. Each round emits its own "above" set while building the next round's histogram, so the refinement costs at most four input passes. The histogram overlays `tie_values`, which this path re-derives rather than trusting, so LDS is unchanged at 26928 B. The fast path branches on `count_eq > kMaxNumTie` and is otherwise untouched. Also pin the ROCm `__launch_bounds__` occupancy argument to the physical wave floor: HIP reads it as waves per SIMD where CUDA reads it as blocks per SM, so `kOccupancy` asked ROCm for 2 waves/SIMD, below the 4 a 1024-thread block already forces. The CUDA spelling stays because `topk_small_batch_cluster_kernel` shadows `kOccupancy` with a template parameter. Testing. Pre-existing cases draw from `torch.randn` and tolerate MAX_PERMIT_ERROR = 5, so they never reach the overflow. Added narrow, narrow_bin, tiny and two_values distributions that collapse a row into one coarse bin, plus overflow with DUAL_OUTPUT and overflow on an unaligned ragged window. On the kernel on main those 14 cases fail while the 2 all_equal controls pass, which pins the fault to truncation rather than to the binning. All 304 pass here. MI355X: the hot path is unchanged (5.324 vs 5.484 us at 6x8192, 12.094 vs 12.267 at 4x32768); rows that overflow now pay for work previously skipped by returning a wrong answer (narrow_bin 17.051 -> 21.684, two_values 36.194 -> 118.811 at 4x32768). 57 -> 79 VGPRs, zero spills. Known gap: the CUDA-only `TopKCluster` path still truncates. Refining there needs cluster-wide rather than block-local histogram and emit counters. Co-authored-by: Cursor <cursoragent@cursor.com>
2 tasks done
xiaofei-zheng
added a commit
to xiaofei-zheng/sglang
that referenced
this pull request
Sep 27, 2026
The coarse bin comes from the top bits of the fp16 cast, so one bin spans a quarter-binade. The collect pass stages threshold-bin candidates under `if (pos < kMaxNumTie)` and phase 4 clips `min(count_eq, kMaxNumTie)`, so an overflowing bin leaves `handle_tie` ranking an arrival-order subset and the kernel returns a wrong selected-value multiset with no diagnostic. Found while profiling GLM-5.2 decode on MI355X: device-side counters over an agentic serving run enter that path on ~0.04% of rows across all 8 ranks, with a largest observed bin of 4139 against the 2048 cap and none of those rows bit-identical. sgl-project#35257 reports a related problem from the CUDA side. sgl-project#39648 restructured this kernel but left all four truncation sites unchanged. Refine the overflowing bin on the exact key instead: up to three radix rounds over the order-preserving key from `extract_exact_bin`, 12 / 12 / 8 bits wide, exiting once the refined set fits the staging buffer or once all 32 bits are consumed, where the survivors are bit-identical and truncating them is exact. Each round emits its own "above" set while building the next round's histogram, so the refinement costs at most four input passes. The histogram overlays `tie_values`, which this path re-derives rather than trusting, so LDS is unchanged at 26928 B. The fast path branches on `count_eq > kMaxNumTie` and is otherwise untouched. Also pin the ROCm `__launch_bounds__` occupancy argument to the physical wave floor: HIP reads it as waves per SIMD where CUDA reads it as blocks per SM, so `kOccupancy` asked ROCm for 2 waves/SIMD, below the 4 a 1024-thread block already forces. The CUDA spelling stays because `topk_small_batch_cluster_kernel` shadows `kOccupancy` with a template parameter. Testing. Pre-existing cases draw from `torch.randn` and tolerate MAX_PERMIT_ERROR = 5, so they never reach the overflow. Added narrow, narrow_bin, tiny and two_values distributions that collapse a row into one coarse bin, plus overflow with DUAL_OUTPUT and overflow on an unaligned ragged window. On the kernel on main those 14 cases fail while the 2 all_equal controls pass, which pins the fault to truncation rather than to the binning. All 304 pass here. MI355X: the hot path is unchanged (5.324 vs 5.484 us at 6x8192, 12.094 vs 12.267 at 4x32768); rows that overflow now pay for work previously skipped by returning a wrong answer (narrow_bin 17.051 -> 21.684, two_values 36.194 -> 118.811 at 4x32768). 57 -> 79 VGPRs, zero spills. Known gap: the CUDA-only `TopKCluster` path still truncates. Refining there needs cluster-wide rather than block-local histogram and emit counters. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
-inftie behaviour.New kernels
topk_bf16_small(topk_transform_bf16_small): bf16 top-k for rows of at most 16384 scores, two radix passes over the raw bf16 bytes, fused with the page-table transform; adapted from DeepSelect's init-window select.block_amax(amax8_varlen): level-one keys of the two-level indexer, the max of each 8-score block with the row's newest block forced to+inf.candidate_block_table(sort_candidate_blocks): counting sort of a row's selected blocks over a bitmap plus the page transform, in place.jit/utils/occupancy.py(get_max_active_clusters): askscudaOccupancyMaxActiveClustershow many clusters of a given width fit at a given occupancy; sizes the Top-k v2 persistent pool at JIT time.attention/dsv4/candidate_table.pyhosts the two indexer helpers;topk.pykeeps only the top-k entry points.Top-k v2 changes
-FLT_MAX, so inactive entries never displace valid-infscores (test_topk_v2_negative_infinity_ties,test_topk_v2_ragged_negative_infinity); the padding bin is taken fromextract_coarse_bin(padding_value()), so the histogram correction does not depend on the platform's NaN encoding.coarse_bin_lower_boundsteps the midpoint up on odd-significand bins (fp32 -> fp16 rounds to nearest even) and returns+/-infat the key-space edges.TopKCluster<N>for N = 8 and 16, inputs interleaved across ranks, rank 0 owns the histogram and threshold behind release/acquire cluster barriers; the persistent-pool size and the cluster-16 small-batch cap come from the occupancy probe, with per-arch defaults as fallback;topk_plan_clustercarries per-arch candidate tables.SUPPORT_CLUSTERlimits the cluster path to sm90/sm100; sm120 and ROCm take the streaming path.plan_topk_v2(static_threshold): a negative value selects the heuristic (was0); no in-tree caller passes the argument.__builtin_assumein the small-batch cluster epilogue) is dropped: the restructured epilogue compiles with nvcc 13.1 through 13.4 for sm_90a, while the previous code still segfaults cicc without it.TieHandleSmemno longer over-aligns its counters;topk_v2_plan_is_writtenreplaces the test kit's use of the private JIT module.Verification
test_topk_v2.py: 288 cases pass on SM90 and SM100.+infnewest block, sorted block table with page mapping, bf16 top-k); they have no registered tests of their own and are covered by the model's end-to-end tests.bench_topk.pyseeds each configuration and addspage_sizeto the paged sweep; not re-run here.CI States
Latest PR Test (Base): 🚫 Run #35064790417
Latest PR Test (Extra): ❌ Run #35064790112
Latest PR Test (AMD ROCm 10): 🚫 Run #35064790372