Repository navigation
[Speculative] Add draft sampling overrides and retain corrected proposal probabilities - #43052
Open
maocheng23 wants to merge 8 commits into
Open
maocheng23 wants to merge 8 commits into
maocheng23 wants to merge 8 commits into
Conversation
maocheng23
marked this pull request as ready for review
October 9, 2026 01:59
maocheng23
requested review from
BBuf,
DarkSharpness,
HaiShaw,
HydraQYH,
JustinTong0323,
Qiaolin-Yu,
Ying1123,
celve,
hnyls2002,
merrymercy,
sogalin,
wisclmy0611,
yuan-luo and
zijiexia
as code owners
October 9, 2026 02:00
Qiaolin-Yu
reviewed
Oct 12, 2026
- Unset draft options apply no cutoff, so the default path runs no top-k/top-p kernel; the apply flags are static server args. - Move the per-device top-k/top-p renorm dispatch into sglang.kernels.ops.sampling with a graph-safe torch reference that keeps FlashInfer's tie semantics; drop srt/sampling/probability_transforms.py. - DSpark keeps the fused MarkovSampleStep draft by default. A cutoff override redraws sampling rows from the cut-off q, and verify and the accept estimator rebuild that q from the kept step logits instead of retaining a probs buffer. - Reuse fast_sample instead of a second Gumbel helper. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
maocheng23
force-pushed
the
codex/speculative-draft-sampling
branch
from
October 14, 2026 00:20
9f90505 to
21c1f2a
Compare
…hand AOT (CUDA), Triton (ROCm) and torch (CPU, NPU) are now KernelSpecs with capability requirements, so the public wrappers resolve through the registry like moe.topk_softmax does. MUSA and XPU keep a direct sgl_kernel call until DeviceType grows those members. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- sample_draft_proposal one-hots q on greedy rows: a row made greedy only by the draft temperature override keeps the target's full p at the verify, so the softmax q over-accepted the argmax. - A DSpark cutoff override redraws from the torch RNG, so the worker declares the draft site an RNG site and the rng sync preset covers it. - Drop the duplicated block in the speculative decoding docs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The kernel registry resolves top_k/top_p_renorm_probs by platform, so on a CUDA host a CPU tensor reached the sgl_kernel op and failed its device check; the CPU unit suites run on GPU boxes too. Name the torch reference for CPU tensors and pin it against the device kernel. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Write the point-mass q only under the draft temperature override: a request-greedy row's target p is one-hot, so its softmax q was already exact, and the masked write on every draft step cost about 2% decode throughput at batch 32 against main. - build_draft_probs takes an out slot, so the single-graph multi-layer runner takes the fused wide-row softmax straight into draft_probs again instead of a generic softmax plus a copy. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Qiaolin-Yu
reviewed
Oct 15, 2026
| from sglang.srt.sampling.draft_sampling import ( | ||
| DraftSamplingParams, | ||
| candidate_probs, | ||
| sample_candidate_path, |
Collaborator
There was a problem hiding this comment.
could we use existing sample_path or maybe merge these two?
- Drop the EAGLE draft graph runner's top_ps buffer: the draft reads only temperature and top_k, the request's top_p is the verify's. - One torch candidate walk: the DFlash2 selector delegates to sample_candidate_path, which keeps the vectorized form LiLiCorr now shares. - The multi-layer runner stages temperature and draft top_k through the fused prepare fill (a top_k slot joins the kernel and its native fallback) instead of a separate copy_from of about nine launches; the server's fixed values are written into the buffers once at allocation. - DSpark rebuilds the cut-off q per step with the same [bs, vocab] calls the draft made, so the verify meets the draft's kernels bit for bit on every platform, and builds it once: the sampling accept and the accept estimator score the same tensor via AcceptOuts.cutoff_draft_probs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Merged
4 of 5 tasks
4 of 5 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Stochastic draft proposals need their own sampling controls, and verification must use the exact distribution each proposal was drawn from. EAGLE and DSpark drafted at the request's temperature only, and the DFlash2 selector and LiLiCorr walks had no way to sharpen or flatten their proposals.
Modifications
--speculative-draft-temperature,--speculative-draft-top-kand--speculative-draft-top-p. Unset, the draft samples at the request's temperature with no cutoff, as before: the default path runs no top-k/top-p kernel, and a disabling value (-1,1) reads as unset. Draft temperature0makes proposals greedy; greedy requests always draft greedily.srt/sampling/draft_sampling.py:DraftSamplingParamscarries the per-request draft policy plus the server's static apply flags;build_draft_probsreturns q = softmax(logits / T) with top-k then top-p applied only when set. Rejection sampling stays unbiased for any q as long as verification uses the q the proposal was drawn from.kernels/ops/sampling:top_k_renorm_probs/top_p_renorm_probsnow carry three registered backends, the sgl_kernel AOT op on CUDA, the Triton port on ROCm, and a graph-safe torch reference with FlashInfer's tie semantics on CPU and for NPU rows the vendor path declines, selected by the kernel registry's capability filter. MUSA and XPU keep a direct sgl_kernel call untilDeviceTypegrows those members. Draft proposals and EAGLE/DFlash target verification share that one path; the per-file import chains are gone.MarkovSampleStepdraft on the default path. A temperature override feeds that kernel, and draft temperature 0 maps to the argmax draft. A top-k/top-p override makes each step argmax in-kernel and redraw the sampling rows from the cut-off q built from the kept step logits; verification and the accept estimator rebuild that same q fromcorrected_logits, so no q buffer is retained.Accuracy Tests
test/registered/unit/spec,test/registered/spec/dsparkandtest/registered/unit/server_argsdirectories: 810 passed; every remaining failure (CUDA-only kernels,tvm_ffi, grpc) fails identically on main in this environment.Speed Tests and Profiling
Pending GPU validation; this PR is a draft. With no draft option set, no renorm kernel runs and DSpark's fused draft kernel is untouched. With a cutoff set, each draft step adds a top-k and/or top-p renormalization pass over the draft support, and DSpark additionally moves its sampling out of the fused kernel; that opt-in cost is what needs measuring against the acceptance-length gain.
Checklist
Review and Merge Process
GPU correctness, model accuracy, and performance validation are pending before marking this PR ready for review.
CI States
Latest PR Test (Base): ❌ Run #38690627433
Latest PR Test (Extra): ❌ Run #38690627226
Latest PR Test (AMD ROCm 10): ❌ Run #38690627473