Skip to content

[Speculative] Add draft sampling overrides and retain corrected proposal probabilities - #43052

Open
maocheng23 wants to merge 8 commits into
sgl-project:mainfrom
maocheng23:codex/speculative-draft-sampling
Open

maocheng23 wants to merge 8 commits into
sgl-project:mainfrom
maocheng23:codex/speculative-draft-sampling

Conversation

@maocheng23

@maocheng23 maocheng23 commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Stochastic draft proposals need their own sampling controls, and verification must use the exact distribution each proposal was drawn from. EAGLE and DSpark drafted at the request's temperature only, and the DFlash2 selector and LiLiCorr walks had no way to sharpen or flatten their proposals.

Modifications

  • Add --speculative-draft-temperature, --speculative-draft-top-k and --speculative-draft-top-p. Unset, the draft samples at the request's temperature with no cutoff, as before: the default path runs no top-k/top-p kernel, and a disabling value (-1, 1) reads as unset. Draft temperature 0 makes proposals greedy; greedy requests always draft greedily.
  • srt/sampling/draft_sampling.py: DraftSamplingParams carries the per-request draft policy plus the server's static apply flags; build_draft_probs returns q = softmax(logits / T) with top-k then top-p applied only when set. Rejection sampling stays unbiased for any q as long as verification uses the q the proposal was drawn from.
  • kernels/ops/sampling: top_k_renorm_probs / top_p_renorm_probs now carry three registered backends, the sgl_kernel AOT op on CUDA, the Triton port on ROCm, and a graph-safe torch reference with FlashInfer's tie semantics on CPU and for NPU rows the vendor path declines, selected by the kernel registry's capability filter. MUSA and XPU keep a direct sgl_kernel call until DeviceType grows those members. Draft proposals and EAGLE/DFlash target verification share that one path; the per-file import chains are gone.
  • EAGLE/EAGLE3/MTP rejection sampling uses it in prefill, eager decode, per-step and single CUDA graphs. The draft graph runner stages temperature/top-k/top-p and the overrides resolve inside the graph.
  • DSpark keeps the fused MarkovSampleStep draft on the default path. A temperature override feeds that kernel, and draft temperature 0 maps to the argmax draft. A top-k/top-p override makes each step argmax in-kernel and redraw the sampling rows from the cut-off q built from the kept step logits; verification and the accept estimator rebuild that same q from corrected_logits, so no q buffer is retained.
  • DFlash2 selector and sampled LiLiCorr apply the policy to each corrected conditional over the K candidates; top-k cannot add tokens. The walk kernel takes q only, with greedy rows one-hot.
  • Plain DFlash, Domino, unsampled LiLiCorr and EAGLE tree search are unchanged.

Accuracy Tests

  • CPU (macOS ARM64, Python 3.12, PyTorch 2.14.1): 82 passed across the draft-sampling, DSpark override, DSpark draft block and accept, DFlash selector, EAGLE graph runner, spec registry and server-args docs coverage suites. The full test/registered/unit/spec, test/registered/spec/dspark and test/registered/unit/server_args directories: 810 passed; every remaining failure (CUDA-only kernels, tvm_ffi, grpc) fails identically on main in this environment.
  • GPU tests are added or updated but not yet run: target/draft cutoff parity, CUDA graph replay with changing parameters, the DSpark draft graph, and the selector/LiLiCorr Triton walk.
  • ruff (lint and format), isort and codespell pass on the changed files.

Speed Tests and Profiling

Pending GPU validation; this PR is a draft. With no draft option set, no renorm kernel runs and DSpark's fused draft kernel is untouched. With a cutoff set, each draft step adds a top-k and/or top-p renormalization pass over the draft support, and DSpark additionally moves its sampling out of the fused kernel; that opt-in cost is what needs measuring against the acceptance-length gain.

Checklist

  • Format code with pre-commit.
  • Add unit tests with CPU/GPU CI registration.
  • Update documentation.
  • Provide model accuracy and speed benchmark results.
  • Follow SGLang code style guidance.

Review and Merge Process

GPU correctness, model accuracy, and performance validation are pending before marking this PR ready for review.


CI States

Latest PR Test (Base): ❌ Run #38690627433
Latest PR Test (Extra): ❌ Run #38690627226
Latest PR Test (AMD ROCm 10): ❌ Run #38690627473

maocheng and others added 2 commits October 13, 2026 17:01
- Unset draft options apply no cutoff, so the default path runs no
  top-k/top-p kernel; the apply flags are static server args.
- Move the per-device top-k/top-p renorm dispatch into
  sglang.kernels.ops.sampling with a graph-safe torch reference that keeps
  FlashInfer's tie semantics; drop srt/sampling/probability_transforms.py.
- DSpark keeps the fused MarkovSampleStep draft by default. A cutoff
  override redraws sampling rows from the cut-off q, and verify and the
  accept estimator rebuild that q from the kept step logits instead of
  retaining a probs buffer.
- Reuse fast_sample instead of a second Gumbel helper.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@maocheng23
maocheng23 force-pushed the codex/speculative-draft-sampling branch from 9f90505 to 21c1f2a Compare October 14, 2026 00:20
maocheng and others added 5 commits October 14, 2026 00:13
…hand

AOT (CUDA), Triton (ROCm) and torch (CPU, NPU) are now KernelSpecs with
capability requirements, so the public wrappers resolve through the
registry like moe.topk_softmax does. MUSA and XPU keep a direct
sgl_kernel call until DeviceType grows those members.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- sample_draft_proposal one-hots q on greedy rows: a row made greedy only
  by the draft temperature override keeps the target's full p at the
  verify, so the softmax q over-accepted the argmax.
- A DSpark cutoff override redraws from the torch RNG, so the worker
  declares the draft site an RNG site and the rng sync preset covers it.
- Drop the duplicated block in the speculative decoding docs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The kernel registry resolves top_k/top_p_renorm_probs by platform, so on a
CUDA host a CPU tensor reached the sgl_kernel op and failed its device
check; the CPU unit suites run on GPU boxes too. Name the torch reference
for CPU tensors and pin it against the device kernel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Write the point-mass q only under the draft temperature override: a
  request-greedy row's target p is one-hot, so its softmax q was already
  exact, and the masked write on every draft step cost about 2% decode
  throughput at batch 32 against main.
- build_draft_probs takes an out slot, so the single-graph multi-layer
  runner takes the fused wide-row softmax straight into draft_probs again
  instead of a generic softmax plus a copy.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
from sglang.srt.sampling.draft_sampling import (
DraftSamplingParams,
candidate_probs,
sample_candidate_path,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we use existing sample_path or maybe merge these two?

- Drop the EAGLE draft graph runner's top_ps buffer: the draft reads only
  temperature and top_k, the request's top_p is the verify's.
- One torch candidate walk: the DFlash2 selector delegates to
  sample_candidate_path, which keeps the vectorized form LiLiCorr now shares.
- The multi-layer runner stages temperature and draft top_k through the
  fused prepare fill (a top_k slot joins the kernel and its native
  fallback) instead of a separate copy_from of about nine launches; the
  server's fixed values are written into the buffers once at allocation.
- DSpark rebuilds the cut-off q per step with the same [bs, vocab] calls
  the draft made, so the verify meets the draft's kernels bit for bit on
  every platform, and builds it once: the sampling accept and the accept
  estimator score the same tensor via AcceptOuts.cutoff_draft_probs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation jit-kernel speculative-decoding

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants