Skip to content

[sglang-miles] Return sampling masks for chain EAGLE/NEXTN decoding - #43047

Draft
nreHieW wants to merge 44 commits into
sgl-project:sglang-milesfrom
nreHieW:proximal/spec-sampling-mask-upstream
Draft

nreHieW wants to merge 44 commits into
sgl-project:sglang-milesfrom
nreHieW:proximal/spec-sampling-mask-upstream

Conversation

@nreHieW

@nreHieW nreHieW commented Oct 8, 2026 •

Copy link
Copy Markdown

Motivation

Currently, return_sampling_mask rejects all speculative-decoding requests. This prevents us from retrieving sampling suports while using EAGLE/NEXTN acceleration.

This PR enables support for returning the sampling support and normalized log probabilities for each emitted token.

Modifications

  • Capture the verifier’s existing filtered probabilities and align masks with accepted tokens, including the final sampled token.
  • Return only masks corresponding to emitted tokens when generation stops within a verify step.

This only supports:

  • CUDA
  • speculative_eagle_topk=1.
  • Default acceptance thresholds.
  • No rejection sampling, min_p, or simulated acceptance.

Accuracy Tests

NA

Speed Tests and Profiling

NA

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #37724186412
Latest PR Test (Extra): ❌ Run #37724186137
Latest PR Test (AMD ROCm 10): ❌ Run #37724186603

Kangyan-Zhou and others added 30 commits September 29, 2026 17:30
…nd pin SentencePiece for InternVL (sgl-project#41777) (sgl-project#41788)

Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
…ty check to a warning (sgl-project#41681) (sgl-project#41802)

Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…ject#22623)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…lication (sgl-project#27268, sgl-project#31759, sgl-project#30913, sgl-project#30912, sgl-project#33299, sgl-project#36384, sgl-project#38165)

Register adapter identities, stream their tensors within weight-update sessions, and publish checksum-verified fresh versions alongside generation. Preserve per-rank failure cleanup and request-prefix cancellation for adapter lifecycle management.

Co-authored-by: Mathew Han <mathewjhan@gmail.com>
Co-authored-by: Ethan (Yusheng) Su <yushengsu.thu@gmail.com>
Co-authored-by: Yueming Yuan <yym022502@gmail.com>
…ism (sgl-project#21278, sgl-project#22663, sgl-project#33478)

Expose rank topology for P2P weight mapping and create replicas using scoped runtime-context overrides. Restore the previous configuration on exit, including failed context entry.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: yueming-yuan <yym022502@gmail.com>
…RL weight checks (sgl-project#29339)

Co-authored-by: yueming-yuan <yym022502@gmail.com>
…30499, sgl-project#37466, sgl-project#37887)

Co-authored-by: fzyzcjy <ch271828n@outlook.com>
Co-authored-by: yueming-yuan <yym022502@gmail.com>
…gl-project#27723)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: yueming-yuan <yym022502@gmail.com>
…tils (sgl-project#35127)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Shi Dong <shi.dong@radixark.ai>
…weight checks (sgl-project#37649)

Co-authored-by: Zhichenzzz <zczeng@uw.edu>
Zhichenzzz and others added 14 commits September 30, 2026 20:30
…load; restore bf16 MoE layout (sgl-project#40821)

Co-authored-by: Yueming Yuan <yym022502@gmail.com>
…PC LoRA updates (sgl-project#40786)

Keep H3 rollout and streamed-weight behavior on the release implementation. The release already contains the LoRA wrapper type guards.

Co-authored-by: WenhaoZhang <42087078+niehen6174@users.noreply.github.com>
Co-authored-by: niehen6174 <niehen6174@gmail.com>
… graph registration (sgl-project#40648, sgl-project#40978, sgl-project#40979, sgl-project#40980, sgl-project#40977, sgl-project#41972)

Retain colocated training fixes above the release MiMo implementation, including address-stable Marlin reload and mandatory TopK layer IDs. Disable NCCL graph registration for TP LM-head all-to-all and pausable graph pools.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…etadata (sgl-project#39265)

Co-authored-by: yueming-yuan <yym022502@gmail.com>
…oject#41925)

Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
…each scheduler actually bound to its clients (sgl-project#42470)

Signed-off-by: rockdu <kangrdu@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.