Skip to content

[Fix] speculative: Clear stale grammar masks before DFlash and DSpark verification - #43518

Open
Hahaa909 wants to merge 2 commits into
sgl-project:mainfrom
Hahaa909:jianghh/clear-stale-verify-grammar
Open

Hahaa909 wants to merge 2 commits into
sgl-project:mainfrom
Hahaa909:jianghh/clear-stale-verify-grammar

Conversation

@Hahaa909

@Hahaa909 Hahaa909 commented Oct 10, 2026 •

Copy link
Copy Markdown

Motivation

Related to #43514. A SamplingBatchInfo can still carry the previous extend/decode-stage grammar_mask when entering speculative verification. The shared DFlash/DSpark logits-adjustment helper currently applies that per-request mask to a zero-valued penalty buffer and broadcasts the result across the entire verify block. An all-masked stale row makes every verify logit -inf; a stale mask with a different batch size can also cause a shape mismatch.

Verify callers already apply a separate grammar mask for each current tree position. Reusing the previous per-request mask here is incorrect even if the batch sizes happen to match.

This is a draft pending normal Linux-runtime execution of the new tests and reproduction of the issue's mixed grammar/non-grammar, long-prefill overlap workload on an accelerator. The function-level regression does not establish that every possible cause of the reported production failure is fixed.

Modifications

  • Clear sampling_info.grammar_mask in apply_dflash_verify_logits_adjustments after validating the verify-logit shape, before applying any sampling adjustments.
  • Remove the stale-mask-triggered dense fallback. Preserve the existing live-penalizer fallback, precomputed-penalty path, custom-logit-processor call, and logit-bias handling.
  • Add a CPU-registered regression test using CustomTestCase, real SamplingBatchInfo/GrammarMask implementations, and CPU tensors. Only the external grammar backend and live penalizer are lightweight test doubles.
  • Leave current tree-position grammar application at the existing DFlash and DSpark call sites unchanged.

Relationship to existing work

Accuracy Tests

Added test/registered/unit/spec/test_dflash_verify_grammar_mask.py with five test methods covering:

  1. Unconstrained verify logits and selected tokens remain unchanged despite a stale mask, including float32/bfloat16 and a changed batch size.
  2. The caller's current grammar mask still constrains each verify-tree position independently.
  3. Live additive penalties and logit bias still apply, with and without a stale mask.
  4. Logit bias works without a live penalizer.
  5. The helper's existing precomputed acc_linear_penalties path survives the stale mask, including float32-to-bfloat16 conversion.

Local validation on Windows used an isolated, non-shipped runner that compiles the unchanged production function/class definitions from the checkout and executes the new tests with CPU PyTorch. It bypasses package initialization and replaces only the CI test-case scaffolding with unittest.TestCase; it is not a normal SGLang suite run. All five test methods expose failures with the original helper and pass with this patch.

Normal test collection was attempted but stops during SGLang package initialization because Windows lacks the Unix resource module. No GPU model/server accuracy test has been run.

Please run in the normal Linux CI environment:

python3 test/registered/unit/spec/test_dflash_verify_grammar_mask.py

The issue's mixed structured-output and long-prefill speculative-decoding workload still needs accelerator validation.

Speed Tests and Profiling

Not benchmarked. The new reset is a host-side attribute assignment, and a stale mask no longer causes allocation/application of the dense penalty buffer. No throughput or latency claim is made.

Checklist

Review and Merge Process

Draft: normal-runtime regression execution and accelerator workload validation are pending before requesting merge.


CI States

Latest PR Test (Base): ❌ Run #40834947733
Latest PR Test (Extra): ❌ Run #40834947836
Latest PR Test (AMD ROCm 10): ❌ Run #40834947791

@Hahaa909
Hahaa909 marked this pull request as ready for review October 10, 2026 07:59
@Hahaa909

Copy link
Copy Markdown
Author

The PR is ready for review, but the CI gate is currently blocked because the run-ci and run-ci-extra labels are missing. I cannot add repository labels from this fork. Could a maintainer add the labels or manually rerun the required CI workflows? The PR is no longer a draft, and the requested Linux/runtime and accelerator validation can proceed once the CI gate is enabled.

pamanseau added a commit to KubeTEE-AI/sglang that referenced this pull request Oct 17, 2026
…e grammar mask, issue sgl-project#43514)

Byte-exact gh pr diff of sgl-project#43518 (Hahaa909, +161/-5,
dflash_utils.py + its unit test). Applied in-image by
docker/kubetee-mimo-overlay/Dockerfile with pre/post sha256 tripwires;
the PR's own unit test is the build smoke check.

Preimage dflash_utils.py @ ed8c313 = 5f8f266964e4dd15b426d197e944c13d7a75c4d1749187a9f09506bc8cfb61a6
Postimage                      = 81b58d09fb65c51d3bd3b73bc17757b8851269ff529f8a4cc2bf8b0818f437e4
test file                      = 5664b9282b4d95137169487ef25f476d156210659606ca1125b074d00a197fa3

Retire this patch when sgl-project#43518 merges upstream.
pamanseau added a commit to KubeTEE-AI/sglang that referenced this pull request Oct 17, 2026
…ject#43518)

Layers upstream PR sgl-project#43518 (Hahaa909 — dflash verify
stale grammar mask, issue sgl-project#43514) onto the digest-pinned
nightly-dev-cu13 base WITHOUT disturbing the base source otherwise
(unlike docker/kubetee-overlay's full tree swap).

Safety: pre/post sha256 drift tripwires on dflash_utils.py (preimage
5f8f2669…, postimage 81b58d09…) + the PR's own unit test runs as the
build smoke check against the patched tree.

Target: XiaomiMiMo/MiMo-V2.6-Pro-RL on B200 CC. Retire when sgl-project#43518
merges upstream — see the SOURCE POLICY CARVE-OUT in the header.
pamanseau added a commit to KubeTEE-AI/sglang that referenced this pull request Oct 17, 2026
Additive inputs (defaults unchanged — existing dsv41 runs unaffected):
  variant=dsv41|mimo  (default dsv41)
  source_sha          [mimo] upstream SHA the base image was built from
  tag_prefix          exact primary tag override

variant=mimo builds docker/kubetee-mimo-overlay/Dockerfile — the base
image's own source tree with upstream PR sgl-project#43518's patch applied in
place (dflash verify stale grammar mask, issue sgl-project#43514; target
XiaomiMiMo/MiMo-V2.6-Pro-RL on B200 CC). Tags
mimo-cu13-<shortsha>-<date> + mimo-cu13-latest.

See the mimo Dockerfile header for the SOURCE POLICY CARVE-OUT and the
retire-on-merge rule.
jpezzulli pushed a commit to jpezzulli/sglang-rtxpro6000 that referenced this pull request Oct 17, 2026
…tments

apply_dflash_verify_logits_adjustments runs on the [bs * draft_token_num, vocab]
verify logits, and it read sampling_info.grammar_mask -- which at that point
belongs to the previous extend/decode step and has one row per REQUEST. It fed
the dense fallback, which applies the mask to a zero buffer and broadcasts the
result over the whole verify block. Two ways that goes wrong, both now pinned by
a regression:

* the transition out of grammar (a request's FSM finished, or the batch was
  filtered down) leaves no new mask and a stale row behind. An all-masked row
  turned every verify logit -inf; a mask with the pre-filter row count raised
  "size of tensor a (2) must match the size of tensor b (3)".
* even a same-sized live mask is the wrong state. Each verify-tree position has
  its own grammar state, and the verify callers already build and apply that
  per-position mask (build_grammar_vocab_mask in dflash_worker_v2 and
  dspark_worker_v2, applied after these adjustments). Broadcasting a
  per-request mask here masks a position by its predecessor's state.

Clear sampling_info.grammar_mask up front and drop it as a fallback trigger, so
per-position verification stays owned by the callers. build_grammar_vocab_mask
already cleared the leftover when it produced a fresh mask; this closes the
remaining path where it produces none. Everything the fallback existed for is
preserved: the custom logit processor still runs first on the unadjusted
logits, a live penalizer still takes the dense path with the same ordering
(penalizer, then logit_bias -- which is why the expectations are not just
elementwise sums), and precomputed acc_additive_penalties and logit_bias still
broadcast over the block. DSpark inherits this through
apply_logits_adjustments_strided and its own callers; its
verify_logits_adjustments_are_noop still counts a lingering mask as work, which
is now merely conservative (the call clears it) rather than required.

Adapted to this tree's acc_additive_penalties spelling of the overlap-penalty
buffer (upstream renamed it acc_linear_penalties). CPU-only: red on the exact
base (6 cases -- poisoned logits, the filtered-row RuntimeError, and a
per-position argmax of all -inf), green after. The mixed grammar/non-grammar
long-prefill overlap workload is unverified here, no GPU.

Source: sgl-project/sglang#43518
(cherry picked from commit 74684d99c58fd04911089e37c87f900d6843302d)
Co-authored-by: jianghh <3232826884@qq.com>
@pamanseau

pamanseau commented Oct 24, 2026 •

Copy link
Copy Markdown

@hnyls2002 (CODEOWNER for speculative/, merged the sibling mask fix #43054) —
could you add run-ci / run-ci-extra here? This is stalled only because the CI
gate can't run without those labels.

Reproducing the symptom in production (SGLang main, 8×B200, --speculative-algorithm=DFLASH

Happy to run the DFLASH + tools battery on our hardware and post numbers if that helps review.

@Hahaa909
Hahaa909 force-pushed the jianghh/clear-stale-verify-grammar branch from 0984717 to 1cbbf72 Compare October 29, 2026 07:21
@github-actions

This comment has been minimized.

@Hahaa909 Hahaa909 changed the title [Spec] Clear stale grammar masks before DFlash and DSpark verification [Fix] speculative: Clear stale grammar masks before DFlash and DSpark verification Oct 30, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants