Skip to content

fix(spec): sync grammar state for mixed decode tails - #40458

Open
pockers21 wants to merge 1 commit into
sgl-project:mainfrom
pockers21:fix/mixed-chunk-grammar-sync
Open

pockers21 wants to merge 1 commit into
sgl-project:mainfrom
pockers21:fix/mixed-chunk-grammar-sync

Conversation

@pockers21

@pockers21 pockers21 commented Sep 20, 2026 •

Copy link
Copy Markdown

Motivation

Grammar state is normally advanced during speculative verify(). Mixed batches run existing decode requests through the target-prefill path, which skips verify().

When grammar overlap is enabled, this can leave the grammar state one step behind while the next token mask is being built. The stale mask may admit a token that the updated grammar state rejects, causing the request to abort with Grammar accept_token failed.

Modifications

  • Synchronize pending grammar state when a mixed batch contains grammar-constrained decode requests.
  • Apply the pending-grammar check to both decode and mixed forward modes.
  • Keep grammar overlap enabled for mixed batches with grammar only on prefill requests.
  • Keep the existing overlap behavior for regular speculative decode batches.
  • Add unit tests for the mixed-batch synchronization decisions.

Accuracy Tests

Tested with a local constrained-generation reproducer using Qwen3.5-9B, NEXTN/MTP, mixed chunk, grammar constraints, and overlap enabled:

  • Before the fix: 0/12 requests completed successfully; 12/12 aborted.
  • After the fix: 12/12 requests completed with valid output; 0/12 aborted.

Unit and related regression tests:

  • test_scheduler_decision_batch_params.py: 5 passed
  • test_batch_result_processor_spec_grammar.py: 6 passed
  • test_schedule_batch_out_of_place.py: 4 passed
  • test_schedule_batch_convert_decode_to_extend.py: 4 passed

Total: 19 passed.

Speed Tests and Profiling

Not run. The additional synchronization is limited to mixed batches whose decode tail contains a grammar-constrained request. Other mixed batches and regular speculative decode batches retain the existing overlap behavior.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #35553531537
Latest PR Test (Extra): ❌ Run #35553531284
Latest PR Test (AMD ROCm 10): ❌ Run #35553531414

@pockers21
pockers21 force-pushed the fix/mixed-chunk-grammar-sync branch from 29ea4b5 to 0c4d91e Compare September 21, 2026 02:14
@pockers21

Copy link
Copy Markdown
Author

Hi maintainers, could someone please trigger CI for this PR?
@Ying1123 @hnyls2002

@sethforprivacy

Copy link
Copy Markdown

Posted by Claude (AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work.

Independent confirmation of this bug and of the fix's decision logic, on a different model and speculative algorithm:

Setup: DeepSeek-V4.1-Flash, DSpark (block size 5), TP4 across 4x GB10 (SM121), SGLang v0.5.22, --enable-mixed-chunk, overlap scheduling on, xgrammar.

Symptom: a constrained-output probe at 8 concurrent requests (24 tool_choice: "required", 24 named-function, 24 json_schema) scored 2/24, 0/24 and 8/24. 60 of 72 requests ended with finish_reason: "abort" (HTTP 200, partial output), and the engine logged 62 Grammar accept_token failed ... Tokens not accepted lines, 54 of them right after a mixed step. One request at a time, or with mixed chunk off: 72/72.

Cause, as you describe: DSpark advertises grammar overlap, so grammar_needs_sync() stays False, and the grammar barrier only runs inside verify(). A MIXED batch goes through the target prefill path, and is_disable_overlap_for_batch only forces the sync for forward_mode.is_decode(). The merged decode rows are therefore sampled against a vocab mask built from FSMs one verify behind.

Fix: we carry the same decision as this PR as an import-time patch on v0.5.22: synchronous path for a MIXED spec batch whose decoding_reqs carry a grammar, with a result still queued; prefill-only grammar keeps overlap. After it: 72/72 at 8 concurrent, twice, and 72/72 while a 4 x 131K-token deep batch was mixing in, with 0 accept failures. Throughput was unchanged on unconstrained traffic.

With #46049 merged, this failure now surfaces as a 500 instead of a silent 200 abort, but the abort itself still happens without this fix. It would be great to see this merged.

A second, separate mixed-chunk issue showed up while verifying this one (silent accuracy loss under --enable-decoder-swa-bounded-replay); filed as #46615.

detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
…entries 122,198,200,211,215,220,267,268,305

Re-plants the scheduler.py fork-hunks (replant/...managers__scheduler.py.fork-hunks.patch)
onto the post-sgl-project#46977 admission structure. tp_worker marker consumed (15/23 B4 markers
still on the tree; per-file counts, damage-state from entry 310's map).

PER-HUNK DECISIONS (all VERIFIED against the merged tree, not taken on faith):
- entry 122 (sgl-project#40458): grammar-sync overlap disable RE-PLANTED for mixed mode
  (is_decode() OR is_mixed()) + the fork's trimmed comments; upstream's
  host-draft-only arm was the pre-sgl-project#40458 form (origin/main verified: absent).
- entry 220 (sgl-project#39765, OPEN port): _has_pending_mamba_cache_update + the early
  pop_and_process/short-circuited disable-overlap RE-PLANTED into
  event_loop_overlap; the require_mlp_sync/disable/extra-buffer gate verbatim.
- entry 198 (sgl-project#37612): THE CATCHER — B3-5 landed the raiser; the
  try/except MambaCowAllocError around init_next_round_input (res=NO_TOKEN,
  request stays queued; the revert block + unconditional alloc_group_end were
  already in the merged tree = the no-leak halves VERIFIED PRESENT) RE-PLANTED
  with the import. COW component test re-anchored to the #46970x kv_storage
  seam the B3-5 replant moved _state() behind (stub was the pre-split
  mamba_allocator shape).
- entry 268 (sgl-project#46643, OPEN port -- VERIFIED NEVER MERGED: git log
  12dc623..2a6c204 --grep 46643 = ZERO; only sgl-project#45491/sgl-project#45435/sgl-project#45472/sgl-project#45670
  of the briefed merged-list arrived in-range): the UNCONDITIONAL batch_is_full
  clear RE-PLANTED on the scheduler side, subsuming the entry-198 hybrid-SSM
  arm and the priority-preemption/is_empty arms (policy side lands in B4-3).
- entry 267 (sgl-project#46496): retract_decode new_token_ratio FLOOR clamp RE-PLANTED.
- entry 211: triton_load_watch.prewarm_device_loads(self.model_worker) line
  RE-PLANTED between install() and mark_serving_started (the utils-side
  function survived the merge; only the call site was dropped).
- entry 200 (sgl-project#43527) CAPACITY PAIR RE-PLANTED: model_runner
  .logical_max_total_num_tokens passes allocator= again (the configurator's
  PageInterleavePoolAllocator -> allocator.size branch SURVIVED the merge
  byte-identical, so the merged tree had the fork predicate with upstream's
  consumer-side multiplications = DOUBLE-COUNT); the three scheduler
  * kv_shard_widening multipliers back to the fork form (pool-stats shape,
  load_inquirer, and the init_req one-page-per-shard subtraction);
  tp_worker drops page_interleave_shard_size x + import. Upstream's
  invariant-checker hunk: OBSOLETE (the constructor no longer takes
  max_total_num_tokens -- absorbed by refactor).
- entry 215 (sgl-project#44351/sgl-project#44401/sgl-project#44420): VERIFIED ALREADY PRESENT at the merge
  (filter_batch-every-step + reset ride, num_held_rows, budget_state FIRST
  statement, PD-gate deletion all byte-compare to the fork form) -- DEDUPED,
  no edit. Entry 107's local early-revision of sgl-project#40940 stays superseded
  (entry 215 rule), and its re-plant here would double-count.
- entry 220's/305's pause-window busy accounting + write_fence_stream +
  is_empty arm: VERIFIED PRESENT (entry 305 landed patch-identical via
  sgl-project#46127; the fence assignment is the B3-2 controller half) -- DEDUPED.

TEST FIXTURE RE-ANCHORS (not B2's: no markers, pure new-upstream-contract
drift, per the entry-198 fixture-adaptation precedent): the two hybrid-ssm
admission stubs gain disagg_prefill_bootstrap_queue=None (upstream's
routing-key bootstrap_queue arg in calc_priority arrived in-range).

PRUNE vs BREAKAGE (managers −523sub audit): the three managers test files
vanished in-range (test_schedule_policy_dfs_weight, test_scheduler_batch_is
_full_latch, test_uno_token_accounting) were ALL deleted by upstream's OWN
sgl-project#45672 prune (3438ed6). Two are byte-identical upstream copies = zero
fork delta. The third carried a 3-line fork comment/stub delta whose
SEMANTICS the re-planted unconditional clear pins: its case was the
is_empty-arm mirror of test_scheduler_hybrid_ssm_admission_latch (present,
green, hybrid_swa/non-hybrid arms included) -- NOT re-homed, recorded.

ACTUALS (damage -> after, CVD=9, prod venv, PYTHONPATH=python, per-file):
  test_scheduler_hybrid_ssm_admission_retry   4F -> ** 4P **
  test_scheduler_hybrid_ssm_admission_latch   3F -> ** 4P **
  test_mamba_prefill_cache_boundary  2F/12subF -> ** 5P/12sub EXACT (220 pin) **
  test_scheduler_retract_ratio                1F -> ** 2P EXACT (267 pin) **
  test_scheduler_init_req_max_new_tokens      1F -> ** 10P/6sub **
  test_scheduler_grammar_mixed_tail_sync      1F -> ** 4P **
  adjacent scheduler-stub files (held_row, running_batch_finished,
  pause_generation, disagg_idle, dp_converted, hicache_events) 56P/64sub
  UNMOVED -- no regression from the clear/catcher/capacity edits.
Lint: ruff format + F821/E9 clean on all five files; imports OK (no GPU
execution, no push).

Co-Authored-By: OpenCode <noreply@opencode.ai>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants