Repository navigation
Conversation
44e0c17 to
29ea4b5
Compare
29ea4b5 to
0c4d91e
Compare
|
Hi maintainers, could someone please trigger CI for this PR? |
|
Posted by Claude (AI assistant) on behalf of @sethforprivacy, as part of our local-inference tuning work. Independent confirmation of this bug and of the fix's decision logic, on a different model and speculative algorithm: Setup: DeepSeek-V4.1-Flash, DSpark (block size 5), TP4 across 4x GB10 (SM121), SGLang v0.5.22, Symptom: a constrained-output probe at 8 concurrent requests (24 Cause, as you describe: DSpark advertises grammar overlap, so Fix: we carry the same decision as this PR as an import-time patch on v0.5.22: synchronous path for a MIXED spec batch whose With #46049 merged, this failure now surfaces as a 500 instead of a silent 200 abort, but the abort itself still happens without this fix. It would be great to see this merged. A second, separate mixed-chunk issue showed up while verifying this one (silent accuracy loss under |
…entries 122,198,200,211,215,220,267,268,305 Re-plants the scheduler.py fork-hunks (replant/...managers__scheduler.py.fork-hunks.patch) onto the post-sgl-project#46977 admission structure. tp_worker marker consumed (15/23 B4 markers still on the tree; per-file counts, damage-state from entry 310's map). PER-HUNK DECISIONS (all VERIFIED against the merged tree, not taken on faith): - entry 122 (sgl-project#40458): grammar-sync overlap disable RE-PLANTED for mixed mode (is_decode() OR is_mixed()) + the fork's trimmed comments; upstream's host-draft-only arm was the pre-sgl-project#40458 form (origin/main verified: absent). - entry 220 (sgl-project#39765, OPEN port): _has_pending_mamba_cache_update + the early pop_and_process/short-circuited disable-overlap RE-PLANTED into event_loop_overlap; the require_mlp_sync/disable/extra-buffer gate verbatim. - entry 198 (sgl-project#37612): THE CATCHER — B3-5 landed the raiser; the try/except MambaCowAllocError around init_next_round_input (res=NO_TOKEN, request stays queued; the revert block + unconditional alloc_group_end were already in the merged tree = the no-leak halves VERIFIED PRESENT) RE-PLANTED with the import. COW component test re-anchored to the #46970x kv_storage seam the B3-5 replant moved _state() behind (stub was the pre-split mamba_allocator shape). - entry 268 (sgl-project#46643, OPEN port -- VERIFIED NEVER MERGED: git log 12dc623..2a6c204 --grep 46643 = ZERO; only sgl-project#45491/sgl-project#45435/sgl-project#45472/sgl-project#45670 of the briefed merged-list arrived in-range): the UNCONDITIONAL batch_is_full clear RE-PLANTED on the scheduler side, subsuming the entry-198 hybrid-SSM arm and the priority-preemption/is_empty arms (policy side lands in B4-3). - entry 267 (sgl-project#46496): retract_decode new_token_ratio FLOOR clamp RE-PLANTED. - entry 211: triton_load_watch.prewarm_device_loads(self.model_worker) line RE-PLANTED between install() and mark_serving_started (the utils-side function survived the merge; only the call site was dropped). - entry 200 (sgl-project#43527) CAPACITY PAIR RE-PLANTED: model_runner .logical_max_total_num_tokens passes allocator= again (the configurator's PageInterleavePoolAllocator -> allocator.size branch SURVIVED the merge byte-identical, so the merged tree had the fork predicate with upstream's consumer-side multiplications = DOUBLE-COUNT); the three scheduler * kv_shard_widening multipliers back to the fork form (pool-stats shape, load_inquirer, and the init_req one-page-per-shard subtraction); tp_worker drops page_interleave_shard_size x + import. Upstream's invariant-checker hunk: OBSOLETE (the constructor no longer takes max_total_num_tokens -- absorbed by refactor). - entry 215 (sgl-project#44351/sgl-project#44401/sgl-project#44420): VERIFIED ALREADY PRESENT at the merge (filter_batch-every-step + reset ride, num_held_rows, budget_state FIRST statement, PD-gate deletion all byte-compare to the fork form) -- DEDUPED, no edit. Entry 107's local early-revision of sgl-project#40940 stays superseded (entry 215 rule), and its re-plant here would double-count. - entry 220's/305's pause-window busy accounting + write_fence_stream + is_empty arm: VERIFIED PRESENT (entry 305 landed patch-identical via sgl-project#46127; the fence assignment is the B3-2 controller half) -- DEDUPED. TEST FIXTURE RE-ANCHORS (not B2's: no markers, pure new-upstream-contract drift, per the entry-198 fixture-adaptation precedent): the two hybrid-ssm admission stubs gain disagg_prefill_bootstrap_queue=None (upstream's routing-key bootstrap_queue arg in calc_priority arrived in-range). PRUNE vs BREAKAGE (managers −523sub audit): the three managers test files vanished in-range (test_schedule_policy_dfs_weight, test_scheduler_batch_is _full_latch, test_uno_token_accounting) were ALL deleted by upstream's OWN sgl-project#45672 prune (3438ed6). Two are byte-identical upstream copies = zero fork delta. The third carried a 3-line fork comment/stub delta whose SEMANTICS the re-planted unconditional clear pins: its case was the is_empty-arm mirror of test_scheduler_hybrid_ssm_admission_latch (present, green, hybrid_swa/non-hybrid arms included) -- NOT re-homed, recorded. ACTUALS (damage -> after, CVD=9, prod venv, PYTHONPATH=python, per-file): test_scheduler_hybrid_ssm_admission_retry 4F -> ** 4P ** test_scheduler_hybrid_ssm_admission_latch 3F -> ** 4P ** test_mamba_prefill_cache_boundary 2F/12subF -> ** 5P/12sub EXACT (220 pin) ** test_scheduler_retract_ratio 1F -> ** 2P EXACT (267 pin) ** test_scheduler_init_req_max_new_tokens 1F -> ** 10P/6sub ** test_scheduler_grammar_mixed_tail_sync 1F -> ** 4P ** adjacent scheduler-stub files (held_row, running_batch_finished, pause_generation, disagg_idle, dp_converted, hicache_events) 56P/64sub UNMOVED -- no regression from the clear/catcher/capacity edits. Lint: ruff format + F821/E9 clean on all five files; imports OK (no GPU execution, no push). Co-Authored-By: OpenCode <noreply@opencode.ai>
Motivation
Grammar state is normally advanced during speculative
verify(). Mixed batches run existing decode requests through the target-prefill path, which skipsverify().When grammar overlap is enabled, this can leave the grammar state one step behind while the next token mask is being built. The stale mask may admit a token that the updated grammar state rejects, causing the request to abort with
Grammar accept_token failed.Modifications
Accuracy Tests
Tested with a local constrained-generation reproducer using Qwen3.5-9B, NEXTN/MTP, mixed chunk, grammar constraints, and overlap enabled:
Unit and related regression tests:
test_scheduler_decision_batch_params.py: 5 passedtest_batch_result_processor_spec_grammar.py: 6 passedtest_schedule_batch_out_of_place.py: 4 passedtest_schedule_batch_convert_decode_to_extend.py: 4 passedTotal: 19 passed.
Speed Tests and Profiling
Not run. The additional synchronization is limited to mixed batches whose decode tail contains a grammar-constrained request. Other mixed batches and regular speculative decode batches retain the existing overlap behavior.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #35553531537
Latest PR Test (Extra): ❌ Run #35553531284
Latest PR Test (AMD ROCm 10): ❌ Run #35553531414