Skip to content

[Spec] Hold the grammar bitmask in one GrammarMask type across all decode paths - #32409

Merged
hnyls2002 merged 5 commits into
mainfrom
lsyin/grammar-outparam
Jul 25, 2026
Merged

hnyls2002 merged 5 commits into
mainfrom
lsyin/grammar-outparam

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Jul 25, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up cleanup to #30096 / #31753.

  • generate_token_bitmask stamped its grammar onto a caller-supplied spec input purely to hand it back, so four classes carried a field written and read within one verify. It is returned alongside the bitmask now, and the four declarations are gone.
  • SamplingBatchInfo kept the same pair as two loose fields (vocab_mask + apply_mask_func, a bound method whose owner was invisible to the type). Both paths now hold one GrammarMask (bitmask + the backend that applies it), defined next to BaseGrammarObject.
  • The grammar barrier moves into build_grammar_vocab_mask, so "advance the FSM before traversing" lives at the one place it matters instead of being repeated at each worker.

CI States

Latest PR Test (Base): 🚫 Run #30177166305
Latest PR Test (Extra): ❌ Run #30177166225

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@hnyls2002
hnyls2002 requested a review from Fridge003 as a code owner July 25, 2026 21:58
@hnyls2002 hnyls2002 changed the title [Spec] Return the grammar with the bitmask instead of stamping it on spec inputs [Spec] Hold the grammar bitmask in one GrammarMask type across all decode paths Jul 25, 2026
@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/rerun-test test_sampling_batch_info.py test_constrained_decoding.py test_constrained_decoding_spec_reasoning.py test_eagle_constrained_decoding.py test_spec_eagle.py test_spec_ngram.py test_spec_standalone.py test_frozen_kv_mtp.py test_dflash.py test_basic_sanity_dspark.py

@github-actions

github-actions Bot commented Jul 25, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test_sampling_batch_info.py test_constrained_decoding.py test_constrained_decoding_spec_reasoning.py test_eagle_constrained_decoding.py test_spec_eagle.py test_spec_ngram.py test_spec_standalone.py test_frozen_kv_mtp.py test_dflash.py test_basic_sanity_dspark.py:

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/sampling/test_sampling_batch_info.py

🚀 1-gpu-5090 (3 tests): ✅ View workflow run

cd test/ && python3 registered/constrained_decoding/test_constrained_decoding.py
cd test/ && python3 registered/spec/eagle/test_spec_eagle.py
cd test/ && python3 registered/spec/dflash/test_dflash.py

🚀 2-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/spec/test_constrained_decoding_spec_reasoning.py

🚀 1-gpu-h100 (5 tests): ✅ View workflow run

cd test/ && python3 registered/spec/eagle/test_eagle_constrained_decoding.py
cd test/ && python3 registered/spec/test_spec_ngram.py
cd test/ && python3 registered/spec/test_spec_standalone.py
cd test/ && python3 registered/spec/test_frozen_kv_mtp.py
cd test/ && python3 registered/core/test_basic_sanity_dspark.py

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Jul 25, 2026
@hnyls2002
hnyls2002 merged commit 3da1071 into main Jul 25, 2026
76 of 117 checks passed
@hnyls2002
hnyls2002 deleted the lsyin/grammar-outparam branch July 25, 2026 22:27
Atituiset pushed a commit to Atituiset/sglang that referenced this pull request Sep 10, 2026
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 24, 2026
…mmar with DFlash (group D); sgl-project#31488/sgl-project#32409 checked

Applicability (27B weg2 serving): the front forwards the whole client body to
D in leg 2, and D validated every request with validate_dflash_request,
which ABORTED return_logprob and every grammar (json_schema / regex / ebnf /
structural_tag). An Anthropic tool_choice any|tool becomes OpenAI
required/named and a structure constraint (serving_chat), so such agent
requests and every logprob request died on D. Full-feature serving -> port.

sgl-project#33459 (36853b8): validate_dflash_request admits return_logprob for DFLASH
(DSpark keeps the refusal, as upstream); the ValueError at the top of
DFlashWorkerV2.forward_batch_generation is gone; the verify computes the
committed run's logprobs with the fork's compute_spec_v2_logprobs
(output_indices = arange(bs*block).view(bs, block), steps = block-1: row j
predicts out_tokens[:, j]) -> the spec-v2 layout batch_result_processor.
_apply_decode_logprobs already reads (next_token_logprobs[i][:commit_len]).

sgl-project#30096 (d021990), ADAPTED: upstream builds a GrammarTree and overlaps the
FSM advance via the sgl-project#31488 grammar barrier; the fork has neither and runs
EAGLE v2 grammar on the synchronous generate_token_bitmask path (overlap is
already disabled for spec+grammar decode batches by
is_disable_overlap_for_batch, and the result processor already advances the
grammar for any spec-v2 result). So: DFlashVerifyInput.grammar field; the
rank-synced draft tokens go to the host before the verify launch (grammar
batches only); after the forward, _dflash_grammar_vocab_mask walks the
LINEAR block (next = i+1, no siblings) with generate_token_bitmask, clears a
stale sampling_info.vocab_mask (as EAGLE does), and the mask is applied to
the verify logits after apply_dflash_verify_logits_adjustments and before
any accept path (greedy/Triton, sampling, selector). Masks are identical on
every rank (synced tokens, identical grammar states); accepts stay synced.
validate_dflash_request admits grammar for DFLASH, DSpark keeps refusing.
The scheduler passes its spec_algorithm (new optional 3rd arg).

sgl-project#31488 (e7e8aaa, overlap grammar with spec verify: grammar barrier,
GrammarTree, eagle/frozen-kv/multi-layer workers, 214 lines) and sgl-project#32409
(3da1071, GrammarMask type across 16 files) are checked and NOT ported:
throughput/refactor over infrastructure the fork does not have; the
synchronous form above is correct without them.

Risk: none for requests without logprobs/grammar (both paths are gated on
batch.return_logprob / batch.has_grammar). Grammar batches pay one D2H of
the draft block and a CPU chain walk per verify step.

Tests (CPU, incg 3 GiB cgroup):
- new test_dflash_logprobs_grammar_33459_30096.py (7): base 1e49837
  6 failed / 1 passed (the unchanged hidden-states refusal), with the port
  7 passed.
- base == port: test_spec_utils_traverse_tree 1/1,
  test_batch_result_processor_spec_grammar 2/2; test_dflash_*.py group
  66 passed (base 61 + the 5 red sgl-project#37818 tests of this branch).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant