Repository navigation
[Spec] Hold the grammar bitmask in one GrammarMask type across all decode paths - #32409
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
GrammarMask type across all decode paths
|
/rerun-test test_sampling_batch_info.py test_constrained_decoding.py test_constrained_decoding_spec_reasoning.py test_eagle_constrained_decoding.py test_spec_eagle.py test_spec_ngram.py test_spec_standalone.py test_frozen_kv_mtp.py test_dflash.py test_basic_sanity_dspark.py |
|
Results for 🚀 🚀 🚀 🚀 |
|
/tag-and-rerun-ci |
…mmar with DFlash (group D); sgl-project#31488/sgl-project#32409 checked Applicability (27B weg2 serving): the front forwards the whole client body to D in leg 2, and D validated every request with validate_dflash_request, which ABORTED return_logprob and every grammar (json_schema / regex / ebnf / structural_tag). An Anthropic tool_choice any|tool becomes OpenAI required/named and a structure constraint (serving_chat), so such agent requests and every logprob request died on D. Full-feature serving -> port. sgl-project#33459 (36853b8): validate_dflash_request admits return_logprob for DFLASH (DSpark keeps the refusal, as upstream); the ValueError at the top of DFlashWorkerV2.forward_batch_generation is gone; the verify computes the committed run's logprobs with the fork's compute_spec_v2_logprobs (output_indices = arange(bs*block).view(bs, block), steps = block-1: row j predicts out_tokens[:, j]) -> the spec-v2 layout batch_result_processor. _apply_decode_logprobs already reads (next_token_logprobs[i][:commit_len]). sgl-project#30096 (d021990), ADAPTED: upstream builds a GrammarTree and overlaps the FSM advance via the sgl-project#31488 grammar barrier; the fork has neither and runs EAGLE v2 grammar on the synchronous generate_token_bitmask path (overlap is already disabled for spec+grammar decode batches by is_disable_overlap_for_batch, and the result processor already advances the grammar for any spec-v2 result). So: DFlashVerifyInput.grammar field; the rank-synced draft tokens go to the host before the verify launch (grammar batches only); after the forward, _dflash_grammar_vocab_mask walks the LINEAR block (next = i+1, no siblings) with generate_token_bitmask, clears a stale sampling_info.vocab_mask (as EAGLE does), and the mask is applied to the verify logits after apply_dflash_verify_logits_adjustments and before any accept path (greedy/Triton, sampling, selector). Masks are identical on every rank (synced tokens, identical grammar states); accepts stay synced. validate_dflash_request admits grammar for DFLASH, DSpark keeps refusing. The scheduler passes its spec_algorithm (new optional 3rd arg). sgl-project#31488 (e7e8aaa, overlap grammar with spec verify: grammar barrier, GrammarTree, eagle/frozen-kv/multi-layer workers, 214 lines) and sgl-project#32409 (3da1071, GrammarMask type across 16 files) are checked and NOT ported: throughput/refactor over infrastructure the fork does not have; the synchronous form above is correct without them. Risk: none for requests without logprobs/grammar (both paths are gated on batch.return_logprob / batch.has_grammar). Grammar batches pay one D2H of the draft block and a CPU chain walk per verify step. Tests (CPU, incg 3 GiB cgroup): - new test_dflash_logprobs_grammar_33459_30096.py (7): base 1e49837 6 failed / 1 passed (the unchanged hidden-states refusal), with the port 7 passed. - base == port: test_spec_utils_traverse_tree 1/1, test_batch_result_processor_spec_grammar 2/2; test_dflash_*.py group 66 passed (base 61 + the 5 red sgl-project#37818 tests of this branch).
Follow-up cleanup to #30096 / #31753.
generate_token_bitmaskstamped its grammar onto a caller-supplied spec input purely to hand it back, so four classes carried a field written and read within one verify. It is returned alongside the bitmask now, and the four declarations are gone.SamplingBatchInfokept the same pair as two loose fields (vocab_mask+apply_mask_func, a bound method whose owner was invisible to the type). Both paths now hold oneGrammarMask(bitmask + the backend that applies it), defined next toBaseGrammarObject.build_grammar_vocab_mask, so "advance the FSM before traversing" lives at the one place it matters instead of being repeated at each worker.CI States
Latest PR Test (Base): 🚫 Run #30177166305
Latest PR Test (Extra): ❌ Run #30177166225