Skip to content

[DFlash] Support grammar constrained decoding support for DFlash - #28680

Open
dcw02 wants to merge 1 commit into
mainfrom
dcw02/dflash-grammar-support
Open

dcw02 wants to merge 1 commit into
mainfrom
dcw02/dflash-grammar-support

Conversation

@dcw02

@dcw02 dcw02 commented Jun 18, 2026 •

Copy link
Copy Markdown
Collaborator
  • Enables grammar-constrained decoding during DFlash speculative verification.
  • Adds reusable DFlash grammar mask handling for JSON/grammar token filtering.
  • Covers DFlash spec-v2 constrained decoding through the existing spec feature kit.

Server command:

SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 venv/bin/python -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --host 127.0.0.1 --port 20554 \
    --trust-remote-code \
    --attention-backend trtllm_mha \
    --tp-size 1 \
    --dtype bfloat16 \
    --max-running-requests 32 \
    --cuda-graph-max-bs 32 \
    --mamba-scheduler-strategy no_buffer \
    --enforce-piecewise-cuda-graph \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path z-lab/Qwen3-8B-DFlash-b16 \
    --speculative-draft-attention-backend fa4 \
    --grammar-backend xgrammar

Question:

Janet's ducks lay 16 eggs per day. She eats 3 for breakfast and uses 4 to bake muffins. She sells the rest for $2 per egg. How much money does she make every day?

Unconstrained request:

sampling_params = {
    "temperature": 0.0,
    "top_p": 1.0,
    "top_k": 1,
    "max_new_tokens": 1024
}

Unconstrained output:

... Janet makes \boxed{18} dollars every day.

Grammar-constrained request:

sampling_params = {
    "temperature": 0.0,
    "top_p": 1.0,
    "top_k": 1,
    "max_new_tokens": 1024,
    "json_schema": "{\"type\":\"object\",\"properties\":{\"reasoning\":{\"type\":\"string\"},\"answer\":{\"type\":\"integer\"}},\"required\":
    [\"reasoning\",\"answer\"],\"additionalProperties\":false}"
}

Grammar-constrained output:

{
    "reasoning": "Janet's ducks lay 16 eggs per day. She uses 3 eggs for breakfast and 4 eggs for muffins, so she uses 3 + 4 = 7 eggs. The remaining eggs
    are 16 - 7 = 9 eggs. She sells these 9 eggs at $2 per egg, so the total money made is 9 * 2 = $18.",
    "answer": 18
}

CI States

Latest PR Test (Base): ❌ Run #27791182071
Latest PR Test (Extra): ❌ Run #27791181896
Latest PR Test (AMD ROCm 10): ❌ Run #27791182028

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@dcw02

dcw02 commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator Author

/rerun-test test/registered/spec/dflash/test_dflash.py

@github-actions

github-actions Bot commented Jun 18, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/spec/dflash/test_dflash.py:

🚀 1-gpu-5090 (1 test): ❌ View workflow run

cd test/ && python3 registered/spec/dflash/test_dflash.py

@dcw02

dcw02 commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator Author

/rerun-test test/registered/spec/dflash/test_dflash.py

@github-actions

github-actions Bot commented Jun 18, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/spec/dflash/test_dflash.py:

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/spec/dflash/test_dflash.py

@pamanseau

Copy link
Copy Markdown

Heads-up that this PR appears to be superseded: #30096 ("[DFLASH] Support grammar-constrained decoding in speculative verify") merged on 2026-07-25 with a different implementation — the GrammarMask type applied in dflash_worker_v2.py via build_grammar_vocab_mask / GrammarTree.from_linear_chain (from #32409) — and that is what's live on main today.

The helpers introduced here (generate_dflash_linear_vocab_mask, reset_dflash_vocab_mask_to_all_allowed, _dflash_vocab_mask_allows_token) never landed on main, and the branch is now conflicting.

Unless there's capability here that #30096 doesn't cover, this can probably be closed. Residual bug in the merged verify path is being tracked in #43514 / #43518.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants