Skip to content

[Fix] glm47: Let the GLM47 tool grammar follow always-thinking templates (GLM-5.3) - #42984

Open
jinwukong wants to merge 2 commits into
sgl-project:mainfrom
jinwukong:fix/glm47-grammar-always-think
Open

jinwukong wants to merge 2 commits into
sgl-project:mainfrom
jinwukong:fix/glm47-grammar-always-think

Conversation

@jinwukong

@jinwukong jinwukong commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Refs #41939 (the tool-bearing branch of that report; the "loops even with thinking on" reports there look like a separate cause).

With --tool-call-parser glm47, every request that carries non-strict tools gets the full-assistant EBNF. _process_messages builds that grammar from the client's raw chat_template_kwargs.enable_thinking:

thinking_mode=True if enable_thinking is None else bool(enable_thinking)

GLM-5.3 templates ignore that kwarg: the generation prompt always ends with <|assistant|><think>, which is exactly why template_detection classifies them as glm53_always_think (ReasoningToggleConfig(special_case="always")). So a client that still sends enable_thinking: false (the GLM-4.5/4.7 habit, or a server-wide --default-chat-template-kwargs) gets a grammar with thinking_block ::= "" whose text rule excludes every special token, </think> included. The model is already inside the think block and the grammar never lets it out. ReasonerGrammarBackend passes full_assistant_ebnf through unwrapped, so nothing else allows the token either.

What that looks like on a production SGLang build (c9a8fba99, zai-org/GLM-5.3-Flash, --reasoning-parser glm45 --tool-call-parser glm47, 8×H200, one non-strict add tool, temperature 0):

request finish_reason completion_tokens content tool_calls reasoning_content
no chat_template_kwargs tool_calls 40 empty add(a=2, b=2) 87 chars
{"enable_thinking": false} length 1024 (max) empty none 4107 chars, ends in I'll make the call. I'll make the call. …

Modifications

  • serving_chat.py: when the detected template always thinks (reasoning_config.always_on), build the glm47 grammar with thinking enabled regardless of enable_thinking. Toggle templates (GLM-4.5/4.7) keep the previous behaviour, and so does the no-reasoning-parser case for templates that honour the kwarg.
  • Unlike [Bugfix] Use template-aware thinking mode for GLM47 tool grammar #42761 (thanks @kflansburg for the earlier diagnosis), this keys off the template config rather than --reasoning-parser, so a GLM-5.3 deployment without a reasoning parser is covered too; the raw grammar is what gates </think> in both setups.

Accuracy Tests

No model forward code changes. Two regression cases in test/registered/unit/entrypoints/openai/test_serving_chat.py drive the real _process_messages path and compile the returned EBNF with xgrammar:

  • always-think template + enable_thinking: false, with --reasoning-parser glm45 and with none: the grammar now accepts …</think>answer and …</think><tool_call>add…</tool_call>. Both subcases fail on main.
  • toggle template + enable_thinking: false: the grammar still rejects a stray </think>.

Results (Python 3.12, CPU): on main the always-think case fails in both subtests (SUBFAILED(reasoning_parser='glm45'), SUBFAILED(reasoning_parser=None)); with the fix both cases pass, and the whole test_serving_chat.py passes (157 tests, 104 subtests).

Token-level check with the real zai-org/GLM-5.3-Flash tokenizer (same grammar builder as the server, xgrammar.TokenizerInfo.from_huggingface), after feeding a thinking prefix:

grammar </think> accepted after thinking text <tool_call>add…</tool_call> accepted after </think> turn can end
main: thinking_block ::= "" no no no
this PR: thinking_block ::= thinking_block_content "</think>" yes yes yes

I could not restart a patched server on the production GPUs, so the post-fix evidence is the compiled-grammar check above rather than a live run; happy to add one if a reviewer can run it.

cc @JustinTong0323 @zRzRzRzRzRzRzR

Speed Tests and Profiling

Not applicable: only the boolean passed to generate_glm_grammar changes.

Checklist


CI States

Latest PR Test (Base): ❌ Run #41066107535
Latest PR Test (Extra): ❌ Run #41066106925
Latest PR Test (AMD ROCm 10): ✅ Run #41066107241

@jinwukong

Copy link
Copy Markdown
Contributor Author

Thanks @zRzRzRzRzRzRzR for the review. @JustinTong0323 (codeowner) could you take a look, and @slin1237 could you tag run-ci so the gates can run? Two files, +89/-3.

@JustinTong0323

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 8, 2026
@jinwukong

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@jinwukong

Copy link
Copy Markdown
Contributor Author

CI triage for the red jobs on this run, none touch the two files in this PR: gpu-large (1) and 2-gpu-large (2) died on HuggingFace 429 while downloading test models (test_inkling_unified, test_unified_radix_cache_kl_full); XPU is test_xpu_int4_dense (GPTQ act_order); Xeon is test_subblock_sparse_attention asserting CUDA on a CPU runner; NPU is test_npu_hicache_mha; the rest are check-pr-test-health / finish aggregators. The CPU unit suite that carries the new tests passed. Requested a rerun of the failed jobs.

@zRzRzRzRzRzRzR

Copy link
Copy Markdown
Contributor

/rerun-failed-ci

@jinwukong

Copy link
Copy Markdown
Contributor Author

Field data point for prioritisation: on a production GLM-5.3-Flash deployment (8xH200, EAGLE, glm47/glm45 parsers, build from 09-16) an internal client that sends chat_template_kwargs.enable_thinking=false hit this today on 23 of its 76 requests: finish_reason=length with reasoning_tokens=131072, zero answer tokens, ~5 minutes of GPU time each. A trivial repro on that node (prompt 'Reply with the single word OK.', max_tokens=256, enable_thinking=false) returns 'OK. OK. OK. ...' in reasoning until the limit, while the same request without the kwarg answers in 45 tokens. That build still applied the grammar to tool-less requests; current main only does so with tools, which this PR covers.

GLM-5.3 templates open <think> on every turn and ignore the
enable_thinking kwarg, but the full-assistant EBNF for glm47 tool calls
was built from the raw kwarg. A client sending enable_thinking=false
therefore got a grammar with an empty thinking block whose text rule
excludes </think>, so the model could never leave the thinking block
and looped in reasoning_content until max_tokens.

Use the detected template's reasoning config: when it always thinks,
build the grammar with thinking enabled regardless of the kwarg. Toggle
templates keep the previous behaviour.

Refs sgl-project#41939
@jinwukong
jinwukong force-pushed the fix/glm47-grammar-always-think branch from fb80f2b to 5a145a9 Compare October 12, 2026 09:01
@jinwukong

Copy link
Copy Markdown
Contributor Author

Rebased onto today's main (450837e); the diff is unchanged and test/registered/unit/entrypoints/openai/test_serving_chat.py passes locally (154 passed).

@JustinTong0323 this one already has the GLM maintainer's approval and a green CI run from 10-09. When you get to the function_call queue, the three sibling fixes #43067 (rope override keeps rope_theta), #43119 and #43120 (two Kimi-K3 strict-grammar holes) are rebased the same way and still waiting for run-ci.

@jinwukong

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

1 similar comment
@jinwukong

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

Preserve the always-thinking GLM grammar fix and its regression tests while adopting upstream effective_tools handling and all other upstream changes.
@github-actions

This comment has been minimized.

@jinwukong jinwukong changed the title [Bugfix] Let the GLM47 tool grammar follow always-thinking templates (GLM-5.3) [Fix] glm47: Let the GLM47 tool grammar follow always-thinking templates (GLM-5.3) Oct 31, 2026

@zRzRzRzRzRzRzR zRzRzRzRzRzRzR left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The grammar should follow the template’s actual thinking behavior.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants