Repository navigation
Conversation
|
Thanks @zRzRzRzRzRzRzR for the review. @JustinTong0323 (codeowner) could you take a look, and @slin1237 could you tag run-ci so the gates can run? Two files, +89/-3. |
|
/tag-and-rerun-ci |
|
/rerun-failed-ci |
|
CI triage for the red jobs on this run, none touch the two files in this PR: gpu-large (1) and 2-gpu-large (2) died on HuggingFace 429 while downloading test models (test_inkling_unified, test_unified_radix_cache_kl_full); XPU is test_xpu_int4_dense (GPTQ act_order); Xeon is test_subblock_sparse_attention asserting CUDA on a CPU runner; NPU is test_npu_hicache_mha; the rest are check-pr-test-health / finish aggregators. The CPU unit suite that carries the new tests passed. Requested a rerun of the failed jobs. |
|
/rerun-failed-ci |
|
Field data point for prioritisation: on a production GLM-5.3-Flash deployment (8xH200, EAGLE, glm47/glm45 parsers, build from 09-16) an internal client that sends chat_template_kwargs.enable_thinking=false hit this today on 23 of its 76 requests: finish_reason=length with reasoning_tokens=131072, zero answer tokens, ~5 minutes of GPU time each. A trivial repro on that node (prompt 'Reply with the single word OK.', max_tokens=256, enable_thinking=false) returns 'OK. OK. OK. ...' in reasoning until the limit, while the same request without the kwarg answers in 45 tokens. That build still applied the grammar to tool-less requests; current main only does so with tools, which this PR covers. |
GLM-5.3 templates open <think> on every turn and ignore the enable_thinking kwarg, but the full-assistant EBNF for glm47 tool calls was built from the raw kwarg. A client sending enable_thinking=false therefore got a grammar with an empty thinking block whose text rule excludes </think>, so the model could never leave the thinking block and looped in reasoning_content until max_tokens. Use the detected template's reasoning config: when it always thinks, build the grammar with thinking enabled regardless of the kwarg. Toggle templates keep the previous behaviour. Refs sgl-project#41939
fb80f2b to
5a145a9
Compare
|
Rebased onto today's @JustinTong0323 this one already has the GLM maintainer's approval and a green CI run from 10-09. When you get to the function_call queue, the three sibling fixes #43067 (rope override keeps |
|
/rerun-failed-ci |
1 similar comment
|
/rerun-failed-ci |
Preserve the always-thinking GLM grammar fix and its regression tests while adopting upstream effective_tools handling and all other upstream changes.
Motivation
Refs #41939 (the tool-bearing branch of that report; the "loops even with thinking on" reports there look like a separate cause).
With
--tool-call-parser glm47, every request that carries non-strict tools gets the full-assistant EBNF._process_messagesbuilds that grammar from the client's rawchat_template_kwargs.enable_thinking:GLM-5.3 templates ignore that kwarg: the generation prompt always ends with
<|assistant|><think>, which is exactly whytemplate_detectionclassifies them asglm53_always_think(ReasoningToggleConfig(special_case="always")). So a client that still sendsenable_thinking: false(the GLM-4.5/4.7 habit, or a server-wide--default-chat-template-kwargs) gets a grammar withthinking_block ::= ""whose text rule excludes every special token,</think>included. The model is already inside the think block and the grammar never lets it out.ReasonerGrammarBackendpassesfull_assistant_ebnfthrough unwrapped, so nothing else allows the token either.What that looks like on a production SGLang build (
c9a8fba99,zai-org/GLM-5.3-Flash,--reasoning-parser glm45 --tool-call-parser glm47, 8×H200, one non-strictaddtool,temperature 0):contenttool_callsreasoning_contentchat_template_kwargstool_callsadd(a=2, b=2){"enable_thinking": false}lengthI'll make the call. I'll make the call. …Modifications
serving_chat.py: when the detected template always thinks (reasoning_config.always_on), build the glm47 grammar with thinking enabled regardless ofenable_thinking. Toggle templates (GLM-4.5/4.7) keep the previous behaviour, and so does the no-reasoning-parser case for templates that honour the kwarg.--reasoning-parser, so a GLM-5.3 deployment without a reasoning parser is covered too; the raw grammar is what gates</think>in both setups.Accuracy Tests
No model forward code changes. Two regression cases in
test/registered/unit/entrypoints/openai/test_serving_chat.pydrive the real_process_messagespath and compile the returned EBNF with xgrammar:enable_thinking: false, with--reasoning-parser glm45and with none: the grammar now accepts…</think>answerand…</think><tool_call>add…</tool_call>. Both subcases fail onmain.enable_thinking: false: the grammar still rejects a stray</think>.Results (Python 3.12, CPU): on
mainthe always-think case fails in both subtests (SUBFAILED(reasoning_parser='glm45'),SUBFAILED(reasoning_parser=None)); with the fix both cases pass, and the wholetest_serving_chat.pypasses (157 tests, 104 subtests).Token-level check with the real
zai-org/GLM-5.3-Flashtokenizer (same grammar builder as the server,xgrammar.TokenizerInfo.from_huggingface), after feeding a thinking prefix:</think>accepted after thinking text<tool_call>add…</tool_call>accepted after</think>main:thinking_block ::= ""thinking_block ::= thinking_block_content "</think>"I could not restart a patched server on the production GPUs, so the post-fix evidence is the compiled-grammar check above rather than a live run; happy to add one if a reviewer can run it.
cc @JustinTong0323 @zRzRzRzRzRzRzR
Speed Tests and Profiling
Not applicable: only the boolean passed to
generate_glm_grammarchanges.Checklist
CI States
Latest PR Test (Base): ❌ Run #41066107535
Latest PR Test (Extra): ❌ Run #41066106925
Latest PR Test (AMD ROCm 10): ✅ Run #41066107241