Skip to content

[Fix] glm47: Disable tool grammar for tool_choice="none" - #41233

Merged
nvpohanh merged 11 commits into
sgl-project:mainfrom
toulzx:fix/glm47-constraint-without-active-tools
Oct 20, 2026
Merged

nvpohanh merged 11 commits into
sgl-project:mainfrom
toulzx:fix/glm47-constraint-without-active-tools

Conversation

@toulzx

@toulzx toulzx commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

With --tool-call-parser glm47, a request that supplies tools but explicitly sets tool_choice="none" still receives the GLM47 full_assistant_ebnf grammar.

That grammar describes a complete assistant turn. Reaching its terminal state finishes generation through the grammar path, which does not consult ignore_eos. An ignore_eos=true request can therefore stop at the model's end-of-turn token instead of reaching max_tokens, even though the request disabled tool calls.

Behavior scope

Request Expected behavior Ownership
No tools Do not install the GLM47 tool-call grammar Already fixed and tested by #39273
tools + tool_choice="none" Do not install the GLM47 tool-call grammar Core fix in this PR
tools + tool_choice="auto" Preserve the existing GLM47 grammar behavior Regression coverage
tools + tool_choice="required" Preserve the existing behavior Regression coverage
tools + named tool choice Preserve named-tool selection semantics Regression coverage
Tools on system/developer messages Include them in the active GLM47 grammar Adjacent consistency fix requested during review

Relationship to #39273

The original version of this PR covered two cases:

  1. requests with no tools; and
  2. requests with tools plus tool_choice="none".

While this PR was under review, #39273 independently added the bool(effective_tools) guard and a no-tools regression test as part of its GLM-5.3 work. Commit f17f7705a5a67c137edf279edfdbd5460f2ceb6e therefore already fixed case 1.

This branch was subsequently rebased onto main commit 4b384df0c94dbd0fe30627c27c98c0441b4ebd96, which contains #39273. The duplicated no-tools production change and duplicated test were removed. The remaining core fix in this PR is case 2: tools are available, but the request explicitly disables tool calls with tool_choice="none".

The no-tools history is retained here so reports that found the original analysis through this PR can follow that part of the fix to #39273.

Modifications

Scope the GLM47 grammar to requests that provide tools and allow tool calls:

glm_constraint = (
    self.tool_call_parser == "glm47"
    and bool(effective_tools)
    and request.tool_choice != "none"
    and not any(tool.function.strict for tool in effective_tools)
)

auto, required, and named-tool choices retain their existing grammar behavior. Other tool-call parsers are unchanged.

Message-scoped tool consistency

_effective_tools(request) includes tools declared on system and developer messages. That effective set decides whether the GLM47 grammar is active, but the parser previously received only top-level request.tools. A request with message-scoped tools could therefore activate a grammar that omitted those functions.

The GLM47 parser now receives effective_tools. The regression test uses the real XGrammar parser and verifies message-scoped tools for auto, required, and named-tool choices, as well as the disabled none case.

Test cleanup

Deliberately not changed:

  • GLM47 grammar generation or grammar-terminal finish handling.
  • Generic Jinja rendering of message-scoped tools, which affects multiple model families and needs separate cross-model coverage.
  • Template compatibility for the developer role.

Accuracy Tests

No model forward code or kernel is changed. The checks below validate request semantics and grammar construction rather than model accuracy.

Unit tests

Environment:

  • Python 3.12.3
  • pytest 9.1.1
  • XGrammar available in the test image

Command:

PYTHONPATH=/tmp/pr41233-src/python python3 -m pytest \
  test/registered/unit/entrypoints/openai/test_serving_chat.py \
  -k glm47 -v

On baseline 4b384df0c94dbd0fe30627c27c98c0441b4ebd96, with the regression test applied but without the production fix:

24 failed, 2 passed, 154 deselected, 16 warnings

The existing #39273 no-tools test passed. The new matrix failed for the message-scoped-tool cases and, importantly for this PR, for every tool_choice="none" subtest, including the case with a top-level tool.

With the fix applied:

2 passed, 154 deselected, 16 warnings, 24 subtests passed

The complete serving-chat unit file was also run on the fixed snapshot:

PYTHONPATH=/tmp/pr41233-src/python python3 -m pytest \
  test/registered/unit/entrypoints/openai/test_serving_chat.py -v
156 passed, 63 warnings, 126 subtests passed in 46.16s

GLM-5.3 end-to-end reproduction

The E2E comparison specifically tested tools plus tool_choice="none"; it was not a no-tools test.

Source snapshots:

  • Before: main commit 99463390f83a39776d452ba12199941b96a4f36f
  • After: the same source snapshot plus the production diff from local fix c5649e50245567827c63ac9d9dc3fafb2334a00b
  • The relevant production code in that fix is identical to the subsequently rebased branch.

Hardware and model:

  • 1 node with 8 x NVIDIA H200 GPUs
  • Model path: /models/zai-org/GLM-5.3
  • Served model name: glm-5.3
  • Container tag: sglang:dev-cu13-260921
  • Launch requested TP=8 and DP-attention DP=8. The effective server log reported tp_size=8, attn_dp_size=8, and dp_size=1.
  • The launch command passed --chunked-prefill-size 32768; the normalized server arguments logged an effective chunked_prefill_size=4096.
  • SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK=1 was set.

Server command:

SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK=1 \
PYTHONPATH=/tmp/pr41233-src/python \
python3 -m sglang.launch_server \
  --model-path /models/zhipuai/GLM-5.3 \
  --served-model-name glm-5.3 \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --context-length 102400 \
  --tp 8 \
  --dp 8 \
  --enable-dp-attention \
  --moe-a2a-backend deepep \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 1 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 2 \
  --chunked-prefill-size 32768 \
  --max-running-requests 16 \
  --mem-fraction-static 0.85

Request, repeated five times against each server:

curl --fail --silent --show-error \
  -H 'Content-Type: application/json' \
  http://127.0.0.1:30000/v1/chat/completions \
  --data-binary @- <<'JSON'
{
  "model": "glm-5.3",
  "messages": [
    {
      "role": "user",
      "content": "Answer exactly with the word OK and nothing else."
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the weather for a city.",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }
  ],
  "tool_choice": "none",
  "temperature": 0,
  "seed": 42,
  "ignore_eos": true,
  "max_tokens": 300,
  "stop": null,
  "stream": false
}
JSON

Results:

Snapshot Completion tokens (5 runs) Finish reason matched_stop Tool calls
Baseline 99463390f8 66, 61, 47, 54, 67 stop for all runs 154827 for all runs none
Fix c5649e5024 300, 300, 300, 300, 300 length for all runs null for all runs none

Expected behavior: because the request explicitly disables tool calls and sets ignore_eos=true, no tool grammar should be installed and generation should continue until the 300-token cap. The fixed snapshot did so in all five runs.

Speed Tests and Profiling

No throughput benchmark was run or claimed. Requests that continue to use the GLM47 grammar retain the same grammar-masking path. This PR only removes an inapplicable grammar from requests that explicitly disable tool calls.

Checklist

  • Format the changed code according to the repository style.
  • Add regression coverage according to the unit-test admission rules.
  • Record the exact source snapshots and commands used for validation.
  • Documentation update. Not needed for this internal request-routing fix.
  • Accuracy/speed benchmark. No model-forward or performance change is claimed; targeted unit and E2E correctness results are provided above.

CI States

Latest PR Test (Base): ✅ Run #39219926986
Latest PR Test (Extra): ❌ Run #39219926899
Latest PR Test (AMD ROCm 10): ❌ Run #39219926930

@nvpohanh nvpohanh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] Severity: style | Confidence: High

Issue

The end-to-end before/after table under Accuracy Tests has no repro steps. It lacks the server launch command, the exact request body for each row, the GPU type and count, and the TP/DP config. The first row also does not say whether the ignore_eos=true request had no tools or had tools with tool_choice="none". Main already fixes the no-tools case (#39273), so the row needs to say which case it measured.

Fix

Add the sglang.launch_server ... --tool-call-parser glm47 command, the request body (or curl command) for each row, and the GPU and parallel config.

@YouNeedCryDear

Copy link
Copy Markdown
Contributor

@toulzx Could you take a look at the above comments and address those when possible?

@toulzx

toulzx commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@YouNeedCryDear @nvpohanh thanks for review, I will address the above issues.

toulzx added 2 commits October 8, 2026 17:04
---
The GLM47 grammar was installed for requests that disable tool calls, truncating
output because grammar termination ends generation without consulting `ignore_eos`.
---
The parser was built from the top-level `tools`, so requests that declare them on
a message got a grammar with no functions and every call was rejected.
@toulzx
toulzx force-pushed the fix/glm47-constraint-without-active-tools branch from 3c0568c to 314642b Compare October 9, 2026 04:00
@toulzx toulzx changed the title [Fix] Scope the GLM47 tool grammar to requests that allow tool calls [Fix] Disable the GLM47 tool grammar for tool_choice="none" Oct 9, 2026
@toulzx

toulzx commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor Author

@YouNeedCryDear @nvpohanh

Updated the branch after review.

  • I rebased it onto 4b384df0c94dbd0fe30627c27c98c0441b4ebd96, which already contains [AMD] [GLM-5.3-Flash] Enable FP8 and MXFP4 serving on gfx950 #39273 (f17f7705a5). That absorbs the no-tools fix from [AMD] [GLM-5.3-Flash] Enable FP8 and MXFP4 serving on gfx950 #39273 and leaves this PR focused on the remaining tools + tool_choice="none" case, plus the previously reviewed message-scoped-tool consistency fix.
  • I removed the two redundant tests called out in review and kept the real-parser matrix.
  • I added the required black-box regression docstring.
  • The unit and E2E results in the description remain tied to the exact source snapshots on which they were run; I did not relabel them as results from a newer main.

I did not rebase again after those tests. As a private merge-safety check, I compared 4b384df0 with current upstream main 3e13f1b2: neither changed file was modified in the intervening 60 commits, and git merge-tree reports a clean merge for the updated candidate. By contrast, the old pre-#39273 PR head conflicts in both files, which is why the earlier rebase was retained.

@toulzx toulzx closed this Oct 9, 2026
@toulzx toulzx reopened this Oct 9, 2026
@toulzx

toulzx commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

btw, related #42984 updates the same GLM47 grammar path for always-thinking templates. The two fixes address separate and complementary request semantics:

When both changes are integrated, the tool_choice != "none" guard, FunctionCallParser(effective_tools, ...), and the reasoning_config.always_on handling should all be preserved. I reviewed the two diffs for logical compatibility; the combined changes have not been runtime-tested.

@nvpohanh

Copy link
Copy Markdown
Collaborator

/rerun-test test/registered/unit/entrypoints/openai/test_serving_chat.py

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/entrypoints/openai/test_serving_chat.py

@nvpohanh

Copy link
Copy Markdown
Collaborator

@JustinTong0323 could you review this?

@JustinTong0323

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 16, 2026

@JustinTong0323 JustinTong0323 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@toulzx toulzx changed the title [Fix] Disable the GLM47 tool grammar for tool_choice="none" [Fix] glm47: Disable tool grammar for tool_choice="none" Oct 16, 2026
@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest main into this branch, push a small test-only fix, or post a note here.

Anyone can stop me at any time: just post a comment asking me to stop (for example, stop babysitting). I'll stop at my next check, within 30 minutes, and take no further action on this PR.

@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] The relevant CI failure (https://github.com/sgl-project/sglang/actions/runs/39122195642/job/117700563806) is not caused by this PR. It is a known breakage on main, tracked in #45010. CI for this PR is blocked until that is fixed.

@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest main into this branch, push a small test-only fix, or post a note here.

Anyone can stop me at any time: just post a comment asking me to stop (for example, stop babysitting). I'll stop at my next check, within 30 minutes, and take no further action on this PR.

@nvpohanh

Copy link
Copy Markdown
Collaborator

@toulzx please don't merge with main again to let CI complete. thanks

@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] All NV pipelines have passed.

@nvpohanh
nvpohanh merged commit 79ed13f into sgl-project:main Oct 20, 2026
330 of 358 checks passed
@tianlang-wq

Copy link
Copy Markdown

nice

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants