Repository navigation
Conversation
nvpohanh
left a comment
There was a problem hiding this comment.
[by Claude Code] Severity: style | Confidence: High
Issue
The end-to-end before/after table under Accuracy Tests has no repro steps. It lacks the server launch command, the exact request body for each row, the GPU type and count, and the TP/DP config. The first row also does not say whether the ignore_eos=true request had no tools or had tools with tool_choice="none". Main already fixes the no-tools case (#39273), so the row needs to say which case it measured.
Fix
Add the sglang.launch_server ... --tool-call-parser glm47 command, the request body (or curl command) for each row, and the GPU and parallel config.
|
@toulzx Could you take a look at the above comments and address those when possible? |
|
@YouNeedCryDear @nvpohanh thanks for review, I will address the above issues. |
--- The GLM47 grammar was installed for requests that disable tool calls, truncating output because grammar termination ends generation without consulting `ignore_eos`.
--- The parser was built from the top-level `tools`, so requests that declare them on a message got a grammar with no functions and every call was rejected.
3c0568c to
314642b
Compare
|
Updated the branch after review.
I did not rebase again after those tests. As a private merge-safety check, I compared |
|
btw, related #42984 updates the same GLM47 grammar path for always-thinking templates. The two fixes address separate and complementary request semantics:
When both changes are integrated, the tool_choice != "none" guard, FunctionCallParser(effective_tools, ...), and the reasoning_config.always_on handling should all be preserved. I reviewed the two diffs for logical compatibility; the combined changes have not been runtime-tested. |
|
/rerun-test test/registered/unit/entrypoints/openai/test_serving_chat.py |
|
🚀 |
|
@JustinTong0323 could you review this? |
|
/tag-and-rerun-ci |
tool_choice="none"
|
[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest Anyone can stop me at any time: just post a comment asking me to stop (for example, |
|
[by Claude Code] The relevant CI failure (https://github.com/sgl-project/sglang/actions/runs/39122195642/job/117700563806) is not caused by this PR. It is a known breakage on |
Picks up sgl-project#45038, which fixes the unrelated CI failure https://github.com/sgl-project/sglang/actions/runs/39122195642/job/117700563806 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest Anyone can stop me at any time: just post a comment asking me to stop (for example, |
|
@toulzx please don't merge with main again to let CI complete. thanks |
|
[by Claude Code] All NV pipelines have passed. |
|
nice |
Motivation
With
--tool-call-parser glm47, a request that supplies tools but explicitly setstool_choice="none"still receives the GLM47full_assistant_ebnfgrammar.That grammar describes a complete assistant turn. Reaching its terminal state finishes generation through the grammar path, which does not consult
ignore_eos. Anignore_eos=truerequest can therefore stop at the model's end-of-turn token instead of reachingmax_tokens, even though the request disabled tool calls.Behavior scope
toolstools+tool_choice="none"tools+tool_choice="auto"tools+tool_choice="required"tools+ named tool choicesystem/developermessagesRelationship to #39273
The original version of this PR covered two cases:
tool_choice="none".While this PR was under review, #39273 independently added the
bool(effective_tools)guard and a no-tools regression test as part of its GLM-5.3 work. Commitf17f7705a5a67c137edf279edfdbd5460f2ceb6etherefore already fixed case 1.This branch was subsequently rebased onto main commit
4b384df0c94dbd0fe30627c27c98c0441b4ebd96, which contains #39273. The duplicated no-tools production change and duplicated test were removed. The remaining core fix in this PR is case 2: tools are available, but the request explicitly disables tool calls withtool_choice="none".The no-tools history is retained here so reports that found the original analysis through this PR can follow that part of the fix to #39273.
Modifications
Scope the GLM47 grammar to requests that provide tools and allow tool calls:
auto,required, and named-tool choices retain their existing grammar behavior. Other tool-call parsers are unchanged.Message-scoped tool consistency
_effective_tools(request)includes tools declared onsystemanddevelopermessages. That effective set decides whether the GLM47 grammar is active, but the parser previously received only top-levelrequest.tools. A request with message-scoped tools could therefore activate a grammar that omitted those functions.The GLM47 parser now receives
effective_tools. The regression test uses the real XGrammar parser and verifies message-scoped tools forauto,required, and named-tool choices, as well as the disablednonecase.Test cleanup
Deliberately not changed:
developerrole.Accuracy Tests
No model forward code or kernel is changed. The checks below validate request semantics and grammar construction rather than model accuracy.
Unit tests
Environment:
Command:
On baseline
4b384df0c94dbd0fe30627c27c98c0441b4ebd96, with the regression test applied but without the production fix:The existing #39273 no-tools test passed. The new matrix failed for the message-scoped-tool cases and, importantly for this PR, for every
tool_choice="none"subtest, including the case with a top-level tool.With the fix applied:
The complete serving-chat unit file was also run on the fixed snapshot:
GLM-5.3 end-to-end reproduction
The E2E comparison specifically tested tools plus
tool_choice="none"; it was not a no-tools test.Source snapshots:
99463390f83a39776d452ba12199941b96a4f36fc5649e50245567827c63ac9d9dc3fafb2334a00bHardware and model:
/models/zai-org/GLM-5.3glm-5.3sglang:dev-cu13-260921tp_size=8,attn_dp_size=8, anddp_size=1.--chunked-prefill-size 32768; the normalized server arguments logged an effectivechunked_prefill_size=4096.SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK=1was set.Server command:
Request, repeated five times against each server:
Results:
matched_stop99463390f8stopfor all runsc5649e5024lengthfor all runsExpected behavior: because the request explicitly disables tool calls and sets
ignore_eos=true, no tool grammar should be installed and generation should continue until the 300-token cap. The fixed snapshot did so in all five runs.Speed Tests and Profiling
No throughput benchmark was run or claimed. Requests that continue to use the GLM47 grammar retain the same grammar-masking path. This PR only removes an inapplicable grammar from requests that explicitly disable tool calls.
Checklist
CI States
Latest PR Test (Base): ✅ Run #39219926986
Latest PR Test (Extra): ❌ Run #39219926899
Latest PR Test (AMD ROCm 10): ❌ Run #39219926930