Repository navigation
Conversation
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
@JustinTong0323 @CatherineSue, when you have a chance, could one of you review the parser semantics and trigger CI for the current commit (for example with |
…streaming content
When a tool call is cut off by max_tokens, non-streaming responses
leaked raw markup into content while streaming dropped it:
- HermesDetector.detect_and_parse returned the entire original text
(opener tag + partial JSON) whenever any block failed to parse, and
one truncated call also discarded all valid calls before it.
- Qwen25Detector.detect_and_parse leaked a bare trailing <tool_call>
opener (cut before the newline of the full bot_token) into content.
Both streaming paths already drop the markup, and finish_reason
("length") tells the client the call is incomplete, so non-streaming
now drops it too. Complete-but-malformed blocks keep the existing
fallback behavior (original text returned unchanged), and an
unterminated block containing complete JSON is still parsed.
bee7ef9 to
d211268
Compare
|
Rebased cleanly onto current Fresh validation on commit
The PR is GitHub-mergeable and the new upstream CI run has started. @JustinTong0323 @CatherineSue, a parser-semantics review once CI settles would be appreciated. |
|
Thanks @AG0708. Closing this because it has had no updates in 91 days. Reopen it if the work is still relevant. Some directories moved recently, so an older branch may need retargeting: |
Motivation
Fixes #30480.
When a tool call is truncated by
max_tokens, non-streaming responses leak rawtool-call markup into
contentwhile streaming drops it — the same requestdiverges based only on
stream:HermesDetector.detect_and_parsereturns the entire original text(opener tag + partial JSON) whenever any block fails to parse, and the single
outer
trymeans one truncated call also discards all valid calls before it.Qwen25Detector.detect_and_parseleaks a bare trailing<tool_call>opener(output cut before the
\nof the full bot_token) into content.Streaming already drops the markup in both detectors, and
finish_reason(
"length") tells the client the call is incomplete, so non-streaming shoulddrop it too. Same failure class as vllm-project/vllm#47903.
Modifications
hermes_detector.py: parse unterminated blocks (the<tool_call>(.*)regexalternative) under a per-block guard — a truncated block is dropped with a
warning instead of leaking the whole text, and valid calls parsed from other
blocks survive. Behavior deliberately preserved: a complete but malformed
block still returns the original text unchanged
(
test_malformed_json_returns_original_textuntouched), and an unterminatedblock containing complete JSON is still parsed as a call.
qwen25_detector.py: strip a bare trailing opener fromnormal_textwhenthe full bot_token never appeared; a literal
<tool_call>elsewhere incontent is not touched.
opener, valid-call-before-truncated-call survival, unterminated-complete-JSON
parsing, literal opener mid-content untouched, plus a complete-call control.
Note: this adds
test/registered/unit/function_call/test_qwen25_detector.py,which #26386 (open, test-only) also proposes to create — happy to rebase on top
if that lands first.
Full
test/registered/unit/function_call/suite: 396 passed; the 11 failuresin
test_function_call_parser.py(structural-tag tests) fail identically onclean main in my environment and are unrelated.
Accuracy Tests
N/A — no model forward changes.
Speed Tests and Profiling
N/A — parsing-only change on the truncation error path.
Checklist
CI States
Latest PR Test (Base): ❌ Run #31538019489
Latest PR Test (Extra): ❌ Run #31538019406