Skip to content

fix(function_call): stop leaking truncated tool-call markup into non-streaming content - #30481

Closed
AG0708 wants to merge 1 commit into
sgl-project:mainfrom
AG0708:fix/truncated-tool-call-markup-leak
Closed

AG0708 wants to merge 1 commit into
sgl-project:mainfrom
AG0708:fix/truncated-tool-call-markup-leak

Conversation

@AG0708

@AG0708 AG0708 commented Jul 8, 2026 •

Copy link
Copy Markdown

Motivation

Fixes #30480.

When a tool call is truncated by max_tokens, non-streaming responses leak raw
tool-call markup into content while streaming drops it — the same request
diverges based only on stream:

  • HermesDetector.detect_and_parse returns the entire original text
    (opener tag + partial JSON) whenever any block fails to parse, and the single
    outer try means one truncated call also discards all valid calls before it.
  • Qwen25Detector.detect_and_parse leaks a bare trailing <tool_call> opener
    (output cut before the \n of the full bot_token) into content.

Streaming already drops the markup in both detectors, and finish_reason
("length") tells the client the call is incomplete, so non-streaming should
drop it too. Same failure class as vllm-project/vllm#47903.

Modifications

  • hermes_detector.py: parse unterminated blocks (the <tool_call>(.*) regex
    alternative) under a per-block guard — a truncated block is dropped with a
    warning instead of leaking the whole text, and valid calls parsed from other
    blocks survive. Behavior deliberately preserved: a complete but malformed
    block still returns the original text unchanged
    (test_malformed_json_returns_original_text untouched), and an unterminated
    block containing complete JSON is still parsed as a call.
  • qwen25_detector.py: strip a bare trailing opener from normal_text when
    the full bot_token never appeared; a literal <tool_call> elsewhere in
    content is not touched.
  • Unit tests for both (7 new tests): truncation mid-arguments, bare trailing
    opener, valid-call-before-truncated-call survival, unterminated-complete-JSON
    parsing, literal opener mid-content untouched, plus a complete-call control.

Note: this adds test/registered/unit/function_call/test_qwen25_detector.py,
which #26386 (open, test-only) also proposes to create — happy to rebase on top
if that lands first.

$ PYTHONPATH=python python -m pytest test/registered/unit/function_call/test_hermes_detector.py test/registered/unit/function_call/test_qwen25_detector.py -q
19 passed in 1.98s

Full test/registered/unit/function_call/ suite: 396 passed; the 11 failures
in test_function_call_parser.py (structural-tag tests) fail identically on
clean main in my environment and are unrelated.

Accuracy Tests

N/A — no model forward changes.

Speed Tests and Profiling

N/A — parsing-only change on the truncation error path.

Checklist


CI States

Latest PR Test (Base): ❌ Run #31538019489
Latest PR Test (Extra): ❌ Run #31538019406

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@AG0708

AG0708 commented Jul 10, 2026

Copy link
Copy Markdown
Author

@JustinTong0323 @CatherineSue, when you have a chance, could one of you review the parser semantics and trigger CI for the current commit (for example with /tag-and-rerun-ci)? The repository lint job passed, and locally the two focused files pass (19 tests) along with the full test/registered/unit/function_call/ suite (396 tests; the 11 pre-existing structural-tag failures reproduce on clean main). No GPU or model evaluation is needed for this parser-only truncation path. Thanks.

…streaming content

When a tool call is cut off by max_tokens, non-streaming responses
leaked raw markup into content while streaming dropped it:

- HermesDetector.detect_and_parse returned the entire original text
  (opener tag + partial JSON) whenever any block failed to parse, and
  one truncated call also discarded all valid calls before it.
- Qwen25Detector.detect_and_parse leaked a bare trailing <tool_call>
  opener (cut before the newline of the full bot_token) into content.

Both streaming paths already drop the markup, and finish_reason
("length") tells the client the call is incomplete, so non-streaming
now drops it too. Complete-but-malformed blocks keep the existing
fallback behavior (original text returned unchanged), and an
unterminated block containing complete JSON is still parsed.
@AG0708
AG0708 force-pushed the fix/truncated-tool-call-markup-leak branch from bee7ef9 to d211268 Compare August 11, 2026 21:28
@AG0708

AG0708 commented Aug 11, 2026

Copy link
Copy Markdown
Author

Rebased cleanly onto current main (59450c4f1); the reported Hermes/Qwen25 bug is still present there, and a fresh search found no competing fix. The related #31786 changes only DeepSeek streaming behavior.

Fresh validation on commit d21126848:

  • focused Hermes + Qwen25 files: 19 passed
  • remaining function-call suite in the CPU-isolated harness: 427 passed, 15 deselected, 14 subtests passed (the deselected DeepSeek cases hit an unrelated local Transformers registration conflict)
  • all changed-file pre-commit hooks passed, including ruff, black, isort, codespell, and registered-test validation
  • scripts/ci/check_registered_tests.py and git diff --check passed

The PR is GitHub-mergeable and the new upstream CI run has started. @JustinTong0323 @CatherineSue, a parser-semantics review once CI settles would be appreciated.

@github-actions

Copy link
Copy Markdown
Contributor

Thanks @AG0708. Closing this because it has had no updates in 91 days.

Reopen it if the work is still relevant.

Some directories moved recently, so an older branch may need retargeting:
sgl-kernel/ -> python/sglang/kernels/aot/, python/sglang/jit_kernel/
-> python/sglang/kernels/jit/, docs/ -> docs/docs/ (.mdx),
bench_serving.py -> benchmark/serving.py, test/srt/ -> test/registered/.

@github-actions github-actions Bot closed this Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Non-streaming leaks raw <tool_call> markup into content when a tool call is truncated by max_tokens (hermes, qwen25)

1 participant