Skip to content

[Fix] Reasoning parser: bare </think> from hybrid thinking models leaks into content - #36285

Open
ChenZiHong-Gavin wants to merge 3 commits into
sgl-project:mainfrom
ChenZiHong-Gavin:fix-dangling-think-end-leak
Open

ChenZiHong-Gavin wants to merge 3 commits into
sgl-project:mainfrom
ChenZiHong-Gavin:fix-dangling-think-end-leak

Conversation

@ChenZiHong-Gavin

@ChenZiHong-Gavin ChenZiHong-Gavin commented Aug 25, 2026 •

Copy link
Copy Markdown

Motivation

Hybrid thinking models (thinks_internally=True: DeepSeek-V4, Qwen3, GLM-4.5, ...) keep the opening <think> tag in the chat template, so a model that enters thinking on its own (thinking not enabled for the request) emits only reasoning followed by a bare closing tag:

Let me reconcile the evidence... Let's write the answer.</think>The root cause is...

The base detector is not in reasoning state and never sees an opening tag, so it routes the entire chain-of-thought plus the dangling </think> into content, and reasoning_content stays empty. Observed live on DeepSeek-V4-Flash served with --reasoning-parser deepseek-v4 (reproduced 7/7 on complex prompts that trip spontaneous thinking); downstream consumers of content display the raw chain-of-thought to end users.

Related earlier reports of the same state-desync family: #17551, #17943, #4711, #15446. This PR addresses the direction where the output lands in content (the inverse of #17551).

Modifications

BaseReasoningFormatDetector: treat a bare think_end_token as the close of template-opened reasoning, for hybrid models only.

  • Non-streaming: text containing a bare closer splits around it — prefix becomes reasoning_content, suffix content.
  • Streaming: when the closer shows up in the not-in-reasoning path, reclassify what is still buffered as reasoning, strip the tag, and emit the rest as content (chunks already streamed out are unrecoverable, so streaming is best-effort by nature). A tail that could be a split closing tag is held back and flushed by finish() if it never completes.
  • Guards — the handling turns itself off as soon as anything contradicts the dangling-closer reading:
    • detector is not thinks_internally (DeepSeek-R1-style force detectors already handle this shape natively);
    • reasoning was explicitly entered (stripped_think_start) or already closed (_saw_think_end), or force_reasoning is set;
    • an opening tag has passed through as content — tracked across chunk boundaries with a rolling tail — which makes a later bare closer quoted markup, not a boundary.

The existing chunk-size-invariance tests (TestStreamingChunkSizeInvariance) pass unchanged: the guards were shaped so quoted-tag and tool-interrupt divergence sets stay exactly as documented.

Accuracy Tests

Not applicable (no model forward changes). Parser unit tests: test/registered/unit/parser/test_reasoning_parser.py — 119 tests pass, including 7 new ones covering: non-streaming split, non-hybrid detectors unchanged, closer-after-real-block stays content, streaming single-chunk split, tag split across chunks, closer after closed reasoning stays content, held-back partial flushes on finish.

Speed Tests and Profiling

Not applicable — one extra find on the not-in-reasoning streaming path for hybrid detectors, plus a rolling tail bounded by len(think_start_token).

Checklist

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #32930443543
Latest PR Test (Extra): ❌ Run #32930443359
Latest PR Test (AMD ROCm 7.2): ❌ Run #32930443472

…ontent

Hybrid thinking models (thinks_internally=True: DeepSeek-V4, Qwen3,
GLM-4.5, ...) keep the opening <think> tag in the chat template, so a
model that enters thinking on its own emits only reasoning followed by
a bare closing tag. The base detector, not being in reasoning state and
never seeing an opening tag, routed the entire chain-of-thought plus the
dangling </think> into normal content (observed live on
DeepSeek-V4-Flash: content = 'thoughts...</think>answer', no
reasoning_content).

Treat a bare think_end_token as the close of template-opened reasoning:
non-streaming splits the text around it; streaming reclassifies what is
still buffered, strips the tag, and holds back a possible split tag
suffix. Off unless thinks_internally, and disabled as soon as anything
contradicts that reading: reasoning was explicitly entered or already
closed, force_reasoning is set, or an opening tag has passed through as
content (tracked across chunk boundaries), which makes a later bare
closer quoted markup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ChenZiHong-Gavin

Copy link
Copy Markdown
Author

Hi maintainers, this PR is ready for CI — could someone add the run-ci label (or trigger /tag-and-rerun-ci)? All failing checks so far are just the gate reporting the missing label; no tests have run yet.

For context: this fixes reasoning leaking into content (with a bare </think>) when a hybrid thinking model enters think mode spontaneously — the parser unit suite passes locally (119 tests, 7 new). Happy to address any review feedback. Thanks!

@ChenZiHong-Gavin

Copy link
Copy Markdown
Author

Reopening — I closed this by mistake; nothing about the behavior on main has changed.

To restate the case: a detector with thinks_internally=True that never sees an opening <think> still routes the entire chain-of-thought plus the dangling </think> into content, leaving reasoning_content empty. Consumers of content then render the model's internal monologue verbatim.

Worth noting this is not something a caller can reliably paper over after the fact — stripping a bare closer from an assembled response does nothing for streaming consumers, which see the raw reasoning as it arrives, token by token. The streaming path in this PR is what actually covers that case.

No tests have run on this yet: every failing check is the gate reporting the missing run-ci label. Could a maintainer add run-ci (or trigger /tag-and-rerun-ci)? Happy to rebase onto current main first if that helps, and to address any review feedback. Thanks!

@SilverNarcissus

SilverNarcissus commented Sep 18, 2026 •

Copy link
Copy Markdown

I encounter the same problem. We found the "<//think>" leaked into chat and "<//DSML>" leaked into tool call, which leads to the failure of agent use.

@fungaren

Copy link
Copy Markdown

We met the same problem as well, please take a look @JustinTong0323 @hnyls2002

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants