Skip to content

fix(frontend): standardize resolving thinking controls - #11739

Merged
KrishnanPrash merged 5 commits into
mainfrom
kprashanth/honor-reasoning-effort-render-paths
Aug 20, 2026
Merged

KrishnanPrash merged 5 commits into
mainfrom
kprashanth/honor-reasoning-effort-render-paths

Conversation

@KrishnanPrash

@KrishnanPrash KrishnanPrash commented Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A client can ask a model to think, or not to think, in four different ways. Model families then read four different chat-template keys to find that instruction. Dynamo resolved those controls in more than one place, so the answer a model received depended on which render path served the request.

This PR gives the decision one owner. normalize_reasoning_template_args runs once at the HTTP boundary, for every chat request, before any dispatch. It now writes the decision in all three template dialects, so a family that reads thinking gets the same answer as one that reads enable_thinking. Both custom chat processors forward that result instead of deriving their own.

Why three keys

No single key works for every model. Qwen and GLM read enable_thinking. Kimi and DeepSeek-V4 read thinking. MiniMax-M3 reads thinking_mode. Before this PR, reasoning_effort wrote only enable_thinking, so it reached Qwen and GLM and said nothing to the other three. The SGLang chat processor patched that gap on its own path. The vLLM chat processor did not. That is how the two paths drifted apart.

Precedence

Highest first. A rung only applies when no rung above it decided.

  1. The root thinking param
  2. thinking or enable_thinking in chat_template_args
  3. thinking_mode in chat_template_args
  4. reasoning_effort, top-level first, then the nested copy. A nested value must be a string; null counts as absent
  5. --dyn-default-thinking-mode
  6. The model or chat template's own default

adaptive is the one value that decides not to decide, and where it came from matters. As the root thinking param it is the highest control, so it clears any toggle the client sent. Inside chat_template_args it sits below the bools, so an explicit thinking or enable_thinking beside it wins and replaces it.

Request in, template args out

Every row below is pinned by a unit test.

Request chat_template_args after normalization
nothing unchanged
reasoning_effort: high thinking: true, enable_thinking: true, thinking_mode: enabled, reasoning_effort: high
reasoning_effort: none thinking: false, enable_thinking: false, thinking_mode: disabled, reasoning_effort: none
thinking: {"type": "enabled"} thinking: true, enable_thinking: true, thinking_mode: enabled
thinking: {"type": "disabled"} thinking: false, enable_thinking: false, thinking_mode: disabled
thinking: {"type": "adaptive"} thinking_mode: adaptive only, any client toggle removed
thinking: {"type": "adaptive"} + reasoning_effort: high thinking_mode: adaptive, reasoning_effort: high, no toggle
chat_template_args: {enable_thinking: false} + reasoning_effort: high thinking: false, enable_thinking: false, thinking_mode: disabled, reasoning_effort: high
chat_template_args: {thinking_mode: disabled} + reasoning_effort: high thinking: false, enable_thinking: false, thinking_mode: disabled, reasoning_effort: high
chat_template_args: {reasoning_effort: none} thinking: false, enable_thinking: false, thinking_mode: disabled, reasoning_effort: none
chat_template_args: {enable_thinking: false} thinking: false, enable_thinking: false, thinking_mode: disabled
chat_template_args: {thinking: false, thinking_mode: adaptive} thinking: false, enable_thinking: false, thinking_mode: disabled
chat_template_args: {reasoning_effort: null} unchanged

A grade other than none means think. That includes minimal, which asks for less reasoning, not for none.

Parity with vLLM and SGLang

Behavior This PR Dynamo before vLLM SGLang
reasoning_effort sets a thinking toggle Yes, all three keys Only enable_thinking No, passed to the template untouched No, passed to the template untouched
Accepts reasoning_effort: none Yes Yes No, low/medium/high only No, low/medium/high only
Top-level vs nested reasoning_effort Top-level wins Nested ignored Top-level wins Nested wins, it is popped and promoted
Root thinking param Yes Yes Not supported Not supported
Where the decision is made Frontend only Frontend and the SGLang processor Engine Engine

Neither engine turns reasoning_effort into a thinking decision, and neither accepts none. Dynamo does both, so one request field behaves the same on every backend. That is a deliberate deviation, and it is the reason the SGLang processor grew its own none handling in the first place. With the frontend now writing every dialect, that handling is no longer needed and has been removed.

The one place Dynamo does not match either engine is nested reasoning_effort: vLLM lets the top-level field win, SGLang lets the nested one win. This PR follows vLLM.

What changed

  • chat_completions.rs: one precedence chain replaces the previous partial derivation. It lowercases thinking_mode to match the case-insensitive readers downstream, it no longer returns early when only chat_template_args carries a control, and it derives a toggle from a grade only when that grade is a string.
  • sglang_prepost.py: the processor's own derivation is deleted. It forwards what the frontend resolved, which is what the vLLM processor already did.
  • prepost.py: stops injecting reasoning_effort: None when the request has no grade. The guard above it already protected the nested value, so this line only added a null.
  • _thinking_parity.py: a shared case table, asserted by both backend suites, so the two paths cannot drift again without a test failing.
  • FRONTEND_CASES.md: adds FRONTEND.10 for thinking-control forwarding.

Validation

  • cargo test -p dynamo-llm --lib protocols::openai::chat_completions: 84 passed, 0 failed.
  • cargo fmt --all and cargo clippy -p dynamo-llm --lib clean.
  • vLLM 0.27.1, real venv: test_vllm_processor_unit.py 119 passed.
  • SGLang 0.5.17, real venv: test_sglang_processor_unit.py 227 passed.
  • Both suites run the shared parity table, so every case above is checked on both backends.

@KrishnanPrash
KrishnanPrash requested review from a team as code owners July 15, 2026 23:21

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

@coderabbitai

coderabbitai Bot commented Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The change standardizes reasoning-toggle precedence, forwards selected chat-template controls through the frontend, adds regression coverage, and documents adaptive and graded reasoning behavior.

Changes

Reasoning control propagation

Layer / File(s) Summary
Normalize reasoning template arguments
lib/llm/src/protocols/openai/chat_completions.rs
thinking modes now set or clear enable_thinking consistently; adaptive mode preserves its mode without deriving an on/off toggle. Unit tests cover these combinations.
Forward reasoning controls to chat templates
components/src/dynamo/frontend/prepost.py, components/src/dynamo/frontend/tests/test_vllm_processor_unit.py
_prepare_request forwards selected chat_template_args, propagates reasoning_effort, applies the default enablement, and tests explicit overrides and absent effort values.
Document control precedence
docs/reasoning/README.md
Documents precedence among thinking, chat_template_args, reasoning_effort, and model or parser defaults, including adaptive behavior.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description is detailed and on-topic, but it omits the required Related Issues section and the Where should the reviewer start section. Add the required Related Issues declaration and Where should the reviewer start section, and align the description with the repository template headings.
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main change: standardizing the resolution of frontend thinking controls.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
components/src/dynamo/frontend/tests/test_vllm_processor_unit.py (1)

757-793: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add a regression test for adaptive mode + reasoning_effort.

This suite covers reasoning_effort deriving enable_thinking and an explicit chat_template_args.enable_thinking overriding it, but none of the cases combine thinking_mode: "adaptive" (via recovered chat_template_args) with reasoning_effort present. That combination currently produces a wrong enable_thinking toggle in _prepare_request (see the comment on components/src/dynamo/frontend/prepost.py lines 140-152) and would have been caught by a test like:

def test_adaptive_mode_with_reasoning_effort_no_toggle(self, tokenizer):
    """thinking_mode=adaptive must not derive enable_thinking from reasoning_effort."""
    kwargs = self._kwargs(
        {
            "model": MODEL,
            "messages": self._messages(),
            "chat_template_args": {"thinking_mode": "adaptive"},
            "reasoning_effort": "high",
        },
        tokenizer,
    )
    assert "enable_thinking" not in kwargs
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/frontend/tests/test_vllm_processor_unit.py` around
lines 757 - 793, Add a regression test alongside the existing reasoning-effort
tests in the relevant test class, using _kwargs with chat_template_args
containing thinking_mode="adaptive" and reasoning_effort set to "high". Assert
that the resulting kwargs omit enable_thinking, covering the recovered adaptive
mode without changing the existing tests.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/src/dynamo/frontend/prepost.py`:
- Around line 140-152: Update the reasoning_effort handling near the recovered
thinking keys so it always forwards reasoning_effort, but only synthesizes
enable_thinking when thinking_mode is not "adaptive". Preserve any explicitly
recovered thinking values and ensure adaptive requests do not receive a derived
enable_thinking toggle.

---

Nitpick comments:
In `@components/src/dynamo/frontend/tests/test_vllm_processor_unit.py`:
- Around line 757-793: Add a regression test alongside the existing
reasoning-effort tests in the relevant test class, using _kwargs with
chat_template_args containing thinking_mode="adaptive" and reasoning_effort set
to "high". Assert that the resulting kwargs omit enable_thinking, covering the
recovered adaptive mode without changing the existing tests.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 83284d5d-e8b1-4c77-87c6-7b0ed777bf6b

📥 Commits

Reviewing files that changed from the base of the PR and between 94494d3 and 9879f3b.

📒 Files selected for processing (4)
  • components/src/dynamo/frontend/prepost.py
  • components/src/dynamo/frontend/tests/test_vllm_processor_unit.py
  • docs/reasoning/README.md
  • lib/llm/src/protocols/openai/chat_completions.rs

@github-actions github-actions Bot added fix documentation Improvements or additions to documentation frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Jul 15, 2026
@github-actions

github-actions Bot commented Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

@datadog-official

datadog-official Bot commented Jul 15, 2026 •

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

🚦 3 Pipeline jobs failed

PR | backend-status-check   View in Datadog   GitHub Actions

PR | trtllm-dev / Build multi-arch cuda13.1   View in Datadog   GitHub Actions

PR | vllm-dev / Build multi-arch cuda13.0   View in Datadog   GitHub Actions

ℹ️ Info

🎯 Code Coverage (details)
• Patch Coverage: 87.50%
• Overall Coverage: 43.24% (-4.57%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 4bb8d42 | Docs | Give us feedback!

@github-actions github-actions Bot added the Stale label Aug 15, 2026
@KrishnanPrash
KrishnanPrash force-pushed the kprashanth/honor-reasoning-effort-render-paths branch from 4bb8d42 to b2fe546 Compare August 18, 2026 22:40
@KrishnanPrash KrishnanPrash changed the title fix(frontend): honor reasoning_effort and thinking controls on all chat render paths fix(frontend): let adaptive thinking clear stale reasoning toggles Aug 18, 2026
@ai-dynamo ai-dynamo deleted a comment from github-actions Bot Aug 18, 2026
@KrishnanPrash
KrishnanPrash force-pushed the kprashanth/honor-reasoning-effort-render-paths branch from b2fe546 to 7d58986 Compare August 19, 2026 01:38
@KrishnanPrash KrishnanPrash changed the title fix(frontend): let adaptive thinking clear stale reasoning toggles fix(frontend): resolve thinking controls once, in the frontend Aug 19, 2026
@KrishnanPrash KrishnanPrash changed the title fix(frontend): resolve thinking controls once, in the frontend fix(frontend): resolve thinking controls once Aug 19, 2026
@KrishnanPrash KrishnanPrash changed the title fix(frontend): resolve thinking controls once fix(frontend): standardize resolving thinking controls Aug 19, 2026
@KrishnanPrash
KrishnanPrash force-pushed the kprashanth/honor-reasoning-effort-render-paths branch from 7d58986 to e730cca Compare August 19, 2026 16:58

@indrajit96 indrajit96 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
With a couple of corner cases found please fix them, conditianlly approving!
Thanks a lot for adding this @KrishnanPrash

Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
@KrishnanPrash
KrishnanPrash force-pushed the kprashanth/honor-reasoning-effort-render-paths branch from e730cca to 9911e48 Compare August 19, 2026 20:39
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
@KrishnanPrash
KrishnanPrash merged commit 86af092 into main Aug 20, 2026
119 checks passed
@KrishnanPrash
KrishnanPrash deleted the kprashanth/honor-reasoning-effort-render-paths branch August 20, 2026 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation fix frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants