Repository navigation
fix(frontend): standardize resolving thinking controls - #11739
Conversation
WalkthroughThe change standardizes reasoning-toggle precedence, forwards selected chat-template controls through the frontend, adds regression coverage, and documents adaptive and graded reasoning behavior. ChangesReasoning control propagation
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
components/src/dynamo/frontend/tests/test_vllm_processor_unit.py (1)
757-793: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd a regression test for
adaptivemode +reasoning_effort.This suite covers
reasoning_effortderivingenable_thinkingand an explicitchat_template_args.enable_thinkingoverriding it, but none of the cases combinethinking_mode: "adaptive"(via recoveredchat_template_args) withreasoning_effortpresent. That combination currently produces a wrongenable_thinkingtoggle in_prepare_request(see the comment oncomponents/src/dynamo/frontend/prepost.pylines 140-152) and would have been caught by a test like:def test_adaptive_mode_with_reasoning_effort_no_toggle(self, tokenizer): """thinking_mode=adaptive must not derive enable_thinking from reasoning_effort.""" kwargs = self._kwargs( { "model": MODEL, "messages": self._messages(), "chat_template_args": {"thinking_mode": "adaptive"}, "reasoning_effort": "high", }, tokenizer, ) assert "enable_thinking" not in kwargs🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/src/dynamo/frontend/tests/test_vllm_processor_unit.py` around lines 757 - 793, Add a regression test alongside the existing reasoning-effort tests in the relevant test class, using _kwargs with chat_template_args containing thinking_mode="adaptive" and reasoning_effort set to "high". Assert that the resulting kwargs omit enable_thinking, covering the recovered adaptive mode without changing the existing tests.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@components/src/dynamo/frontend/prepost.py`:
- Around line 140-152: Update the reasoning_effort handling near the recovered
thinking keys so it always forwards reasoning_effort, but only synthesizes
enable_thinking when thinking_mode is not "adaptive". Preserve any explicitly
recovered thinking values and ensure adaptive requests do not receive a derived
enable_thinking toggle.
---
Nitpick comments:
In `@components/src/dynamo/frontend/tests/test_vllm_processor_unit.py`:
- Around line 757-793: Add a regression test alongside the existing
reasoning-effort tests in the relevant test class, using _kwargs with
chat_template_args containing thinking_mode="adaptive" and reasoning_effort set
to "high". Assert that the resulting kwargs omit enable_thinking, covering the
recovered adaptive mode without changing the existing tests.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 83284d5d-e8b1-4c77-87c6-7b0ed777bf6b
📒 Files selected for processing (4)
components/src/dynamo/frontend/prepost.pycomponents/src/dynamo/frontend/tests/test_vllm_processor_unit.pydocs/reasoning/README.mdlib/llm/src/protocols/openai/chat_completions.rs
4bb8d42 to
b2fe546
Compare
b2fe546 to
7d58986
Compare
7d58986 to
e730cca
Compare
indrajit96
left a comment
There was a problem hiding this comment.
LGTM!
With a couple of corner cases found please fix them, conditianlly approving!
Thanks a lot for adding this @KrishnanPrash
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
e730cca to
9911e48
Compare
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
Summary
A client can ask a model to think, or not to think, in four different ways. Model families then read four different chat-template keys to find that instruction. Dynamo resolved those controls in more than one place, so the answer a model received depended on which render path served the request.
This PR gives the decision one owner.
normalize_reasoning_template_argsruns once at the HTTP boundary, for every chat request, before any dispatch. It now writes the decision in all three template dialects, so a family that readsthinkinggets the same answer as one that readsenable_thinking. Both custom chat processors forward that result instead of deriving their own.Why three keys
No single key works for every model.
QwenandGLMreadenable_thinking.KimiandDeepSeek-V4readthinking.MiniMax-M3readsthinking_mode. Before this PR,reasoning_effortwrote onlyenable_thinking, so it reached Qwen and GLM and said nothing to the other three. The SGLang chat processor patched that gap on its own path. The vLLM chat processor did not. That is how the two paths drifted apart.Precedence
Highest first. A rung only applies when no rung above it decided.
thinkingparamthinkingorenable_thinkinginchat_template_argsthinking_modeinchat_template_argsreasoning_effort, top-level first, then the nested copy. A nested value must be a string;nullcounts as absent--dyn-default-thinking-modeadaptiveis the one value that decides not to decide, and where it came from matters. As the rootthinkingparam it is the highest control, so it clears any toggle the client sent. Insidechat_template_argsit sits below the bools, so an explicitthinkingorenable_thinkingbeside it wins and replaces it.Request in, template args out
Every row below is pinned by a unit test.
chat_template_argsafter normalizationreasoning_effort: highthinking: true,enable_thinking: true,thinking_mode: enabled,reasoning_effort: highreasoning_effort: nonethinking: false,enable_thinking: false,thinking_mode: disabled,reasoning_effort: nonethinking: {"type": "enabled"}thinking: true,enable_thinking: true,thinking_mode: enabledthinking: {"type": "disabled"}thinking: false,enable_thinking: false,thinking_mode: disabledthinking: {"type": "adaptive"}thinking_mode: adaptiveonly, any client toggle removedthinking: {"type": "adaptive"}+reasoning_effort: highthinking_mode: adaptive,reasoning_effort: high, no togglechat_template_args: {enable_thinking: false}+reasoning_effort: highthinking: false,enable_thinking: false,thinking_mode: disabled,reasoning_effort: highchat_template_args: {thinking_mode: disabled}+reasoning_effort: highthinking: false,enable_thinking: false,thinking_mode: disabled,reasoning_effort: highchat_template_args: {reasoning_effort: none}thinking: false,enable_thinking: false,thinking_mode: disabled,reasoning_effort: nonechat_template_args: {enable_thinking: false}thinking: false,enable_thinking: false,thinking_mode: disabledchat_template_args: {thinking: false, thinking_mode: adaptive}thinking: false,enable_thinking: false,thinking_mode: disabledchat_template_args: {reasoning_effort: null}A grade other than
nonemeans think. That includesminimal, which asks for less reasoning, not for none.Parity with vLLM and SGLang
reasoning_effortsets a thinking toggleenable_thinkingreasoning_effort: nonelow/medium/highonlylow/medium/highonlyreasoning_effortthinkingparamNeither engine turns
reasoning_effortinto a thinking decision, and neither acceptsnone. Dynamo does both, so one request field behaves the same on every backend. That is a deliberate deviation, and it is the reason the SGLang processor grew its ownnonehandling in the first place. With the frontend now writing every dialect, that handling is no longer needed and has been removed.The one place Dynamo does not match either engine is nested
reasoning_effort: vLLM lets the top-level field win, SGLang lets the nested one win. This PR follows vLLM.What changed
chat_completions.rs: one precedence chain replaces the previous partial derivation. It lowercasesthinking_modeto match the case-insensitive readers downstream, it no longer returns early when onlychat_template_argscarries a control, and it derives a toggle from a grade only when that grade is a string.sglang_prepost.py: the processor's own derivation is deleted. It forwards what the frontend resolved, which is what the vLLM processor already did.prepost.py: stops injectingreasoning_effort: Nonewhen the request has no grade. The guard above it already protected the nested value, so this line only added a null._thinking_parity.py: a shared case table, asserted by both backend suites, so the two paths cannot drift again without a test failing.FRONTEND_CASES.md: addsFRONTEND.10for thinking-control forwarding.Validation
cargo test -p dynamo-llm --lib protocols::openai::chat_completions: 84 passed, 0 failed.cargo fmt --allandcargo clippy -p dynamo-llm --libclean.test_vllm_processor_unit.py119 passed.test_sglang_processor_unit.py227 passed.