Skip to content

fix: inherit tool_call_parser and reasoning_parser in LoRA model registration - #7559

Merged
biswapanda merged 1 commit into
ai-dynamo:mainfrom
AmeenP:fix/lora-tool-call-parser
Mar 23, 2026
Merged

biswapanda merged 1 commit into
ai-dynamo:mainfrom
AmeenP:fix/lora-tool-call-parser

Conversation

@AmeenP

@AmeenP AmeenP commented Mar 21, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #7558

Summary

  • LoRA adapter MDCs now include tool_call_parser and reasoning_parser from the base model config
  • Fixes tool call parsing not working for LoRA model requests

Problem

When a LoRA adapter is registered via register_model() in handlers.py:load_lora, no runtime_config is passed. The published MDC has tool_call_parser: None, causing the frontend to skip tool parsing for LoRA requests — even though the worker is configured with --dyn-tool-call-parser hermes.

Fix

Build a ModelRuntimeConfig inheriting parsers from the base model config before calling register_model() for LoRA adapters:

runtime_config = ModelRuntimeConfig()
runtime_config.tool_call_parser = self.config.dyn_tool_call_parser
runtime_config.reasoning_parser = self.config.dyn_reasoning_parser

await register_model(
    ...
    runtime_config=runtime_config,
    ...
)

Files Changed

  • components/src/dynamo/vllm/handlers.py — add ModelRuntimeConfig import; construct and pass runtime_config in load_lora method

Test Plan

  • Built patched image from latest main (1.0)
  • Deployed to production cluster with Qwen3-30B-A3B MoE + LoRA adapters
  • Verified tool call request to LoRA model returns structured tool_calls with finish_reason: "tool_calls"
  • Verified base model tool calling still works

Summary by CodeRabbit

  • Bug Fixes
    • Improved LoRA model loading to ensure proper application of runtime configuration for tool and reasoning parsers, enhancing consistency and reliability across model operations.

@AmeenP
AmeenP requested a review from a team as a code owner March 21, 2026 05:36
@AmeenP
AmeenP requested a review from a team March 21, 2026 05:36
@copy-pr-bot

copy-pr-bot Bot commented Mar 21, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi AmeenP! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added external-contribution Pull request is from an external contributor fix backend::vllm Relates to the vllm backend labels Mar 21, 2026
@AmeenP
AmeenP force-pushed the fix/lora-tool-call-parser branch from af7b0ca to d4c38cf Compare March 21, 2026 05:37
@coderabbitai

coderabbitai Bot commented Mar 21, 2026 •

Copy link
Copy Markdown
Contributor

Walkthrough

Extended LoRA model registration in the vLLM handler to construct and pass a ModelRuntimeConfig containing tool_call_parser and reasoning_parser, previously omitted during LoRA registration.

Changes

Cohort / File(s) Summary
LoRA Runtime Configuration
components/src/dynamo/vllm/handlers.py
Added ModelRuntimeConfig import from dynamo.llm. Updated BaseWorkerHandler.load_lora() to construct a ModelRuntimeConfig instance with tool_call_parser and reasoning_parser from worker config, then pass it to register_model() during LoRA registration.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title directly summarizes the main change: inheriting tool_call_parser and reasoning_parser in LoRA model registration, matching the code changes.
Description check ✅ Passed The description includes all required sections (Overview via Summary, Details, Files Changed) and clearly explains the problem, fix, and test plan with specific context.
Linked Issues check ✅ Passed The PR fully addresses issue #7558 by constructing ModelRuntimeConfig with inherited parsers and passing it to register_model(), resolving the root cause of tool_call_parser being None in LoRA MDCs.
Out of Scope Changes check ✅ Passed All changes are scoped to the identified issue: adding ModelRuntimeConfig import and modifying load_lora() to pass runtime_config; no unrelated changes present.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Tip

CodeRabbit can scan for known vulnerabilities in your dependencies using OSV Scanner.

OSV Scanner will automatically detect and report security vulnerabilities in your project's dependencies. No additional configuration is required.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@components/src/dynamo/vllm/handlers.py`:
- Around line 780-792: Run the code formatter (black) on the modified file to
satisfy CI; specifically reformat the block where ModelRuntimeConfig is
instantiated and configured (runtime_config.tool_call_parser,
runtime_config.reasoning_parser) and the subsequent register_model call that
references ModelInput.Tokens, ModelType.Chat | ModelType.Completions,
self.generate_endpoint, self.config.model, and user_data so indentation/line
breaks match project style and black's expectations, then commit the resulting
changes.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d6415550-2439-480c-bfce-993e15d53779

📥 Commits

Reviewing files that changed from the base of the PR and between 28cfdcc and d4c38cf.

📒 Files selected for processing (1)
  • components/src/dynamo/vllm/handlers.py

Comment thread components/src/dynamo/vllm/handlers.py Resolved
…stration

LoRA adapter MDCs were published to etcd without runtime_config,
causing tool_call_parser and reasoning_parser to be None. The frontend
reads these from the MDC to decide whether to apply tool parsing, so
LoRA requests returned raw <tool_call> XML in content instead of
structured tool_calls.

Build a ModelRuntimeConfig inheriting parsers from the base model
config before calling register_model() for LoRA adapters, matching
the pattern already used for base model registration in main.py.

Signed-off-by: AmeenP <ameenp360@gmail.com>
@AmeenP
AmeenP force-pushed the fix/lora-tool-call-parser branch from d4c38cf to 0aa8109 Compare March 21, 2026 05:47
@biswapanda

Copy link
Copy Markdown
Contributor

/ok to test 0aa8109

@biswapanda
biswapanda merged commit d2d97f1 into ai-dynamo:main Mar 23, 2026
73 checks passed
WenjiaoYue pushed a commit to WenjiaoYue/dynamo that referenced this pull request Mar 25, 2026
…stration (ai-dynamo#7559)

Signed-off-by: AmeenP <ameenp360@gmail.com>
Signed-off-by: WenjiaoYue <wenjiao.yue@intel.com>
yao531441 pushed a commit to yao531441/dynamo that referenced this pull request May 13, 2026
…stration (ai-dynamo#7559)

Signed-off-by: AmeenP <ameenp360@gmail.com>

This branch was previously deployed

1 inactive deployment
GITLAB — 0aa81096 Deployed Mar 21, 2026 by copy-pr-bot[bot] via Trigger CI Pipeline #21416
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::vllm Relates to the vllm backend external-contribution Pull request is from an external contributor fix size/S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LoRA model registrations missing tool_call_parser in runtime_config — tool calls not parsed

2 participants