Repository navigation
fix: inherit tool_call_parser and reasoning_parser in LoRA model registration - #7559
Conversation
|
👋 Hi AmeenP! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
af7b0ca to
d4c38cf
Compare
WalkthroughExtended LoRA model registration in the vLLM handler to construct and pass a Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment Tip CodeRabbit can scan for known vulnerabilities in your dependencies using OSV Scanner.OSV Scanner will automatically detect and report security vulnerabilities in your project's dependencies. No additional configuration is required. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@components/src/dynamo/vllm/handlers.py`:
- Around line 780-792: Run the code formatter (black) on the modified file to
satisfy CI; specifically reformat the block where ModelRuntimeConfig is
instantiated and configured (runtime_config.tool_call_parser,
runtime_config.reasoning_parser) and the subsequent register_model call that
references ModelInput.Tokens, ModelType.Chat | ModelType.Completions,
self.generate_endpoint, self.config.model, and user_data so indentation/line
breaks match project style and black's expectations, then commit the resulting
changes.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: d6415550-2439-480c-bfce-993e15d53779
📒 Files selected for processing (1)
components/src/dynamo/vllm/handlers.py
…stration LoRA adapter MDCs were published to etcd without runtime_config, causing tool_call_parser and reasoning_parser to be None. The frontend reads these from the MDC to decide whether to apply tool parsing, so LoRA requests returned raw <tool_call> XML in content instead of structured tool_calls. Build a ModelRuntimeConfig inheriting parsers from the base model config before calling register_model() for LoRA adapters, matching the pattern already used for base model registration in main.py. Signed-off-by: AmeenP <ameenp360@gmail.com>
d4c38cf to
0aa8109
Compare
|
/ok to test 0aa8109 |
…stration (ai-dynamo#7559) Signed-off-by: AmeenP <ameenp360@gmail.com> Signed-off-by: WenjiaoYue <wenjiao.yue@intel.com>
…stration (ai-dynamo#7559) Signed-off-by: AmeenP <ameenp360@gmail.com>
Fixes #7558
Summary
tool_call_parserandreasoning_parserfrom the base model configProblem
When a LoRA adapter is registered via
register_model()inhandlers.py:load_lora, noruntime_configis passed. The published MDC hastool_call_parser: None, causing the frontend to skip tool parsing for LoRA requests — even though the worker is configured with--dyn-tool-call-parser hermes.Fix
Build a
ModelRuntimeConfiginheriting parsers from the base model config before callingregister_model()for LoRA adapters:Files Changed
components/src/dynamo/vllm/handlers.py— addModelRuntimeConfigimport; construct and passruntime_configinload_loramethodTest Plan
tool_callswithfinish_reason: "tool_calls"Summary by CodeRabbit