What's the plan to support for other VLMs beyond Moondream?

[prothnh]

Can I self host Magnitude and get it configured to work with a local instance of Phi-3.5 Vision?

[anerli]

Hi - phi-3.5 vision from what I understand is not actually a grounded model, in other words, it doesn't know where things are on a screenshot are exactly.
However there are plans (very soon) to support other VLMs that are grounded - for example Claude 4/3.7, UI-TARS, Molmo, and others

[prothnh]

That sounds wonderful.

…

________________________________ From: Anders ***@***.***> Sent: Wednesday, June 4, 2025 6:30 PM To: magnitudedev/magnitude ***@***.***> Cc: Paul Roth ***@***.***>; Author ***@***.***> Subject: Re: [magnitudedev/magnitude] What's the plan to support for other VLMs beyond Moondream? (Discussion #44) [External Sender: Use CAUTION when opening links/attachments] Hi - phi-3.5 vision from what I understand is not actually a grounded model, in other words, it doesn't know where things are on a screenshot are exactly. However there are plans (very soon) to support other VLMs that are grounded - for example Claude 4/3.7, UI-TARS, Molmo, and others — Reply to this email directly, view it on GitHub<#44 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BN2PZRVTOC3X2ML45P7O2BD3B5XO5AVCNFSM6AAAAAB6TZWQHOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMZXGI4DIMY>. You are receiving this because you authored the thread.Message ID: ***@***.***>

________________________________ This message may contain confidential information, legally privileged information or other information subject to legal restrictions. If you are not a designated recipient of this message, or an agent responsible for delivering it to a designated recipient, please do not read, copy, use or disclose this message or its attachments, and notify the sender by replying to this message and delete or destroy all copies of this message and attachments.

[anerli]

Btw - this is now a thing on v0.2.0+, more info in docs: https://docs.magnitude.run/customizing/llm-configuration