Improving / working around Moondream accuracy

[anerli]

Moondream 2B is susceptible to misclick targets with descriptions that are even somewhat complex. Bigger Moondream models are on the way but how should Magnitude address these low-accuracy situations?

[anerli]

Potential approaches:

Soln A: Bigger Moondream

Once bigger Moondream model is out, it should be a lot better at picking up these semi-trivial target descriptions

Soln B: Zoom In

Zoom in approach: Zoom into the area where the target lives and reprompt moondream for a more accurate click.

Challenges:

• How do you know what general area?
• How do you know if it missed and that the zoom in is necessary?

Find general area

Approaches:

• Have planner designate some area every time (e.g. with some high level spatial description, or by showing a subdivision of the screen before taking screenshot)
  or
• Center around initial click

When did it miss?

• Once Moondream supports exposing token probabilities, can potentially use token probability as an indicator of confidence, trigger zoom in on low logp
  or
• Just do every time (not ideal, 2x moondream calls for each click instead of 1x) - unless planner designates the area in same prompt and we just zoom in immediately
  or
• some kind of observer (i.e. planner checks if missed) - but cant know when observer needs to trigger, so then just 1 moondream call + 1 observer call is even more expensive