[WIP Integration] LocateAnything-3B (MoonViT-SO-400M + Eagle MLP + Qwen2.5-3B): spatial localization failure and token count mismatch in clip.cpp / mtmd

[chandra-ps612]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

I have a working in-progress integration of nvidia/LocateAnything-3B into the mtmd / clip.cpp framework. The model loads, warms up, and runs inference to completion — but spatial localization is completely broken: it always outputs <0><0><1000><1000> regardless of the prompt or image. There is also a token count discrepancy (252 produced vs 256 expected).

Architecture: MoonViT-SO-400M (vision encoder, head_dim=72, 2D RoPE, 27-layer ViT) + Eagle MLP projector (2-layer MLP with intermediate LayerNorm) + Qwen2.5-3B Instruct (LLM backbone), with a 2×2 spatial patch merge between encoder and projector.

Environment

Item	Value
llama.cpp commit	e4406fed5 (build 8540)
GPU	NVIDIA GeForce GTX 1050 Ti (4030 MiB, compute 6.1)
OS / Arch	Linux x86_64, GNU 11.4.0
LLM weights	Q4_K_M, 1.96 GiB
mmproj weights	BF16 GGUF

Reproduction

./build/bin/llama-mtmd-cli \
  -m  ./LocateAnything-3B/LocateAnything-3B-LLM-Q4_K_M.gguf \
  --mmproj ./LocateAnything-3B/mmproj-LocateAnything-3B-BF16.gguf \
  --image ./Sample_dataset_312.jpg \
  -p "Locate all the instances that matches the following description: person." \
  --temp 0.1 -n 512 -c 2048 -ngl 10 --image-max-tokens 256

Actual output:
person<0><0><1000><1000>

Expected output:
person<120><45><480><790> (tight box around person)

Loaded hparams from log

projector: moonvit
image_size: 896
patch_size: 14
n_merge: 2
n_embd: 1152
n_head: 16
n_layer: 27
image_max_pixels: 200704
warmup image: 448 × 448

Bug 1 — Token count: 252 produced, 256 expected

Expected: (448/14)² / (2×2) = 1024 / 4 = 256 visual tokens
Actual log: decoding image batch 1/1, n_tokens_batch = 252

Suspected cause: letterboxing rounding the image to a non-multiple of patch_size × merge_k = 28 (e.g. 444 instead of 448), producing ~250 tokens instead of 256.

Bug 2 — Spatial output collapsed to full image

The <0><0><1000><1000> output is the classic symptom of lost spatial encoding. Three ranked suspects:

1. pos_row / pos_col not filled before graph execution — tensors are declared as ggml_set_input and named "pos_row" / "pos_col", but I am not certain clip.cpp is writing the actual row/column I32 indices into them before ggml_backend_graph_compute. If zero-initialized, every patch looks like position (0,0).

2. build_patch_merge_permute() permutation order not matching LocateAnything-3B's training (PyTorch pixel-unshuffle ordering).

3. v.position_embd.weight (shape [hidden_size, 4096]) always sliced from offset 0 regardless of actual image resolution.

Bug 3 — head_dim=72 flash attention assert (worked around)

MoonViT: hidden_size=1152, n_heads=16 → head_dim=72.
ggml_flash_attn_ext asserts head_dim ∈ {64,80,96,112,128,256} and crashes.
Worked around with manual scaled dot-product attention in moonvit_encoder_layer(). Warmup passes cleanly. Flagging in case there is a preferred framework-level way to handle non-standard head_dim.

Questions for maintainers

1. pos_row / pos_col fill path — where in clip.cpp is the correct place to write I32 row/column indices into named input tensors before compute? Is there a pattern in qwen2vl.cpp or pixtral.cpp to follow?

2. clip_n_output_tokens — should this be set to the post-merge count (256) or pre-merge (1024)?

3. Preprocessing rounding — should image dimensions be constrained to a multiple of patch_size × merge_k = 28 to guarantee an integer token count after merging?

Implementation gists

• moonvit.cpp (full ggml graph): https://gist.github.com/chandra-ps612/70b698866c7ecb6e60aeecb21114a1be
• models.h (struct declaration): https://gist.github.com/chandra-ps612/fee5400a3f08d68f8a64561a8f8b88f0
• clip-impl.h (projector hooks): https://gist.github.com/chandra-ps612/fee5400a3f08d68f8a64561a8f8b88f0
• clip.cpp (dispatch changes): https://gist.github.com/chandra-ps612/1542d764772ad3517d1c71a0d6bcf906
• convert_moonvit_to_gguf.py: https://gist.github.com/chandra-ps612/convert_moonvit_to_gguf
• extract_and_convert_llm.py: https://gist.github.com/chandra-ps612/90a478a584bfeafa7594c268ceaef8da

Related: #23784 (plain feature request for LocateAnything)
HuggingFace model: https://huggingface.co/nvidia/LocateAnything-3B

Motivation

LocateAnything-3B is the strongest text and character grounding model I have tested at this scale — outperforming Qwen2.5-VL 7B and Gemma3 12B on tight bounding-box localization tasks in my personal benchmarks. At Q4_K_M quantization it fits entirely within 4 GB VRAM, making it usable on consumer hardware (GTX 1050 Ti class and above). No llama.cpp-compatible integration currently exists. A working mtmd integration would make this model accessible for local inference without requiring PyTorch or the full HuggingFace stack.

Possible Implementation

A full WIP implementation already exists across the gists linked in the Feature Description above. The core additions required are:

1. moonvit.cpp — ggml computation graph for MoonViT-SO-400M:

   • Patch embedding via ggml_conv_2d with [KW, KH, IC, OC] kernel layout
   • Learnable positional embedding sliced from v.position_embd.weight [hidden_size, 4096]
   • 27 encoder layers with LayerNorm → QKV projection → 2D RoPE → manual scaled dot-product attention (head_dim=72, bypassing ggml_flash_attn_ext) → FFN (GELU)
   • 2×2 patch merge via build_patch_merge_permute()
   • Eagle MLP projector: LayerNorm → Linear → GELU → Linear

2. models.h — add clip_graph_moonvit struct declaration alongside existing encoders

3. clip-impl.h — register PROJECTOR_TYPE_MOONVIT in the projector dispatch table

4. clip.cpp — handle pos_row / pos_col I32 fill for 2D RoPE (the part currently unclear and blocking correct spatial output)

5. convert_moonvit_to_gguf.py — Python conversion script to export MoonViT + Eagle MLP weights to BF16 GGUF

The remaining blockers are the three bugs described above, particularly the pos_row/pos_col fill path (Bug 2 Cause A) which is the most likely root cause of the spatial collapse. Guidance from maintainers familiar with qwen2vl.cpp or pixtral.cpp's 2D position ID fill path would unblock this immediately.

[richiejp]

EDIT: Sorry the below is Claude's raw slop. The verification code is at https://github.com/richiejp/moonvit-verify. Doing layer-by-layer verification usually fixes these kinds of issues, but I haven't fully checked it.

I did a layer-by-layer numeric verification of the MoonViT vision tower (PyTorch reference modeling_vit.py vs the ggml graph reconstructed from your gists), and it pins down why the boxes collapse to <0><0><1000><1000>. The spatial collapse is entirely in the vision encoder, and it's three independent bugs — not the pos_row/pos_col fill (that part is actually correct).

Method

Two cross-checks, both on a 448×448 image (32×32 patches → 256 tokens), CPU/F32 for determinism:

1. Reference-exact (weights only): take the real MoonVitPretrainedModel, inject each suspect op's ggml behaviour one at a time, and diff every block output. No GGUF/converter involved, so the weights are identical and any divergence is purely algorithmic.
2. Real compiled binary: reconstruct the integration from the gists onto f8d4abae8, add a tap:* dump after ggml_backend_sched_graph_compute, feed the identical normalized pixels via clip_encode_float_image, and diff the real ggml taps against the reference.

The final merged vision tokens that go to the LLM end up at cosine ≈ 0.11 vs the reference — i.e. essentially decorrelated, which is exactly why grounding degenerates to the whole image.

The three bugs

1. Patch-embed conv output layout (moonvit.cpp)

ggml_conv_2d returns ne = [OW, OH, OC, N] (grid_w, grid_h, hidden), not [OC, OW, OH, N]. The "F2" fix flattens it directly to [hidden, n_patches], which interleaves channels with spatial positions. The conv tap alone is already at cosine 0.44. Adding the missing permute fixes it exactly:

// ggml_conv_2d output is [grid_w, grid_h, hidden, 1]
patches = ggml_cont(ctx0, ggml_permute(ctx0, patches, 1, 2, 0, 3)); // -> [hidden, grid_w, grid_h, 1]
patches = ggml_reshape_2d(ctx0, patches, hidden_size, n_patches);    // patch idx = gh*grid_w + gw

→ conv tap cosine 0.44 → 1.00000 (max-abs 0.0).

2. 2D RoPE uses the wrong primitive (build_rope_2d) — the main grounding bug

build_rope_2d(pos_row, pos_col, …, false) is sectioned: dims 0–35 are rotated by the row position, dims 36–71 by the column position. MoonViT's apply_rope/Rope2DPosEmb is interleaved: adjacent pairs alternate axis across all 72 dims (even pairs → column freq, odd pairs → row freq). The frequency sets are identical but the dim→axis mapping is completely different, so build_rope_2d structurally cannot reproduce it. Injecting the sectioned rope into the reference (everything else exact) gives:

tap	cosine vs reference
00_patch+pos	1.0000
L00	0.994
final_ln / merge	0.152

A drop-in fix that reproduces the reference to cosine 1.0000 is to apply the rotation with host-precomputed cos/sin (= real/imag of the reference freqs_cis) over adjacent pairs, instead of build_rope_2d. Note pos_row/pos_col are filled correctly — that hypothesis from the issue is a red herring.

3. Positional embedding is flat-sliced instead of interpolated (moonvit.cpp)

The reference Learnable2DInterpPosEmb bicubic-interpolates the learnable 64×64×dim grid to the actual patch grid. The graph instead takes the first n_patches rows of the [hidden, 4096] table. Injecting the flat-slice (everything else exact):

tap	cosine vs reference
00_patch+pos	0.606
final_ln / merge	0.371

(For grids >4096 patches the current code also silently zeroes the pos-emb via the else branch.) Fix: interpolate the 64×64 grid to (grid_h, grid_w) before adding.

Putting it together

config (vs reference)	merge cosine
flat-slice pos-emb only	0.371
sectioned rope only	0.152
all three (= current C++)	0.114
all fixes applied	1.0000 (max-abs ~1e-4 at every layer)

And the real compiled binary, once bug #1 is fixed, matches the injected-emulation at every layer (merge cosine 0.1128 vs 0.114), confirming the emulation faithfully models the ggml graph and that bugs #2/#3 are present in the real binary exactly as diagnosed.

Separately

• 252 vs 256 tokens is a preprocessing mismatch, independent of the math above: the reference processor caps at 4096 patches then pads each side up to a multiple of patch_size * merge = 28. llama.cpp's resize/letterbox needs to match that to land on the right grid.
• Minor: moonvit.cpp uses LayerNorm eps = 1e-6, but the reference nn.LayerNorm default is 1e-5.

Happy to share the verification harness (flake + scripts: reference run, ggml-tap dump, comparison) if useful.

[yuuko-eth]

@richiejp @chandra-ps612 guys! I've tried to make one based on your findings! 👐

The finished port is at the fork here @ mtmd-grounders, and I've exported GGUFs here. The path implemented is the normal autoregressive one, no PBD yet as that's more invasive. Exported weights have been tested against the PyTorch version, and quantization showed minimal degradation for grounding usage. I'm not sure if this is the right way to contribute, but as I've got it working e2e (one Apple Silicon v.s. one Ubuntu + 3090 machine), it'd be nice to share regardless!

To use the fork, please pull and build the branch. Server mode needs --special to preserve the coordinate tokens. For detailed usage, please refer to the HuggingFace model card here.

Grounding call to server mode example, assuming screen.jpg:

PORT=8095
PROMPT="Locate the Apple logo."

# Media marker so that the llama server knows where to splice the image.
# Use LLAMA_MEDIA_MARKER env if customisation
# For now, we fetch the (randomized) media marker and base64-encode the image to a file
MARKER=$(curl -s http://127.0.0.1:$PORT/props | jq -r .media_marker)

# Convert image to base64 for payload
base64 -w0 ./screenshot.jpg > /tmp/screenshot.b64

# Assemble request payload
jq -n --arg p "<|im_start|>user
${MARKER}${PROMPT}<|im_end|>
<|im_start|>assistant
" --rawfile img /tmp/screenshot.b64 \
  '{prompt:{prompt_string:$p,multimodal_data:[$img]},n_predict:64,temperature:0}' > /tmp/la_payload.json

# POST to llama-server
curl -s http://127.0.0.1:$PORT/completion -H "Content-Type: application/json" -d @/tmp/la_payload.json

I hope this is useful! 🙏

[chandra-ps612]

I've successfully added Nvidia/LocateAnything-3B support in llama.cpp framework locally.

[richiejp]

Are you planning to submit a PR for this? @chandra-ps612

[chandra-ps612]

@richiejp
Yupp, very soon!

[suspicious-pineapple]

is your fork available somewhere?

[kalle07]

pls merge into main ! ;)

[pritam-dey3]

we are waiting very eagerly @chandra-ps612 ! Don't mean to rush you though.. just expression of my eagerness!

[sfallah]

@pritam-dey3
#24749
It is still a vibe-coded DRAFT and will take a while before it is ready for merge into master, but it runs.