mlx backend: degenerate looping output with gemma-4 E4B (chat template apparently not applied)

[enneo-service]

LocalAI version: 4.4.1 (Homebrew bottle)
Environment: macOS (Apple Silicon, M1 Max), darwin/arm64
Backend: mlx (darwin image installs fine)

What happened

Serving mlx-community/gemma-4-E4B-it-qat-4bit through the mlx backend produces degenerate, looping output that echoes the prompt — the chat template does not seem to be applied.

Model config:

name: gemma-qat-mlx
backend: mlx
parameters:
  model: mlx-community/gemma-4-E4B-it-qat-4bit

Request:

curl http://127.0.0.1:1240/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"gemma-qat-mlx","messages":[{"role":"user","content":"Reply with exactly: MLX inside LocalAI works"}],"max_tokens":30}'

Response content:

 exactly: MLX inside LocalAI works exactly: MLX inside LocalAI works exactly: MLX inside Local

— prompt-fragment echo repeated until max_tokens, on every request.

Expected

The same checkpoint served by mlx_vlm.server (mlx-vlm 0.6.2) on the same machine replies correctly ("MLX inside LocalAI works"), so the weights and the MLX runtime are fine — the difference points at prompt/chat-template handling in the LocalAI mlx backend (gemma-4 E4B is a vision-language architecture; possibly the backend applies a plain-LM template or none at all).

Secondary observation

Cold load through the backend took ~82 s for this model; mlx_vlm.server loads the same checkpoint from the same HF cache in ~10 s. Worth a look once the output issue is addressed.

Happy to provide debug logs or test patches on this machine.

[localai-bot]

Fixed by #10274 (merged).

Root cause: gemma-4 E4B is a vision-language model (it declares the image-text-to-text pipeline tag on HuggingFace), but the mlx importer routed every mlx-community/* model to the text-only mlx-lm backend. That backend's tokenizer doesn't carry the VLM processor's chat template, so the template was effectively not applied and you got the prompt-echo loop. mlx_vlm.server works because it resolves the template from the processor — which is exactly what LocalAI's mlx-vlm backend does.

What changed:

• The importer now detects the image-text-to-text pipeline tag and routes such models to the mlx-vlm backend automatically.
• The plain mlx backend now warns loudly when a model has no chat template, so a misroute surfaces instead of silently looping.

⚠️ One thing for your specific config: an explicit backend: always wins, so the hand-written backend: mlx in your YAML won't be auto-corrected. For that model, set backend: mlx-vlm (or re-import it through the importer, which will now pick mlx-vlm for you). The processor-aware chat template will then be applied and the output should be correct.

The secondary cold-load time observation (~82s vs ~10s) is a separate matter and not addressed here — happy to track it in its own issue if you'd like.

Thanks for the precise report and the mlx_vlm.server comparison; it pointed straight at the routing.