DiffusionGemma

[AlejandroJL]

Failed to load model.

Error when loading model: ValueError: Model type diffusion_gemma not supported. Error: No module named 'mlx_vlm.speculative.drafters.diffusion_gemma'

Thanks!

[Jakeshadow]

LM Studio's MLX engine doesn't support diffusion_gemma yet
unfortunately. I hit the same wall.

Two options that work right now while waiting for native support:

vLLM (fastest, but needs NVIDIA GPU):
pip install vllm --pre
python -c "from vllm import LLM; llm = LLM('google/diffusiongemma-26B-A4B-it'); print(llm.generate('Hello'))"

llama.cpp PR branch (works on anything, including Mac):
git clone -b pr/24427 https://github.com/ggerganov/llama.cpp.git
cd llama.cpp && make -j
./llama-cli -m diffusiongemma-Q4_K_M.gguf -p "Hello"

Both give 200-1000+ tok/s depending on GPU. I've got a full
comparison of all the methods that actually work here:
https://diffusiongemma.dev/

Fair warning: Apple Silicon doesn't get the speedup NVIDIA does
with this model — it really wants CUDA tensor cores.