Eval bug: Gemma 4 E4B crashes with corrupt stack on 94a220cd6

[moontato]

Name and Version

$./llama-server --version
version: 9496 (94a220c)

Operating systems

Linux

GGML backends

CUDA

Hardware

Nvidia AGX Orin

Models

gemma-4-E4B-it-Q8_0 (Unsloth)

Problem description & steps to reproduce

After pulling on 2026-06-03, Gemma 4 E4B crashes on every inference attempt. The previous pull (aa50b2c, 2026-05-27) worked fine. Rebuilding at aa50b2c fixes the crash, confirming this is a regression.

Build command used:

CUDACXX=/usr/local/cuda/bin/nvcc cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES="87" \
  -DGGML_CUDA_F16=ON \
  -DGGML_CUDA_NO_VMM=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build --config Release -j8

First Bad Commit

No response

Relevant log output

Logs

Jun 03 17:36:52 orinserver llama-server[1331339]: [56595] 0.00.802.665 I srv    load_model: loading model '/ssd/llamacpp_models/gemma-4-E4B-it-Q8_0.gguf'
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] /ssd/llama.cpp/tools/mtmd/clip.cpp:4391: Unknown projector type
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] [New LWP 1332389]
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] [New LWP 1332390]
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] [New LWP 1332391]
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] [New LWP 1332394]
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] [New LWP 1332395]
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] [New LWP 1332396]
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] [New LWP 1332397]
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] [New LWP 1332398]
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] [New LWP 1332399]
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] [New LWP 1332400]
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] [New LWP 1332401]
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] [New LWP 1332402]
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] [New LWP 1332403]
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] [New LWP 1332404]
Jun 03 17:36:53 orinserver llama-server[1331339]: [56595] [New LWP 1332405]
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] [Thread debugging using libthread_db enabled]
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] 0x0000ffff91bd2c0c in __GI___wait4 (pid=<optimized out>, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] 27        ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] #0  0x0000ffff91bd2c0c in __GI___wait4 (pid=<optimized out>, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] 27        in ../sysdeps/unix/sysv/linux/wait4.c
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] #1  0x0000ffff912bf41c in ggml_print_backtrace () from /ssd/llama.cpp/build/bin/libggml-base.so.0
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] #2  0x0000ffff912bf5b0 in ggml_abort () from /ssd/llama.cpp/build/bin/libggml-base.so.0
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] #3  0x0000ffff916ceb64 in clip_n_mmproj_embd(clip_ctx const*) () from /ssd/llama.cpp/build/bin/libmtmd.so.0
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] #4  0x0000ffff916cecec in clip_graph::clip_graph(clip_ctx*, clip_image_f32 const&) () from /ssd/llama.cpp/build/bin/libmtmd.so.0
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] #5  0x0000ffff916cefc8 in clip_image_build_graph(clip_ctx*, clip_image_f32_batch const&) () from /ssd/llama.cpp/build/bin/libmtmd.so.0
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] #6  0x0000ffff916d7e0c in clip_model_loader::reserve_compute_meta(clip_ctx&, clip_image_f32_batch const&) () from /ssd/llama.cpp/build/bin/libmtmd.so.0
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] #7  0x0000ffff916d822c in clip_model_loader::warmup(clip_ctx&, clip_image_f32_batch const&) () from /ssd/llama.cpp/build/bin/libmtmd.so.0
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] #8  0x0000ffff916d51e8 in clip_init(char const*, clip_context_params) () from /ssd/llama.cpp/build/bin/libmtmd.so.0
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] #9  0x0000ffff916567a0 in mtmd_get_memory_usage(char const*, mtmd_context_params) () from /ssd/llama.cpp/build/bin/libmtmd.so.0
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] #10 0x0000ffff91ff6084 in server_context_impl::load_model(common_params&) () from /ssd/llama.cpp/build/bin/libllama-server-impl.so
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] #11 0x0000ffff91f53abc in llama_server(int, char**) () from /ssd/llama.cpp/build/bin/libllama-server-impl.so
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] #12 0x0000ffff91b4fe10 in __libc_start_main (main=0xaaaac1921150 <main>, argc=47, argv=0xffffec684648, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:308
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] 308        ../csu/libc-start.c: No such file or directory.
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] #13 0x0000aaaac1921188 in _start ()
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Jun 03 17:36:54 orinserver llama-server[1331339]: [56595] [Inferior 1 (process 1332384) detached]

[jettoblack]

Yes b9496 also broke loading ggml-org/gemma-4-E4B-it-GGUF (not just Unsloth)

llama-server -hf ggml-org/gemma-4-E4B-it-GGUF fails with:
llama.cpp/tools/mtmd/clip.cpp:4391: Unknown projector type

[sdroege]

Commit breaking it is a731805 according to git bisect

[ngxson]

fixed: #24091