Misc. bug: llama-server aborts when loading model (OpenVINO)

[jduerstock]

Name and Version

$ ./llama-server --version
GGML OpenVINO Backend: device CPU is not available, fallback to CPU
OpenVINO: using device CPU
version: 8920 (15fa3c4)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

./llama-server -hf unsloth/Qwen3.6-27B-GGUF:Q8_0 --host 0.0.0.0

Problem description & steps to reproduce

Program aborts before attempting to serve the first query

First Bad Commit

No response

Relevant log output

$ ./llama-server -hf unsloth/Qwen3.6-27B-GGUF:Q8_0 --host 0.0.0.0
GGML OpenVINO Backend: device CPU is not available, fallback to CPU
OpenVINO: using device CPU
common_download_file_single_online: HEAD failed, status: 404
no remote preset found, skipping
Downloading mmproj-BF16.gguf ─────────────────────────────────────── 100%
Downloading Qwen3.6-27B-Q8_0.gguf ────────────────────────────────── 100%
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build_info: b8920-15fa3c493
system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
Running without SSL
init: using 15 threads for HTTP server
start: binding port with default address family
main: loading model
srv load_model: loading model '.cache/huggingface/hub/models--unsloth--Qwen3.6-27B-GGUF/snapshots/82d411acf4a06cfb8d9b073a5211bf410bfc29bf/Qwen3.6-27B-Q8_0.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
common_params_fit_impl: getting device memory data for initial parameters:
/home/runner/work/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:898: pre-allocated tensor (cache_r_l0 (view) (copy of )) in a buffer (OPENVINO0) that cannot run the operation (CPY)
[New LWP 67607]
[New LWP 67606]
[New LWP 67605]
[New LWP 67604]
[New LWP 67603]
[New LWP 67602]
[New LWP 67601]
[New LWP 67600]
[New LWP 67599]
[New LWP 67598]
[New LWP 67597]
[New LWP 67596]
[New LWP 67595]
[New LWP 67594]
[New LWP 67593]
[New LWP 67592]
[New LWP 67557]

This GDB supports auto-downloading debuginfo from the following URLs:
https://debuginfod.ubuntu.com
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1".
__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
⚠️ warning: 56 ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: No such file or directory
#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
56 in ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S
#1 0x0000706e10ca067c in __internal_syscall_cancel (a1=, a2=, a3=, a4=, a5=0, a6=0, nr=61) at ./nptl/cancellation.c:49
⚠️ warning: 49 ./nptl/cancellation.c: No such file or directory
#2 __syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:75
75 in ./nptl/cancellation.c
#3 0x0000706e10d1cdcf in __GI___wait4 (pid=, stat_loc=, options=, usage=) at ../sysdeps/unix/sysv/linux/wait4.c:30
⚠️ warning: 30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#4 0x0000706e11758273 in ggml_print_backtrace () from libggml-base.so.0
#5 0x0000706e1175841b in ggml_abort () from libggml-base.so.0
#6 0x0000706e11770af8 in ggml_backend_sched_backend_id_from_cur(ggml_backend_sched*, ggml_tensor*) () from libggml-base.so.0
#7 0x0000706e11772b7f in ggml_backend_sched_split_graph () from libggml-base.so.0
#8 0x0000706e114b6d27 in llama_context::graph_reserve(unsigned int, unsigned int, unsigned int, llama_memory_context_i const*, bool, unsigned long*) () from libllama.so.0
#9 0x0000706e114b78c8 in llama_context::sched_reserve() () from libllama.so.0
#10 0x0000706e114bb9e9 in llama_context::llama_context(llama_model const&, llama_context_params) () from libllama.so.0
#11 0x0000706e114bc568 in llama_init_from_model () from libllama.so.0
#12 0x0000706e11a29f77 in common_get_device_memory_data(char const*, llama_model_params const*, llama_context_params const*, std::vector<ggml_backend_device*, std::allocator<ggml_backend_device*> >&, unsigned int&, unsigned int&, unsigned int&, ggml_log_level) () from libllama-common.so.0
#13 0x0000706e11a2ada3 in common_params_fit_impl(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, ggml_log_level) () from libllama-common.so.0
#14 0x0000706e11a2ec62 in common_fit_params(char const*, llama_model_params*, llama_context_params*, float*, llama_model_tensor_buft_override*, unsigned long*, unsigned int, ggml_log_level) () from libllama-common.so.0
#15 0x0000706e119f9531 in common_init_result::common_init_result(common_params&) () from libllama-common.so.0
#16 0x0000706e119fa72c in common_init_from_params(common_params&) () from libllama-common.so.0
#17 0x000063c7d2f310e8 in server_context_impl::load_model(common_params&) ()
#18 0x000063c7d2e8d27a in main ()
[Inferior 1 (process 67555) detached]
Aborted (core dumped) ./llama-server -hf unsloth/Qwen3.6-27B-GGUF:Q8_0 --host 0.0.0.0

[jduerstock]

After a bit of poking around with strace, it seems that it it needs openvino 2026.0.0 installed:
5899 newfstatat(AT_FDCWD, "/lib/openvino-2026.0.0/libopenvino_intel_npu_plugin.so", 0x7ffca9ed3000, 0) = -1 ENOENT (No such file or directory)

Would it make sense, and would it be possible to include the library (or libraries) along with the released package?

[Zulkifli-Intel]

I ran the command ./llama-server -hf unsloth/Qwen3.6-27B-GGUF:Q8_0 --host 0.0.0.0 on my side but encounter other issue:

llama.cpp$ ./build/ReleaseOV/bin/llama-server -hf unsloth/Qwen3.6-27B-GGUF:Q8_0 --host 0.0.0.0
OpenVINO: using device CPU
common_download_file_single_online: HEAD failed, status: -1
no remote preset found, skipping
get_repo_commit: error: HTTPLIB failed: Could not establish connection
get_repo_files: failed to resolve commit for unsloth/Qwen3.6-27B-GGUF
error: failed to download model from Hugging Face

So I downloaded the model manually using wget https://huggingface.co/unsloth/Qwen3.6-27B-GGUF/resolve/main/Qwen3.6-27B-Q8_0.gguf and ran with this command:
./build/ReleaseOV/bin/llama-server -m ~/models/Qwen3.6-27B-Q8_0.gguf --port 8080 -c 1024

I received the same output:

llama_memory_recurrent:  OPENVINO0 RS buffer size =   598.50 MiB
llama_memory_recurrent: size =  598.50 MiB (     4 cells,  64 layers,  4 seqs), R (f32):   22.50 MiB, S (f32):  576.00 MiB
sched_reserve: reserving ...
/home/***/llama.cpp/ggml/src/ggml-backend.cpp:898: pre-allocated tensor (cache_r_l0 (view) (copy of )) in a buffer (OPENVINO0) that cannot run the operation (CPY)
[New LWP 144049]
[New LWP 144047]
[New LWP 144048]
[New LWP 144046]
[New LWP 144045]
[New LWP 144044]
[New LWP 144043]
[New LWP 144037]
[New LWP 144036]
[New LWP 144035]
[New LWP 144034]
[New LWP 144033]
[New LWP 144032]
[New LWP 144031]
[New LWP 144030]
[New LWP 144029]
[New LWP 144028]
[New LWP 144026]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1".
0x0000706daa510813 in __GI___wait4 (pid=144050, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30     ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#0  0x0000706daa510813 in __GI___wait4 (pid=144050, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30      in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x0000706daab58273 in ggml_print_backtrace () from /home/***/llama.cpp/build/ReleaseOV/bin/libggml-base.so.0
#2  0x0000706daab5841b in ggml_abort () from /home/***/llama.cpp/build/ReleaseOV/bin/libggml-base.so.0
#3  0x0000706daab70b28 in ggml_backend_sched_backend_id_from_cur(ggml_backend_sched*, ggml_tensor*) () from /home/***/llama.cpp/build/ReleaseOV/bin/libggml-base.so.0
#4  0x0000706daab72baf in ggml_backend_sched_split_graph () from /home/***/llama.cpp/build/ReleaseOV/bin/libggml-base.so.0
#5  0x0000706daacd95c7 in llama_context::graph_reserve(unsigned int, unsigned int, unsigned int, llama_memory_context_i const*, bool, unsigned long*) () from /home/***/llama.cpp/build/ReleaseOV/bin/libllama.so.0
#6  0x0000706daacda168 in llama_context::sched_reserve() () from /home/***/llama.cpp/build/ReleaseOV/bin/libllama.so.0
#7  0x0000706daacde2b9 in llama_context::llama_context(llama_model const&, llama_context_params) () from /home/***/llama.cpp/build/ReleaseOV/bin/libllama.so.0
#8  0x0000706daacdee38 in llama_init_from_model () from /home/***/llama.cpp/build/ReleaseOV/bin/libllama.so.0
#9  0x0000706dab1fdedd in common_init_result::common_init_result(common_params&) () from /home/***/llama.cpp/build/ReleaseOV/bin/libllama-common.so.0
#10 0x0000706dab1fe88c in common_init_from_params(common_params&) () from /home/***/llama.cpp/build/ReleaseOV/bin/libllama-common.so.0
#11 0x00005cb2b6a84058 in server_context_impl::load_model(common_params&) ()
#12 0x00005cb2b69dc7ba in main ()
[Inferior 1 (process 144019) detached]
Aborted (core dumped)

I tried with the Llama-3.2-1B-Instruct-Q4_0.gguf (validated model) and was able to run Llama.cpp Server with OpenVINO Backend:

sched_reserve: reserving ...
sched_reserve: Flash Attention was auto, set to enabled
sched_reserve: resolving fused Gated Delta Net support:
sched_reserve: fused Gated Delta Net (autoregressive) enabled
sched_reserve: fused Gated Delta Net (chunked) enabled
sched_reserve:  OPENVINO0 compute buffer size =   254.50 MiB
sched_reserve: OPENVINO0_HOST compute buffer size =     6.01 MiB
sched_reserve: graph nodes  = 503
sched_reserve: graph splits = 1
sched_reserve: reserve took 119.52 ms, sched copies = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv    load_model: initializing slots, n_slots = 4
no implementations specified for speculative decoding
slot   load_model: id  0 | task -1 | new slot, n_ctx = 1024
no implementations specified for speculative decoding
slot   load_model: id  1 | task -1 | new slot, n_ctx = 1024
no implementations specified for speculative decoding
slot   load_model: id  2 | task -1 | new slot, n_ctx = 1024
no implementations specified for speculative decoding
slot   load_model: id  3 | task -1 | new slot, n_ctx = 1024
srv    load_model: prompt cache is enabled, size limit: 8192 MiB
srv    load_model: use `--cache-ram 0` to disable the prompt cache
srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
srv          init: init: idle slots will be saved to prompt cache and cleared upon starting a new task
init: chat template, example_format: '<|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 11 May 2026

You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>

Hello<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Hi there<|eot_id|><|start_header_id|>user<|end_header_id|>

How are you?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

'
srv          init: init: chat template, thinking = 0
main: model loaded
main: server is listening on http://127.0.0.1:8080
main: starting the main loop...
srv  update_slots: all slots are idle

This output is expected as the Qwen3.6-27B-Q8_0.gguf model is not yet supported and validated, as it is not listed in the Validated Models.

[jduerstock]

I worked out that part of the problem was a missing shared library.

Is there any ETA on when a 'modern' model (i.e. Qwen3.6 or Gemma4) will be supported?

[Zulkifli-Intel]

I'm not able to share when these models will be enabled. If the models are validated and supported, they will be added to Validated Models list.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.