Which llama.cpp Windows package to choose for my hardware

[domasofan]

Hi all,

I am working with llama.cpp for a few weeks now.
I am using Windows 11, i have a Intel I5-12500 Processor, Intel UHD Graphics 770 GPU and 32 GB RAM.
Various packages are provided in the Github releases section.
Currently I am using the AVX2 build of llama.cpp which works quite well.
My question is which package would fit my processor best so i can get the maximum out of my hardware?

Greetings and thanks,
Simon

[AbraxasVi]

You can try SYCL, OpenBLAS, and Vulkan buildings, these versions can call GPU resources to run llama.cpp, which may bring some help to improve performance/speed.

[domasofan]

Hi,

Thanks for the info.

Just out of curiosity i did some benchmarks on llama.cpp Build 5193 with various backends.
I had to break the benchmark a bit because i canceled the first one before it finished.

The command i used:
$ llama-bench --flash-attn 1 --model "c:\ai\models\Qwen2.5-7B-Instruct-Q4_K_M.gguf" -pg 1024,256

Results:

Vulkan:

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) UHD Graphics 770 (Intel Corporation) | uma: 1 | fp16: 1 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none

model	size	params	backend	ngl	fa	test	t/s
qwen2 7B Q4_K - Medium	4.36 GiB	7.62 B	Vulkan,RPC	99	1	pp512	18.69 ± 0.02
qwen2 7B Q4_K - Medium	4.36 GiB	7.62 B	Vulkan,RPC	99	1	tg128	4.24 ± 0.00

AVX2:

model	size	params	backend	ngl	fa	test	t/s
qwen2 7B Q4_K - Medium	4.36 GiB	7.62 B	RPC	99	1	pp512	30.64 ± 0.10
qwen2 7B Q4_K - Medium	4.36 GiB	7.62 B	RPC	99	1	tg128	6.90 ± 0.01

SYCL:

model	size	params	backend	ngl	fa	test	t/s
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
qwen2 7B Q4_K - Medium	4.36 GiB	7.62 B	SYCL	99	1	pp512	42.76 ± 0.06
qwen2 7B Q4_K - Medium	4.36 GiB	7.62 B	SYCL	99	1	tg128	2.55 ± 0.00

OpenBLAS:

model	size	params	backend	threads	fa	test	t/s
qwen2 7B Q4_K - Medium	4.36 GiB	7.62 B	BLAS,RPC	6	1	pp512	34.64 ± 0.12
qwen2 7B Q4_K - Medium	4.36 GiB	7.62 B	BLAS,RPC	6	1	tg128	6.90 ± 0.00