Skip to content

Vulkan: add GGML_OP_FUSED_MUL_UNARY#580

Merged
ikawrakow merged 1 commit into
mainfrom
ik/vulkan_fused_mul_unary
Jul 3, 2025
Merged

Vulkan: add GGML_OP_FUSED_MUL_UNARY#580
ikawrakow merged 1 commit into
mainfrom
ik/vulkan_fused_mul_unary

Conversation

@ikawrakow

Copy link
Copy Markdown
Owner

The tiniest of performance increases, barely measurable.

But now we no longer need to special-case Vulkan when building the graph.

Of note: I went to measure mainline llama.cpp Vulkan performance with my setup (RTX-4080 in a Ryzen-5975WX box, Nvidia driver 575, so coopmat2 enabled). Interestignly enough, ik_llama.cpp is 10-15% faster than mainline for a context of 16k tokens, even though there aren't any noticeable Vulkan optimizations in ik_llama.cpp yet. Is mainline paying the price for the grand KV cache unification?

Here sweep-bench results for LlaMA-3.1-8B-Instruct

Mainline llama.cpp (build: 5821)

PP TG N_KV T_PP s S_PP t/s T_TG s S_TG t/s
1024 256 0 0.244 4200.63 2.742 93.37
1024 256 1024 0.256 4002.55 2.730 93.78
1024 256 2048 0.282 3634.36 2.827 90.57
1024 256 3072 0.297 3452.54 2.897 88.36
1024 256 4096 0.319 3212.57 2.956 86.62
1024 256 5120 0.335 3056.04 3.045 84.08
1024 256 6144 0.349 2937.19 3.126 81.90
1024 256 7168 0.365 2807.69 3.184 80.40
1024 256 8192 0.378 2710.13 3.284 77.97
1024 256 9216 0.396 2589.00 3.364 76.10
1024 256 10240 0.407 2514.06 3.453 74.13
1024 256 11264 0.424 2415.06 3.518 72.77
1024 256 12288 0.441 2322.30 3.621 70.70
1024 256 13312 0.455 2249.22 3.704 69.12
1024 256 14336 0.468 2190.30 3.786 67.62
1024 256 15360 0.485 2111.54 3.852 66.45

ik_llama.cpp with this PR

PP TG N_KV T_PP s S_PP t/s T_TG s S_TG t/s
1024 256 0 0.242 4234.03 2.673 95.77
1024 256 1024 0.253 4052.19 2.646 96.74
1024 256 2048 0.265 3866.73 2.718 94.20
1024 256 3072 0.283 3618.48 2.775 92.25
1024 256 4096 0.286 3584.28 2.830 90.45
1024 256 5120 0.298 3441.80 2.905 88.13
1024 256 6144 0.321 3194.99 2.983 85.81
1024 256 7168 0.335 3059.90 3.034 84.37
1024 256 8192 0.340 3007.63 3.089 82.87
1024 256 9216 0.353 2897.76 3.128 81.83
1024 256 10240 0.364 2814.86 3.192 80.21
1024 256 11264 0.372 2753.81 3.241 78.99
1024 256 12288 0.380 2692.31 3.291 77.78
1024 256 13312 0.397 2580.87 3.370 75.97
1024 256 14336 0.412 2486.45 3.444 74.33
1024 256 15360 0.423 2420.49 3.498 73.19

@ikawrakow ikawrakow merged commit 8a0c38f into main Jul 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant