I'm getting only half the speed compared to llama.cpp, what could I do?

[CHNtentes]

I'm using R7 7700X + RTX 4070 + DDR5 6000 32GB, with following command:

.\llama-server.exe --port 9999 --device CUDA0 --temp 1.0 --min-p 0.0 --top-k 20 --top-p 0.95 --jinja -fa on -m D:\Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q3_K_P.gguf -c 131072 -ctk q8_0 -ctv q8_0 -np 1 --alias local -t 12 --fit on --reasoning on --chat-template-kwargs '{\"preserve_thinking\": true}' -ub 2048 --no-mmap

With llama.cpp I get:

prompt eval time =     241.05 ms /    17 tokens (   14.18 ms per token,    70.52 tokens per second)
       eval time =   37713.43 ms /  2407 tokens (   15.67 ms per token,    63.82 tokens per second)
      total time =   37954.48 ms /  2424 tokens

And with ik_llama.cpp (changed --fit on to --fit, other parameters are the same) I get:

prompt eval time =     571.89 ms /    17 tokens (   33.64 ms per token,    29.73 tokens per second)
       eval time =   82240.97 ms /  2364 tokens (   34.79 ms per token,    28.74 tokens per second)
      total time =   82812.86 ms /  2381 tokens

[ikawrakow]

With an R7 7700X in the mix you should continue using llama.cpp.

See this answer instead.

[CHNtentes]

Thanks for your reply! What kind of setup would benefit from ik_llama.cpp?

[ikawrakow]

CPU-only, CUDA-only, hybrid CUDA/CPU. 2+ Nvidia GPU's will give you graph parallel. Depending on model, this can be better than the more recently added split mode tensor in llama.cpp.

[SamuelOliveirads]

Really? I thought a Zen 4 with an RTX 4000 series GPU would perform well here. Before switching to a Threadripper, I was using an R7 5700X, and it actually performed better here, you just need to set the right parameters.

[FNsi]

Maybe he is using 4070 8g? Then the model cannot fit in vram and explained his speed; also that pp is too low, and I double check it's 3.6A3B, so his setting let that slower than my cpu...

[ikawrakow]

Oops, I was distracted by something else yesterday and thought "R7 7700X" refers to some AMD GPU that I don't know about and we are dealing with a system with 1 AMD and 1 Nvidia GPU, which does not work in ik_llama.cpp (one cannot use both GPUs).

I guess, --fit is not doing a good job. The better option would be to experiment with --n-cpu-moe until it fits and runs. For instance,

.\llama-server.exe --port 9999 --device CUDA0 --temp 1.0 --min-p 0.0 --top-k 20 --top-p 0.95 --jinja -fa on \
     -m D:\Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q3_K_P.gguf \
     -c 131072 -ctk q8_0 -ctv q8_0 -np 1 --n-cpu-moe 30 -wgt 1 -ub 2048 --no-mmap
     --alias local -t 12 --reasoning on --chat-template-kwargs '{\"preserve_thinking\": true}'

uses about 7725 MiB of VRAM with an IQ4_XS version of Qwen-3.6-35B-A3B (I don't feel like downloading the specific model you are using). With that, I get 80 t/s on one of my computers (RTX-4080 + Ryzen-7950X). If I leave all MoE tensors on the CPU (--n-cpu-moe 99 or --cpu-moe), I still get about 65 t/s with this setup. With partial offload, token generation performance is dominated by CPU memory bandwidth, so your 7700X shouldn't be worse than my 7950X (it should be actually better with the DDR5 6000 that you have).

[CHNtentes]

Thanks for your reply again. I tried your new command, unfortunately the performance get worse, and it keeps printing:

Allocating 17.71 GiB of pinned host memory, this may take a while.
Using pinned host memory improves PP performance by a significant margin.
But if it takes too long for your model and amount of patience, kill the process and run using

GGML_CUDA_NO_PINNED=1 your_command_goes_here
    done allocating 17.71 GiB in 0.1 ms

ggml_cuda_host_malloc: failed to allocate 18131.59 MiB of pinned memory: out of memory
llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/41 layers to GPU
llm_load_tensors:        CPU buffer size = 18131.59 MiB

I tried adjusting the value of --n-cpu-moe, from 1 to 99, but this issue persists. It seems that all layers are offloaded to CPU so it's much slower.

[ikawrakow]

Oh, you need to add -ngl 100. And if the pinned memory doesn't work, do as the log message says

[GlisseManTV]

Hi, I'm also having lower TG than in llama.cpp.

For an example I got 20/30 tps with llama.cpp and ik_llama I got like 2-5TPS

Setup:
2*RTX3060 12GB
xeon E5 2683 v4 32gb ddr4

There are my commands:
llama.cpp

SET GGML_CUDA_GRAPH_OPT=1
SET CUDA_VISIBLE_DEVICES=0,1,2
llama-server -hf lordx64/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-Q5_K_M-GGUF:Q5_K_M --mmproj "C:\Users\appuser\Downloads\mmproj-BF16.gguf" --no-mmproj-offload --host 0.0.0.0 --api-key "fS9ygN7grbFzeR7pdJ&@NU7AzoU3X^T4" --jinja --fit-ctx 262144 -ctk bf16 -ctv bf16 --presence-penalty 1.5 --flash-attn 1 --no-warmup --spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-draft-n-min 12 --spec-draft-n-max 48 --no-mmap -dio --fit-target 256 --fit on --checkpoint-every-n-tokens 4096 --ctx-checkpoints 256 -np 1 --kv-unified

ik_llama.cpp

SET GGML_CUDA_GRAPH_OPT=1
SET CUDA_VISIBLE_DEVICES=0,1,2
SET GGML_CUDA_NO_PINNED=1
C:\temp\ik_llama.cpp\build\bin\Debug\llama-server.exe -m "C:\Users\appuser\.cache\huggingface\hub\models--lordx64--Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-Q5_K_M-GGUF\blobs\08aef0f2d9fa9ad3d4c95b6748024c44623fb81ad68b33b8e3773ac379f07011" --mmproj "C:\Users\appuser\Downloads\mmproj-BF16.gguf" --no-mmproj-offload --host 0.0.0.0 --api-key "fS9ygN7grbFzeR7pdJ&@NU7AzoU3X^T4" --jinja -c 262144 -ctk bf16 -ctv bf16 --presence-penalty 1.5 --flash-attn 1 --no-warmup --no-mmap --ctx-checkpoints 256 -np 1 -ngl 100 -wgt 1 --n-cpu-moe 15

I tried your advice, but nothing seems working fine.

do you have any trick to help me?