Optimal parameters for parallel inference using llama-server

[aheid]

First off, hats off to all llama.cpp contributors, it's truly an amazing open-source project in terms of making local AI accessible.

Now, I'm getting my feet wet with SFT, and would like to generate synthetic training data. As such I have a bunch of prompts that I feed llama-server. However I'm not sure I'm extracting the best performance possible from my hardware. I've mostly just done single-issue inference such as chatting or agentic coding, and since this project moves so fast it's hard to know if advice given a year ago is valid today.

I got an AMD 5900X CPU, 96GB RAM, and a NVIDIA 5090 GPU, and I'm running an up-to-date Windows 11 with hardware-accelerated GPU scheduling enabled and CUDA toolkit 12.8. I'm currently using b7488. My current command-line is

llama-server -m %models%\gpt-oss-20b-MXFP4.gguf --no-mmap -c 65536 -np 16 -ngl 99 --reasoning-format none --jinja -fa on --top-k 0 --top-p 0 --min-p 0.05 -b 8192 -ub 1024 --cont-batching

What I'm observing is that I'm not really getting a significant speedup of -np beyond 4 or so, with GPU utilization at around 30% as reported by Windows and GPU-Z. If I set -np 1 then GPU utilization goes up to 60% but average inference speed is roughly the same or slightly lower.

Is this to be expected for this model, or are there some knobs I can turn to make it much better? I've seen --swa-full and --kv-unified, but was not sure if they were relevant nor how to use them correctly if so.

For reference here's some current benchmark results (which has about 80-90% GPU utilization):

model	size	params	backend	ngl	n_ubatch	mmap	test	t/s
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	512	0	pp512	8216.90 ± 76.02
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	512	0	tg1024	297.62 ± 5.30
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1024	0	pp512	8067.09 ± 89.42
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	1024	0	tg1024	298.63 ± 1.11
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	2048	0	pp512	7935.34 ± 221.56
gpt-oss 20B MXFP4 MoE	11.27 GiB	20.91 B	CUDA	99	2048	0	tg1024	295.00 ± 3.15

build: 408616a (7488)

llama-bench -m %models%\gpt-oss-20b-MXFP4.gguf --mmap 0 -ngl 99 -fa on -b 2048 -ub 512,1024,2048 -n 1024

[am17an]

For 5090 you can try #17906 (should be merged soon), it should give a nice PP boost (~20-25%)

[aheid]

That looks very spiffy. I guess I'll have to bite the bullet and get this thing compiled from scratch. Still having dependency nightmares from when I contributed to C++ programs some decades ago.

[am17an]

This is merged now so you can give it a try.

EDIT: not available in release binaries yet, you will need to still compile from source. We're working on distributing it via official releases. (See #18383)

[ggerganov]

For gpt-oss use --top-p 1 instead of --top-p 0:

llama-server -m %models%\gpt-oss-20b-MXFP4.gguf --no-mmap -c 65536 -np 16 --top-k 0 --top-p 1 --min-p 0.05 --temp 1.0 -ub 512

Post the results from this command to see how your setup scales with batch size:

llama-batched-bench -m %models%\gpt-oss-20b-MXFP4.gguf -c 65536 -b 2048 -ub 512 -npp 1024 -ntg 32 -npl 1,2,3,4,6,8,12,16,24,32

[aheid]

Thanks, I'll try --top-p 1. Using --top-p 0 worked fine during chatting and I kinda assumed --top-p 0 would do the same as --top-p 1.

As for bechmarking, here are the results. The GPU utilization started out at around 75% and peaked midway at around 90-96%. So far above what I see from llama-server.

main: n_kv_max = 65536, n_batch = 2048, n_ubatch = 512, flash_attn = -1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -1, n_threads = 12, n_threads_batch = 12

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
1024	32	1	1056	0.160	6408.13	0.108	295.25	0.268	3937.64
1024	32	2	2112	0.185	11055.51	0.352	181.72	0.537	3929.77
1024	32	3	3168	0.278	11038.37	0.348	276.11	0.626	5060.76
1024	32	4	4224	0.367	11159.24	0.379	337.66	0.746	5661.22
1024	32	6	6336	0.552	11134.05	0.391	491.60	0.942	6723.36
1024	32	8	8448	0.734	11156.52	0.421	608.00	1.155	7312.19
1024	32	12	12672	1.113	11041.54	0.592	648.82	1.705	7433.44
1024	32	16	16896	1.483	11046.18	0.547	936.64	2.030	8323.72
1024	32	24	25344	2.212	11112.81	0.610	1259.63	2.821	8983.41
1024	32	32	33792	2.930	11183.11	0.716	1430.11	3.646	9267.83

edit: given how much it likes batching, let me run some more tests against llama-server to ensure I'm loading it down as much as possible and it's not on the client side.

[aheid]

I've verified that my client can easily saturate the server by making a dummy server, so that shouldn't be an issue.

Using the following command-line I get 60% GPU utilization:
llama-server -m %models%\gpt-oss-20b-MXFP4.gguf --no-mmap -c 65536 -np 32 -ngl 99 --reasoning-format none --jinja -fa on --top-k 0 --top-p 1 --min-p 0.05 -b 8192 -ub 1024 --cont-batching --swa-full

I added --swa-full which made some warning about full prompt reprocessing go away. I assume that's a good thing?

Looking at the server performance stats printed, I see token generation hovers at around 25 t/s. If we multiply this with 32 we get ~800 t/s per batch, which is ~56% of the 32-batch llama-batched-bench numbers which had 96% GPU utilization.

Just to be sure this wasn't a Windows thing, I installed Linux on an external drive and compiled the same b7488 release of llama.cpp using CUDA 13. I got nearly identical numbers both for llama-bench and my client code against llama-server. Again in Linux I did notice somewhat improved speed by lowering parallelism to -np 4. So I think we can rule out OS and CUDA version.

I'll live with these results, mostly just curious if there was a way to do better. Perhaps it's just Amdahl's law and my somewhat aging CPU?

On a side note, is it expected that llama-batched-bench numbers for batch size 2 is significantly lower than batch size 1, 3 or 4?

[ggerganov]

I added --swa-full which made some warning about full prompt reprocessing go away. I assume that's a good thing?

Using --swa-full is not really recommended - it will increase the used VRAM for the KV cache significantly for almost no benefits.

The llama-batched-bench does not measure the time for the sampling - it's done on the CPU and when running the real llama-server it makes a difference, especially for big parallelization cases (i.e. we need to sample for each parallel sequence). So my guess is that the lower GPU utilization with llama-server is mostly because of that.

This will be improved with the backend sampling support in #17004 (comment), though it is still not ideal and needs some work in order to behave optimally for -np > 1.

I'll live with these results, mostly just curious if there was a way to do better.

If you are going to keep the server constantly saturated at 32 parallel requests, then try using #17004 (with CUB 3.2) and enabling --backend-sampling. But note that if you don't saturate it, then you will likely not see an improvement with the current implementation.

On a side note, is it expected that llama-batched-bench numbers for batch size 2 is significantly lower than batch size 1, 3 or 4?

My guess is that the ggml_mul_mat_id for small batch sizes could be improved in the CUDA backend. This is how it looks like with the Metal backend on M2 Ultra:

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
1024	32	1	1056	0.434	2361.53	0.248	128.83	0.682	1548.36
1024	32	2	2112	0.852	2403.51	0.379	169.00	1.231	1715.98
1024	32	3	3168	1.277	2405.24	0.494	194.20	1.772	1788.27
1024	32	4	4224	1.700	2409.42	0.606	211.13	2.306	1831.54
1024	32	6	6336	2.545	2414.38	0.815	235.46	3.360	1885.61
1024	32	8	8448	3.385	2420.25	1.042	245.67	4.427	1908.37
1024	32	12	12672	5.084	2417.14	1.313	292.45	6.397	1981.01
1024	32	16	16896	6.780	2416.34	1.583	323.54	8.363	2020.32
1024	32	24	25344	10.153	2420.45	2.127	361.10	12.280	2063.79
1024	32	32	33792	13.540	2420.05	1.954	523.93	15.495	2180.88

[am17an]

#18958 should fix the weirdness around small ( > 1) batch sizes