Prevent llamacpp defaults from locking up consumer hardware #6870
Conversation
|
I'm concerned about your findings potentially not being general and applying just to your particular system. My policy was to follow the defaults in llama.cpp; for the batch size, llama-cpp-python had 512 as the default for a long time, but after seeing the 2048 below, I changed it About the system ceasing to be interactive, wouldn't that be best handled by setting the 'nice' level of llama-server in the launch command?
This is the first time I read that as well, do you have evidence to support that? |
My Goalhere is to be generalized to consumer level hardware. Power users with HEDTs or H100s can finagle settings later. But if you want this to be as accessible as possible to generic 'user' then conservative safe defaults are best. Ideally most of these settings should detect a 'safe' value or use the model defaults, whichever is lower. That can be done later. You know, 80/20 rule. --threads --threads-batch and --batch-sizeJust to clarify threads, thread-batch are cpu specific arguments. Only batch_size affects both cpu and gpu. KV Cache calculationTo clarify how batch size impacts gpu and memory pressure. Here is a breakdown of the KV cache formula: On being niceNiceness does not do anything about io resource contention. Io resource contention examples are PCIe saturation, Memory swap storms, IRQ floods and too many threads under pressure. Once pcie bandwidth is saturated, you get to sit there and either hard reboot or wait for your desktop to come back to life. This can take a long time sometimes. In addition, suppose you flood your system with 2000 threads. And it dutifully chugs along checking each one. Each one starts a memory or pcie transfer and goes to sleep. Now you're in an IRQ storm for each transfer that completes. And each thread may need many transfers. Every irq is going to interrupt the cpu leading to very bad latency. Being nice here doesn't help either. For example. In my own testing, changing batch-size from 2048 to 256 saved 600MB Vram with no noticable drop in tok/s on Gemma3 12B Q4 @ 14k context, but could very well prevent a lock up on many other systems. So my tl;dr is, batch-size shouldn't be tuned by default for peak throughput. It should be tuned for the widest usable default on consumer hardware. Model aware defaults (Not now later) or just 256/512 keeps it accessible. On quantization matching cache formatThis is true by definition. Using FP16 KV cache for a Q4 model does not improve quality but does quad the memory cost. Once a model is quantized below FP16, you aren't going to magically get FP16 accuracy back out. The only exception would be for a lora but the gains would probably not be worth the memory. |
|
I was looking at how ollama handles this to have an additional reference, and apparently they use defaultThreads := systemInfo.GetOptimalThreadCount()
if opts.NumThread > 0 {
params = append(params, "--threads", strconv.Itoa(opts.NumThread))
} else if defaultThreads > 0 {
params = append(params, "--threads", strconv.Itoa(defaultThreads))
}which itself uses // Return the optimal number of threads to use for inference
func (si SystemInfo) GetOptimalThreadCount() int {
if len(si.System.CPUs) == 0 {
return 0
}
coreCount := 0
for _, c := range si.System.CPUs {
coreCount += c.CoreCount - c.EfficiencyCoreCount
}
return coreCount
}see https://github.com/ollama/ollama/blob/main/llm/server.go and https://github.com/ollama/ollama/blob/main/discover/types.go. The second function can't be done in a cross-platform way in Python because it doesn't have easy access to the number of 'efficiency cores'. How does your code compare to this? As to the batch size, there the default is 512 https://github.com/ollama/ollama/blob/424f648632c925ce14a75018c4dcab395e035993/api/types.go#L671; since you mention that 256 was your optimal value but 256/512 are acceptable, I think that we could go for 512. |
|
About recommending quantizing the kv cache, I haven't seen this recommendation in the context of EXL2, which uses a more sophisticated kv cache quantization algorithm than llama.cpp. There it was always framed as an optional feature. I think it's best to not recommend quantizing the cache by default. |
|
My code just does a 'max(1, int(os.cpu_count() / 2))', which should be safe regardless of cpu. I know there are people without HT or who have efficiency cores but that should be a 'safe' number, I'm not aware of a way via python to detect an efficiency core. If you don't mind bringing psutils in, it has a function to identify physical cores. https://psutil.readthedocs.io/en/latest/index.html#psutil.cpu_count Bringing in psutils might be good idea overall because it would allow setting automatic 'safe' values for a whole host of things or possibly benchmarking at a later date to attempt to detect ideal cpu settings. I'm not aware of a cross platform way to detect ideal gpu settings. Maybe just a gpu reference list or something? Or platform specific modules could be developed at a later time. My code doesn't change the quantization level, it just adds further description for it. EXL2 people seems to think Q4 is optimal. Seeing how the behavior might be very different between llamacpp and exl, perhaps having separate settings for this is optimal. Reading up, it seems the EXL2 people consider their Q4 better than FP8 and almost as good as FP16. So the choice should really be FP16 or a Q type for EXL.2 As for batch-size, 512 is good. The only reason I picked 256 is because of no noticeable performance change on my rig and I figured someone with low end hardware might benefit from reduced vram util. Let me know if you want me to try to break the settings out and/or bring in psutil. I'll make the all the appropriate changes then. |
|
I wouldn't mind adding an additional requirement (preferably a simple, no-dependencies one). Two things would benefit from this:
These things are discussed in this issue where this library is discussed https://github.com/ModelCloud/Device-SMI/ but apparently it doesn't explicit say the number of 'efficiency cores', just 'cores' and 'threads'. Also the version on pypi doesn't work on my computer with |
|
I ran 108 speed comparisons for
where I measured prompt processing speed and text generation speed (3 times for each combination, taking the average of the 3 measurements). That gave me 8 columns (prompt processing and text generation for (2 systems) * (2 webui versions)); for each column, I divided values by the maximum for that column, and took the harmonic mean of those 8 values. I got this as the global maximum: Here threads = threads-batch = (physical + logical cores). If I look only at the GPU runs, I get Here threads = threads-batch = (number of physical cores). If I look only at CPU, I get Where threads = physical cores and threads-batch = physical + logical cores. llama.cpp by default sets threads and threads-batch according to the maximum for the GPU case (which includes a case with CPU offloading), so I don't think it's necessary to change that default. But the optimal batch size turned out to be 256, which aligns with what you found. |
2 participants
Checklist:
$ pyflakes modules/shared.py
$ pycodestyle modules/shared.py
$
Default of 0 for threads and batch_threads means to spawn as many threads as there are tokens all at once. This can cause periods where a desktop can cease being interactive, high memory use and swapping on some setups. So I changed to thread=1/2 cpu count and batch_threads=2*cpu count. This should result in about 80% cpu util. People who have H100s that run completely on gpu hardware can up this as needed.
Lowered batch-size as well. 256 is 'safe' for consumer hardware. Uses slightly less vram and system ram and prevents periods of loss of interactivity on the desktop. batch-size -> number of tokens to process at once. When this was set to 2k and batch_threads was set to 0, it would drop 2k threads on cpu/gpu at a time. Obviously not acceptable for consumer hardware given io and interface bandwidth limits. A lower number will actually improve throughput on consumer hardware as it lower resource contention.
Added additional kv cache description. Cache should match the ggufs quantization level. There is no benefit using a fp16 cache for a q_4 model. It just means your cache can only hold 1/4 as many items with no accuracy benefit. Using the correct cache level can increase performance a lot.