Request: NUMA support

[frz121]

Could you please add NUMA support to the application?
Essentially, this is just the "--numa" option for llamaсpp.
This allows models to run on multiprocessor systems (linux based) with significant acceleration.

[rogerfachini]

With the release of Deepseek, those of us running on CPU with EPYC chips would get a noticeable performance benefit with this. I've been testing locally using numactl with koboldcpp to force NUMA aware scheduling (discussed in the llama.cpp repo here: ggml-org#1437).

Having this feature available as a flag in kobold.cpp similar to llama.cpp would be quite helpful.

[jasonsi1993]

With the release of Deepseek, those of us running on CPU with EPYC chips would get a noticeable performance benefit with this. I've been testing locally using numactl with koboldcpp to force NUMA aware scheduling (discussed in the llama.cpp repo here: ggml-org#1437).

Having this feature available as a flag in kobold.cpp similar to llama.cpp would be quite helpful.

Mind if i asking what command are you using to run the numactl? I am testing different numactl combination has not see much improvement.

[rogerfachini]

Mind if i asking what command are you using to run the numactl? I am testing different numactl combination has not see much improvement.

Sure, for context the hardware I'm running is a single-socket EPYC 7763; this command may not work as well on multi socket systems (infinity fabric link between sockets becomes a different bottleneck at that point). :

numactl --interleave=all koboldcpp --model ...

Check your BIOS settings to make sure each CCD is being treated as a separate NUMA node.

[ubergarm]

I'm fusing with numa stuff on a big dual socket Intel Xeon 6980P with upstream llama.cpp if anyone has a similar rig or tips I'm all ears: ggml-org#12088

Thanks!

[LostRuins]

If it's just a small change and someone can PR support for it, I wouldn't mind adding it as an option. However I have literally no way to test numa myself, i also don't have any AMD gpu or AMD cpu, so i would be adding it in the blind.

[rogerfachini]

I think it should be a relatively small change? Digging through the code a bit, llama.cpp implements all of the heavy lifting in the llama_numa_init(strategy) function, which should be called after llama_backend_init(). The scope appears to be adding the numa init call, and wiring that up to a new CLI arg.

I'll tinker with this over the weekend, and submit a PR when I get something working.

I wanted to throw my support behind this feature. I am currently using 2 dual-socket Intel Xeon workstations. Because upstream llama.cpp natively supports --numa (distribute), it properly maps memory pages directly to the local node of the thread requesting it, completely bypassing the massive QPI (QuickPath Interconnect) bottleneck between the two physical CPUs. Without NUMA awareness, dual-socket systems like mine are severely handicapped for CPU inference. The cross-socket memory access destroys memory bandwidth, which unfortunately forces me to fall back to llama-server instead of using koboldcpp, despite vastly preferring Kobold's UI and its handling of mismatched GPUs