Per GPU fit margin

[ikawrakow]

This PR adds the ability to set the fit margin per GPU when auto-fitting the model.

Usage:

some_llama_command other_params --fit [--fit-margin M_global] --gpu-fit-margin GPU1,M1,...

• If --fit-margin is present, it will set the fit margin for all GPUs except those listed after -gpu-fit-margin. If missing, a default fit margin of 1 GiB will be used
• The -gpu-fit-margin argument is optional. One can also use the short form -gfm
• If present, it must be followed by a comma-separated list of GPU index, margin in MiB
• One can have GPU1,M1,GPU2,M2,GPU3,M3 etc. in the same argument. But one can also use multiple -gpu-fit-margin arguments (-gfm GPU1,M1 -gfm GPU2,M2,GPU3,M3)
• If one makes a mistake and assigns margins to the same GPU multiple times, the last assignment wins (e.g., for -gfm 0,2048 -gfm 0,1536 a margin of 1536 MiB will get used for GPU 0).

The feature was requested by @Panchovix

Clearly, my description was confusing. So, here is an usage example

./bin/llama-server $server_args --fit --fit-margin 1024 --gpu-fit-margin 0,3072,4,1536,7,2048

This will result in

• CUDA0 using a margin of 3072 MiB
• CUDA4 using a margin of 1536 MiB
• CUDA7 using a margin of 2048 MiB (if you are lucky enough to have 8 or ore GPUs)
• All other GPUs using a margin of 1024 MiB

[Panchovix]

Many thanks, it works with i.e.:

./llama-server -m '/run/media/pancho/MyDrive/models_llm_2tb/Kimi-K2.6-IQ3_M-00001-of-00010.gguf' -c 32768 --no-mmap -mg 0 -ub 2560 -b 2560 --fit -mla 1 --gpu-fit-margin CUDA0,6144

Allocating 162.76 GiB of pinned host memory, this may take a while.
Using pinned host memory improves PP performance by a significant margin.
But if it takes too long for your model and amount of patience, kill the process and run using

GGML_CUDA_NO_PINNED=1 your_command_goes_here
    done allocating 162.76 GiB in 27535.4 ms

llm_load_tensors: offloading 61 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 62/62 layers to GPU
llm_load_tensors:  CUDA_Host buffer size = 166670.00 MiB
llm_load_tensors:      CUDA0 buffer size = 20990.42 MiB
llm_load_tensors:      CUDA1 buffer size = 28991.58 MiB
llm_load_tensors:      CUDA2 buffer size = 27647.58 MiB
llm_load_tensors:      CUDA3 buffer size = 26639.58 MiB
llm_load_tensors:      CUDA4 buffer size = 20024.27 MiB
llm_load_tensors:      CUDA5 buffer size = 20696.27 MiB
llm_load_tensors:      CUDA6 buffer size = 13744.96 MiB
llm_load_tensors:      CUDA7 buffer size = 41550.20 MiB
llm_load_tensors:      CUDA8 buffer size = 42582.57 MiB

But then if I do either:

./llama-server -m '/run/media/pancho/MyDrive/models_llm_2tb/Kimi-K2.6-IQ3_M-00001-of-00010.gguf' -c 32768 --no-mmap -mg 0 -ub 2560 -b 2560 --fit -mla 1 --gpu-fit-margin CUDA0,6144 --gpu-fit-margin CUDA6,2048

or

./llama-server -m '/run/media/pancho/MyDrive/models_llm_2tb/Kimi-K2.6-IQ3_M-00001-of-00010.gguf' -c 32768 --no-mmap -mg 0 -ub 2560 -b 2560 --fit -mla 1 --gpu-fit-margin CUDA0,6144,CUDA1,1024,CUDA2,1024,CUDA3,1024,CUDA4,1024,CUDA5,1024,CUDA6,2048,CUDA7,1024,CUDA8,102

It doesn't seem to work. My idea with the latter 2 is to set a margin of 6GB for CUDA 0 and 2GB for CUDA 6.

Allocating 155.96 GiB of pinned host memory, this may take a while.
Using pinned host memory improves PP performance by a significant margin.
But if it takes too long for your model and amount of patience, kill the process and run using

GGML_CUDA_NO_PINNED=1 your_command_goes_here
    done allocating 155.96 GiB in 27133.6 ms

llm_load_tensors: offloading 61 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 62/62 layers to GPU
llm_load_tensors:  CUDA_Host buffer size = 159698.00 MiB
llm_load_tensors:      CUDA0 buffer size = 28120.08 MiB
llm_load_tensors:      CUDA1 buffer size = 28991.58 MiB
llm_load_tensors:      CUDA2 buffer size = 27489.93 MiB
llm_load_tensors:      CUDA3 buffer size = 26639.58 MiB
llm_load_tensors:      CUDA4 buffer size = 20024.27 MiB
llm_load_tensors:      CUDA5 buffer size = 20696.27 MiB
llm_load_tensors:      CUDA6 buffer size = 13744.96 MiB
llm_load_tensors:      CUDA7 buffer size = 41550.20 MiB
llm_load_tensors:      CUDA8 buffer size = 42582.57 MiB

(With the first one I get OOM on CUDA 6 after initial load, while on the 2 latter ones I get OOM on CUDA 0)

I also tried: GPU0,6144,GPU6,2048 and it's variants but I get the same behavior.

[ikawrakow]

It is not CUDA0,2048 or GPU0,2048, but just 0,2048.

[Panchovix]

Ohh I see, that worked perfectly now!

[magikRUKKOLA]

Super-useful feature for the multi-GPU setup!

Here is what I am using for the graph and Kimi-K2.6

    --fit \
    --fit-margin $((11 * 512)) \
    --gpu-fit-margin 12,$((12 * 512)) \

nvidia-smi | grep -Eo '[0-9]+MiB.*[0-9]+MiB'
23614MiB /  24576MiB
23956MiB /  24576MiB
23966MiB /  24576MiB
23930MiB /  24576MiB
23958MiB /  24576MiB
23640MiB /  24576MiB
23934MiB /  24576MiB
23940MiB /  24576MiB
23560MiB /  24576MiB
23978MiB /  24576MiB
23710MiB /  24576MiB
24018MiB /  24576MiB
23436MiB /  24576MiB