b1428 OOM error on 3x P40 setup

[quarterturn]

Expected Behavior

llama.cpp able to generate a reply without OOM

Current Behavior

./server -m ./models/llama-2-70b-chat/llama-2-70b-chat-q6k.gguf -ngl 83 --rope-freq-base 26000 -c 8192

llama.cpp works on the prompt and eventually throws a CUDA OOM error

Environment and Context

$ lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  56
  On-line CPU(s) list:   0-55
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
    CPU family:          6
    Model:               79
    Thread(s) per core:  2
    Core(s) per socket:  14
    Socket(s):           2
    Stepping:            1
    CPU(s) scaling MHz:  40%
    CPU max MHz:         3300.0000
    CPU min MHz:         1200.0000
    BogoMIPS:            4788.59
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall n
                         x pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64
                         monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xs
                         ave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tp
                         r_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap
                          intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d
Virtualization features:
  Virtualization:        VT-x
Caches (sum of all):
  L1d:                   896 KiB (28 instances)
  L1i:                   896 KiB (28 instances)
  L2:                    7 MiB (28 instances)
  L3:                    70 MiB (2 instances)
NUMA:
  NUMA node(s):          2
  NUMA node0 CPU(s):     0-13,28-41
  NUMA node1 CPU(s):     14-27,42-55
Vulnerabilities:
  Gather data sampling:  Not affected
  Itlb multihit:         KVM: Mitigation: VMX disabled
  L1tf:                  Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
  Mds:                   Mitigation; Clear CPU buffers; SMT vulnerable
  Meltdown:              Mitigation; PTI
  Mmio stale data:       Mitigation; Clear CPU buffers; SMT vulnerable
  Retbleed:              Not affected
  Spec rstack overflow:  Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Mitigation; Clear CPU buffers; SMT vulnerable

$ nvidia-smi
Wed Oct 25 14:31:05 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla P40           On   | 00000000:03:00.0 Off |                  Off |
| N/A   35C    P0    49W / 250W |  22418MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P40           On   | 00000000:04:00.0 Off |                  Off |
| N/A   37C    P0    52W / 250W |  18816MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla P40           On   | 00000000:A1:00.0 Off |                  Off |
| N/A   38C    P0    51W / 250W |  18816MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2596      C   ./server                        22408MiB |
|    1   N/A  N/A      2596      C   ./server                        18806MiB |
|    2   N/A  N/A      2596      C   ./server                        18806MiB |
+-----------------------------------------------------------------------------+

$ cat /etc/debian_version
12.2

* Operating System, e.g. for Linux:
$ uname -a
Linux t7910 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux

Reverting to b1407, the problem is not present

[quarterturn]

Still present at today's pull:

$ ./server -m ./models/llama-2-70b-chat/llama-2-70b-chat-q6k.gguf -ngl 83 --rope-freq-base 26000 -c 8192 -t 48
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 3 CUDA devices:
  Device 0: Tesla P40, compute capability 6.1
  Device 1: Tesla P40, compute capability 6.1
  Device 2: Tesla P40, compute capability 6.1
{"timestamp":1698416135,"level":"INFO","function":"main","line":2212,"message":"build info","build":1430,"commit":"2f9ec7e"}
{"timestamp":1698416135,"level":"INFO","function":"main","line":2215,"message":"system info","n_threads":48,"n_threads_batch":-1,"total_threads":56,"system_info":"AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | "}
llama_model_loader: loaded meta data with 16 key-value pairs and 723 tensors from ./models/llama-2-70b-chat/llama-2-70b-chat-q6k.gguf (version GGUF V2 (latest))
llama_model_loader: - tensor    0:                token_embd.weight q6_K     [  8192, 32000,     1,     1 ]
llama_model_loader: - tensor    1:               output_norm.weight f32      [  8192,     1,     1,     1 ]
...
llama_model_loader: - tensor  722:           blk.79.ffn_norm.weight f32      [  8192,     1,     1,     1 ]
llama_model_loader: - kv   0:                       general.architecture str
llama_model_loader: - kv   1:                               general.name str
llama_model_loader: - kv   2:                       llama.context_length u32
llama_model_loader: - kv   3:                     llama.embedding_length u32
llama_model_loader: - kv   4:                          llama.block_count u32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32
llama_model_loader: - kv   7:                 llama.attention.head_count u32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32
llama_model_loader: - kv  10:                          general.file_type u32
llama_model_loader: - kv  11:                       tokenizer.ggml.model str
llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr
llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr
llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr
llama_model_loader: - kv  15:               general.quantization_version u32
llama_model_loader: - type  f32:  161 tensors
llama_model_loader: - type q6_K:  562 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format           = GGUF V2 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 8192
llm_load_print_meta: n_head           = 64
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_layer          = 80
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_gqa            = 8
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 28672
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: model type       = 70B
llm_load_print_meta: model ftype      = mostly Q6_K
llm_load_print_meta: model params     = 68.98 B
llm_load_print_meta: model size       = 52.70 GiB (6.56 BPW)
llm_load_print_meta: general.name   = LLaMA v2
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: LF token  = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.24 MB
llm_load_tensors: using CUDA for GPU acceleration
ggml_cuda_set_main_device: using device 0 (Tesla P40) as main device
llm_load_tensors: mem required  =  205.32 MB
llm_load_tensors: offloading 80 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 83/83 layers to GPU
llm_load_tensors: VRAM used: 53760.11 MB
....................................................................................................
llama_new_context_with_model: n_ctx      = 8192
llama_new_context_with_model: freq_base  = 26000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: offloading v cache to GPU
llama_kv_cache_init: offloading k cache to GPU
llama_kv_cache_init: VRAM kv self = 2560.00 MB
llama_new_context_with_model: kv self size  = 2560.00 MB
llama_new_context_with_model: compute buffer total size = 1094.13 MB
llama_new_context_with_model: VRAM scratch buffer: 1088.00 MB
llama_new_context_with_model: total VRAM used: 57408.11 MB (model: 53760.11 MB, context: 3648.00 MB)
Available slots:
 -> Slot 0 - max context: 8192

llama server listening at http://127.0.0.1:8080

{"timestamp":1698416308,"level":"INFO","function":"main","line":2492,"message":"HTTP server listening","hostname":"127.0.0.1","port":8080}
all slots are idle and system prompt is empty, clear the KV cache
{"timestamp":1698416824,"level":"INFO","function":"log_server_request","line":2156,"message":"request","remote_addr":"127.0.0.1","remote_port":48088,"status":200,"method":"GET","path":"/","params":{}}
{"timestamp":1698416824,"level":"INFO","function":"log_server_request","line":2156,"message":"request","remote_addr":"127.0.0.1","remote_port":48100,"status":200,"method":"GET","path":"/","params":{}}
slot 0 is processing [task id: 0]
slot 0 : kv cache rm - [0, end)

CUDA error 2 at ggml-cuda.cu:5705: out of memory
current device: 0

[quarterturn]

B1412 is the newest commit which works for me.

[quarterturn]

on commit 207b519 (HEAD -> master, tag: b1446, origin/master, origin/HEAD) it no longer OOMs but performance is very poor

{"timestamp":1698755693,"level":"INFO","function":"main","line":2492,"message":"HTTP server listening","hostname":"127.0.0.1","port":8080}
all slots are idle and system prompt is empty, clear the KV cache
{"timestamp":1698755703,"level":"INFO","function":"log_server_request","line":2156,"message":"request","remote_addr":"127.0.0.1","remote_port":42772,"status":200,"method":"GET","path":"/","params":{}}
slot 0 is processing [task id: 0]
slot 0 : kv cache rm - [0, end)

print_timings: prompt eval time =   94100.73 ms /  2717 tokens (   34.63 ms per token,    28.87 tokens per second)
print_timings:        eval time =  216553.24 ms /   227 runs   (  953.98 ms per token,     1.05 tokens per second)
print_timings:       total time =  310653.97 ms
slot 0 released (2945 tokens in cache)
{"timestamp":1698756087,"level":"INFO","f

[quarterturn]

I switched to origin/cude-multi-gpu as a test, and find that essentially, I get the same speed whether I offload 83 or zero layers.
Here's what CPU-only gets me:

llm_load_tensors: using CUDA for GPU acceleration
ggml_cuda_set_main_device: using device 0 (Tesla P40) as main device
llm_load_tensors: mem required  = 53965.43 MB
llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/83 layers to GPU
llm_load_tensors: VRAM used: 0.00 MB
....................................................................................................
llama_new_context_with_model: n_ctx      = 8192
llama_new_context_with_model: freq_base  = 26000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: kv self size  = 2560.00 MB
llama_new_context_with_model: compute buffer total size = 1094.13 MB
llama_new_context_with_model: VRAM scratch buffer: 1088.00 MB
llama_new_context_with_model: total VRAM used: 1088.00 MB (model: 0.00 MB, context: 1088.00 MB)
Available slots:
 -> Slot 0 - max context: 8192

llama server listening at http://127.0.0.1:8080

{"timestamp":1698772696,"level":"INFO","function":"main","line":2492,"message":"HTTP server listening","hostname":"127.0.0.1","port":8080}
all slots are idle and system prompt is empty, clear the KV cache
{"timestamp":1698772716,"level":"INFO","function":"log_server_request","line":2156,"message":"request","remote_addr":"127.0.0.1","remote_port":36886,"status":200,"method":"GET","path":"/","params":{}}
slot 0 is processing [task id: 0]
slot 0 : kv cache rm - [0, end)

print_timings: prompt eval time =  262864.64 ms /  4453 tokens (   59.03 ms per token,    16.94 tokens per second)
print_timings:        eval time =  351388.25 ms /   250 runs   ( 1405.55 ms per token,     0.71 tokens per second)
print_timings:       total time =  614252.89 ms
slot 0 released (4704 tokens in cache)
{"timestamp":1698773343,"level":"INFO","function":"log_server_request","line":2156,"message":"request","remote_addr":"127.0.0.1","remote_port":39564,"status":200,"method":"POST","path":"/completion","params":{}}
slot 0 released (4704 tokens in cache)

Is "LLAMA_CUBLAS=1" no longer correct for cublas? It 'uses' the GPUs, but performance suggests otherwise.

[quarterturn]

current 'cuda-cublas-opts' branch seems to fix the OOM and poor performance issues.

[quarterturn]

I reverted to:

commit 2756c4fbffab097736d5116007872d86456a544a (HEAD -> master, tag: b1470, origin/master, origin/HEAD)
Author: Georgi Gerganov <[email protected]>
Date:   Thu Nov 2 11:20:21 2023 +0200

using "-ts 2,3,3" I can avoid the OOM, but processing is CPU-slow, like what you'd get if you gave it a single thread.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.