Eval bug: Running llama-server only possible with single AMD GPU, running multiple always causes Segmentation fault regardless of model size

[Zubkover]

Name and Version

Core tl;dr of the issue:

• Having single GPU = 20b 4bit OSS loads and runs with no issues
• Running 2/3/4 = no model suddenly is able to be be loaded and all attempts regardless of llama-server params end up in Segmentation fault (core dumped) error

Side issues/questions:

• Do you explicitly need to have all GPUs installed at build time? I noticed that having single working GPU (on which I built initial llama.cpp on) and then installing 3 more causes llama-server to auto-crash with immediate effect and Killed console output (due to ROCm failing with

ERROR Received I2C_NAK_7B0ADDR_NOACK
ERROR WriteI2CData() - I2C error occured
Failed to read EEPROM table header

Honestly speaking, don't know what I'm doing wrong and I'm 3 days deep into trying to understand based on docs and tutorials if I'm missing something in build/llama-server params 🙈

What I tried:

• played around with llama-server params, including of/off for no-mmap and ngl
• played around with build commands, including solution from Eval bug: usr/lib/gcc/x86_64-pc-linux-gnu/14/include/g++-v14/bits/random.tcc:2668: void std::discrete_distribution<_IntType>::param_type::_M_initialize() [with _IntType = int]: Assertion '__sum > 0' failed. #15551
• updated motherboard BIOS from P1.10 to P1.50
• switched BIOSes Radeon VII <> Radeon Pro VII
• reinstalled Ubuntu 4 times and repeated setup
• tried tensorfiles from rocblas 6.4.4 and 7.1.0-2 (both while running ROCm 7.1.1 as dedicated ROCm 5.6 cannot be installed on Ubuntu 24 apparently)

Operating systems

Linux

GGML backends

BLAS

Hardware

• Ubuntu 24.04.3 LTS (fresh and clean install, only for llama.cpp purposes)
• AsRock H510 PRO BTC+ (@ P1.50 BIOS)
• Celeron G5905
• 4GB RAM
• 4x Radeon Pro VII (gfx906)

Models

No response

Problem description & steps to reproduce

1. Install more than 1 AMD GPU
2. Run any model, even one that would fit within 1 GPU's memory
3. llama-server ends with Segmentation fault (core dumped)

First Bad Commit

No response

Relevant log output

llama-server -m /home/llm/llama.cpp/models/gpt-oss-20b-Q8_0.gguf --host 0.0.0.0 --port 8080

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 ROCm devices:
  Device 0: AMD Radeon (TM) Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
  Device 1: AMD Radeon (TM) Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
  Device 2: AMD Radeon (TM) Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
  Device 3: AMD Radeon (TM) Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
main: setting n_parallel = 4 and kv_unified = true (add -kvu to disable this)
build: 7179 (4abef75f2) with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu
system info: n_threads = 2, n_threads_batch = 2, total_threads = 2

system_info: n_threads = 2 (n_threads_batch = 2) / 2 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

init: using 6 threads for HTTP server
start: binding port with default address family
main: loading model
srv    load_model: loading model '/home/llm/llama.cpp/models/gpt-oss-20b-Q8_0.gguf'
llama_model_load_from_file_impl: using device ROCm0 (AMD Radeon (TM) Pro VII) (0000:03:00.0) - 16348 MiB free
llama_model_load_from_file_impl: using device ROCm1 (AMD Radeon (TM) Pro VII) (0000:06:00.0) - 16348 MiB free
llama_model_load_from_file_impl: using device ROCm2 (AMD Radeon (TM) Pro VII) (0000:09:00.0) - 16348 MiB free
llama_model_load_from_file_impl: using device ROCm3 (AMD Radeon (TM) Pro VII) (0000:0c:00.0) - 16348 MiB free
llama_model_loader: loaded meta data with 37 key-value pairs and 459 tensors from /home/llm/llama.cpp/models/gpt-oss-20b-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gpt-oss
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Gpt-Oss-20B
llama_model_loader: - kv   3:                           general.basename str              = Gpt-Oss-20B
llama_model_loader: - kv   4:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   5:                         general.size_label str              = 20B
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv   8:                               general.tags arr[str,2]       = ["vllm", "text-generation"]
llama_model_loader: - kv   9:                        gpt-oss.block_count u32              = 24
llama_model_loader: - kv  10:                     gpt-oss.context_length u32              = 131072
llama_model_loader: - kv  11:                   gpt-oss.embedding_length u32              = 2880
llama_model_loader: - kv  12:                gpt-oss.feed_forward_length u32              = 2880
llama_model_loader: - kv  13:               gpt-oss.attention.head_count u32              = 64
llama_model_loader: - kv  14:            gpt-oss.attention.head_count_kv u32              = 8
llama_model_loader: - kv  15:                     gpt-oss.rope.freq_base f32              = 150000.000000
llama_model_loader: - kv  16:   gpt-oss.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  17:                       gpt-oss.expert_count u32              = 32
llama_model_loader: - kv  18:                  gpt-oss.expert_used_count u32              = 4
llama_model_loader: - kv  19:               gpt-oss.attention.key_length u32              = 64
llama_model_loader: - kv  20:             gpt-oss.attention.value_length u32              = 64
llama_model_loader: - kv  21:           gpt-oss.attention.sliding_window u32              = 128
llama_model_loader: - kv  22:         gpt-oss.expert_feed_forward_length u32              = 2880
llama_model_loader: - kv  23:                  gpt-oss.rope.scaling.type str              = yarn
llama_model_loader: - kv  24:                gpt-oss.rope.scaling.factor f32              = 32.000000
llama_model_loader: - kv  25: gpt-oss.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  26:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  27:                         tokenizer.ggml.pre str              = gpt-4o
llama_model_loader: - kv  28:                      tokenizer.ggml.tokens arr[str,201088]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  29:                  tokenizer.ggml.token_type arr[i32,201088]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  30:                      tokenizer.ggml.merges arr[str,446189]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  31:                tokenizer.ggml.bos_token_id u32              = 199998
llama_model_loader: - kv  32:                tokenizer.ggml.eos_token_id u32              = 200002
llama_model_loader: - kv  33:            tokenizer.ggml.padding_token_id u32              = 200017
llama_model_loader: - kv  34:                    tokenizer.chat_template str              = {# Chat template fixes by Unsloth #}\n...
llama_model_loader: - kv  35:               general.quantization_version u32              = 2
llama_model_loader: - kv  36:                          general.file_type u32              = 7
llama_model_loader: - type  f32:  289 tensors
llama_model_loader: - type q8_0:   98 tensors
llama_model_loader: - type mxfp4:   72 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q8_0
print_info: file size   = 11.27 GiB (4.63 BPW) 
load: printing all EOG tokens:
load:   - 199999 ('<|endoftext|>')
load:   - 200002 ('<|return|>')
load:   - 200007 ('<|end|>')
load:   - 200012 ('<|call|>')
load: special_eog_ids contains both '<|return|>' and '<|call|>' tokens, removing '<|end|>' token from EOG list
load: special tokens cache size = 21
load: token to piece cache size = 1.3332 MB
print_info: arch             = gpt-oss
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 2880
print_info: n_embd_inp       = 2880
print_info: n_layer          = 24
print_info: n_head           = 64
print_info: n_head_kv        = 8
print_info: n_rot            = 64
print_info: n_swa            = 128
print_info: is_swa_any       = 1
print_info: n_embd_head_k    = 64
print_info: n_embd_head_v    = 64
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 2880
print_info: n_expert         = 32
print_info: n_expert_used    = 4
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = yarn
print_info: freq_base_train  = 150000.0
print_info: freq_scale_train = 0.03125
print_info: n_ctx_orig_yarn  = 4096
print_info: rope_finetuned   = unknown
print_info: model type       = 20B
print_info: model params     = 20.91 B
print_info: general.name     = Gpt-Oss-20B
print_info: n_ff_exp         = 2880
print_info: vocab type       = BPE
print_info: n_vocab          = 201088
print_info: n_merges         = 446189
print_info: BOS token        = 199998 '<|startoftext|>'
print_info: EOS token        = 200002 '<|return|>'
print_info: EOT token        = 199999 '<|endoftext|>'
print_info: PAD token        = 200017 '<|reserved_200017|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 199999 '<|endoftext|>'
print_info: EOG token        = 200002 '<|return|>'
print_info: EOG token        = 200012 '<|call|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 24 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 25/25 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   586.82 MiB
load_tensors:        ROCm0 model buffer size =  3022.41 MiB
load_tensors:        ROCm1 model buffer size =  2590.64 MiB
load_tensors:        ROCm2 model buffer size =  2590.64 MiB
load_tensors:        ROCm3 model buffer size =  2745.70 MiB
........................Segmentation fault (core dumped)

[pwilkin]

Please recompile with DEBUG mode and add a stracktrace for the segmentation fault.

[Zubkover]

cmake -B build -DCMAKE_BUILD_TYPE=Debug I assume? 👀

If so then output is as follows:

llama-server -m /home/llm/llama.cpp/models/gpt-oss-20b-Q8_0.gguf --host 0.0.0.0 --port 8080
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 ROCm devices:
  Device 0: AMD Radeon (TM) Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
  Device 1: AMD Radeon (TM) Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
  Device 2: AMD Radeon (TM) Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
  Device 3: AMD Radeon (TM) Pro VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
register_backend: registered backend ROCm (4 devices)
register_device: registered device ROCm0 (AMD Radeon (TM) Pro VII)
register_device: registered device ROCm1 (AMD Radeon (TM) Pro VII)
register_device: registered device ROCm2 (AMD Radeon (TM) Pro VII)
register_device: registered device ROCm3 (AMD Radeon (TM) Pro VII)
register_backend: registered backend CPU (1 devices)
register_device: registered device CPU (Intel(R) Celeron(R) G5905 CPU @ 3.50GHz)
load_backend: failed to find ggml_backend_init in /home/llm/llama.cpp/build/bin/libggml-hip.so
load_backend: failed to find ggml_backend_init in /home/llm/llama.cpp/build/bin/libggml-cpu.so
main: setting n_parallel = 4 and kv_unified = true (add -kvu to disable this)
build: 7179 (4abef75f2) with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu (debug)
system info: n_threads = 2, n_threads_batch = 2, total_threads = 2

system_info: n_threads = 2 (n_threads_batch = 2) / 2 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

init: using 6 threads for HTTP server
start: binding port with default address family
main: loading model
srv    load_model: loading model '/home/llm/llama.cpp/models/gpt-oss-20b-Q8_0.gguf'
llama_model_load_from_file_impl: using device ROCm0 (AMD Radeon (TM) Pro VII) (0000:03:00.0) - 16348 MiB free
llama_model_load_from_file_impl: using device ROCm1 (AMD Radeon (TM) Pro VII) (0000:06:00.0) - 16348 MiB free
llama_model_load_from_file_impl: using device ROCm2 (AMD Radeon (TM) Pro VII) (0000:09:00.0) - 16348 MiB free
llama_model_load_from_file_impl: using device ROCm3 (AMD Radeon (TM) Pro VII) (0000:0c:00.0) - 16348 MiB free
llama_model_loader: loaded meta data with 37 key-value pairs and 459 tensors from /home/llm/llama.cpp/models/gpt-oss-20b-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gpt-oss
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Gpt-Oss-20B
llama_model_loader: - kv   3:                           general.basename str              = Gpt-Oss-20B
llama_model_loader: - kv   4:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   5:                         general.size_label str              = 20B
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv   8:                               general.tags arr[str,2]       = ["vllm", "text-generation"]
llama_model_loader: - kv   9:                        gpt-oss.block_count u32              = 24
llama_model_loader: - kv  10:                     gpt-oss.context_length u32              = 131072
llama_model_loader: - kv  11:                   gpt-oss.embedding_length u32              = 2880
llama_model_loader: - kv  12:                gpt-oss.feed_forward_length u32              = 2880
llama_model_loader: - kv  13:               gpt-oss.attention.head_count u32              = 64
llama_model_loader: - kv  14:            gpt-oss.attention.head_count_kv u32              = 8
llama_model_loader: - kv  15:                     gpt-oss.rope.freq_base f32              = 150000.000000
llama_model_loader: - kv  16:   gpt-oss.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  17:                       gpt-oss.expert_count u32              = 32
llama_model_loader: - kv  18:                  gpt-oss.expert_used_count u32              = 4
llama_model_loader: - kv  19:               gpt-oss.attention.key_length u32              = 64
llama_model_loader: - kv  20:             gpt-oss.attention.value_length u32              = 64
llama_model_loader: - kv  21:           gpt-oss.attention.sliding_window u32              = 128
llama_model_loader: - kv  22:         gpt-oss.expert_feed_forward_length u32              = 2880
llama_model_loader: - kv  23:                  gpt-oss.rope.scaling.type str              = yarn
llama_model_loader: - kv  24:                gpt-oss.rope.scaling.factor f32              = 32.000000
llama_model_loader: - kv  25: gpt-oss.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  26:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  27:                         tokenizer.ggml.pre str              = gpt-4o
llama_model_loader: - kv  28:                      tokenizer.ggml.tokens arr[str,201088]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  29:                  tokenizer.ggml.token_type arr[i32,201088]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  30:                      tokenizer.ggml.merges arr[str,446189]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  31:                tokenizer.ggml.bos_token_id u32              = 199998
llama_model_loader: - kv  32:                tokenizer.ggml.eos_token_id u32              = 200002
llama_model_loader: - kv  33:            tokenizer.ggml.padding_token_id u32              = 200017
llama_model_loader: - kv  34:                    tokenizer.chat_template str              = {# Chat template fixes by Unsloth #}\n...
llama_model_loader: - kv  35:               general.quantization_version u32              = 2
llama_model_loader: - kv  36:                          general.file_type u32              = 7
llama_model_loader: - type  f32:  289 tensors
llama_model_loader: - type q8_0:   98 tensors
llama_model_loader: - type mxfp4:   72 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q8_0
print_info: file size   = 11.27 GiB (4.63 BPW) 
load: printing all EOG tokens:
load:   - 199999 ('<|endoftext|>')
load:   - 200002 ('<|return|>')
load:   - 200007 ('<|end|>')
load:   - 200012 ('<|call|>')
load: special_eog_ids contains both '<|return|>' and '<|call|>' tokens, removing '<|end|>' token from EOG list
load: special tokens cache size = 21
load: token to piece cache size = 1.3332 MB
print_info: arch             = gpt-oss
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 2880
print_info: n_embd_inp       = 2880
print_info: n_layer          = 24
print_info: n_head           = 64
print_info: n_head_kv        = 8
print_info: n_rot            = 64
print_info: n_swa            = 128
print_info: is_swa_any       = 1
print_info: n_embd_head_k    = 64
print_info: n_embd_head_v    = 64
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 2880
print_info: n_expert         = 32
print_info: n_expert_used    = 4
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = yarn
print_info: freq_base_train  = 150000.0
print_info: freq_scale_train = 0.03125
print_info: n_ctx_orig_yarn  = 4096
print_info: rope_finetuned   = unknown
print_info: model type       = 20B
print_info: model params     = 20.91 B
print_info: general.name     = Gpt-Oss-20B
print_info: n_ff_exp         = 2880
print_info: vocab type       = BPE
print_info: n_vocab          = 201088
print_info: n_merges         = 446189
print_info: BOS token        = 199998 '<|startoftext|>'
print_info: EOS token        = 200002 '<|return|>'
print_info: EOT token        = 199999 '<|endoftext|>'
print_info: PAD token        = 200017 '<|reserved_200017|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 199999 '<|endoftext|>'
print_info: EOG token        = 200002 '<|return|>'
print_info: EOG token        = 200012 '<|call|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 24 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 25/25 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   586.82 MiB
load_tensors:        ROCm0 model buffer size =  3022.41 MiB
load_tensors:        ROCm1 model buffer size =  2590.64 MiB
load_tensors:        ROCm2 model buffer size =  2590.64 MiB
load_tensors:        ROCm3 model buffer size =  2745.70 MiB
........................Segmentation fault (core dumped)

and dump, but it still has??despite being from debug build?

Core was generated by `llama-server -m /home/llm/llama.cpp/models/gpt-oss-20b-Q8_0.gguf --host 0.0.0.0'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000073c4cb44e424 in ?? ()
[Current thread is 1 (LWP 10604)]
(gdb) bt
#0  0x000073c4cb44e424 in ?? ()
#1  0x00007ffe5d470690 in ?? ()
#2  0x000073c5042addae in ?? ()
#3  0x00006173b2390600 in ?? ()
#4  0x28be8ff8ab9e7800 in ?? ()
#5  0x00007ffe5d4707e0 in ?? ()
#6  0x00006173abd1ec10 in ?? ()
#7  0x00007ffe5d4707e0 in ?? ()
#8  0x0000000000000000 in ?? ()

[JohannesGaessler]

I am unable to reproduce the issue using a MI50 + an RX 6800.

[Maks4d]

I have 2 mi50 flashed with bios from Radeon Pro V420 32 GB, and running Ubuntu 24.04.3 LTS. I am using standard driver for gpu. Did not install any dkms from amdgpu-install. I am using rocm sdk from TheRock
python -m pip install --index-url https://rocm.nightlies.amd.com/v2-staging/gfx90X-dcgpu/ "rocm[libraries,devel]"
and then i run rocm-sdk init
add user to necessary groups:
sudo usermod -a -G render,video $USER
reboot
i am installing variables for building like this
export ROCM_PATH=$(rocm-sdk path --root)
export HIP_PLATFORM=amd
export HIP_PATH=$ROCM_PATH
export HIP_CLANG_PATH=$ROCM_PATH/llvm/bin
export HIP_INCLUDE_PATH=$ROCM_PATH/include
export HIP_LIB_PATH=$ROCM_PATH/lib
export HIP_DEVICE_LIB_PATH=$ROCM_PATH/lib/llvm/amdgcn/bitcode
export PATH="$ROCM_PATH/bin:$HIP_CLANG_PATH:$PATH"
export LD_LIBRARY_PATH="$HIP_LIB_PATH:$ROCM_PATH/lib:$ROCM_PATH/lib64:$ROCM_PATH/llvm/lib:${LD_LIBRARY_PATH:-}"
export LIBRARY_PATH="$HIP_LIB_PATH:$ROCM_PATH/lib:$ROCM_PATH/lib64:${LIBRARY_PATH:-}"
export CPATH="$HIP_INCLUDE_PATH:${CPATH:-}"
export PKG_CONFIG_PATH="$ROCM_PATH/lib/pkgconfig:${PKG_CONFIG_PATH:-}"

this is log of koboldcpp, fork of llamacpp. for me, all working fine:
(venv) maksim@maksim-ProLiant-DL380-Gen9:/mnt/Data5TB/koboldcpp$ source ./venv/bin/activate
(venv) maksim@maksim-ProLiant-DL380-Gen9:/mnt/Data5TB/koboldcpp$ python koboldcpp.py

---

Welcome to KoboldCpp - Version 1.102.3
For command line arguments, please refer to --help

---

Detected AMD GPU VRAM from rocminfo: [('AMD Radeon Graphics', '32752'), ('AMD Radeon Graphics', '32752')] MB
Auto Selected HIP Backend (flag=0)

Loading Chat Completions Adapter: /mnt/Data5TB/koboldcpp/kcpp_adapters/AutoGuess.json
Chat Completions Adapter Loaded
System: Linux #36~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Oct 15 15:45:17 UTC 2 x86_64 x86_64
Detected Available GPU Memory: 32752 MB
Detected Available RAM: 247497 MB
Initializing dynamic library: koboldcpp_hipblas.so

Namespace(model=[], model_param='/mnt/Data5TB/Soft/Models/LLM/Deviant-EXPERIMENTAL-V2-70B.i1-Q4_K_M.gguf', port=5001, port_param=5001, host='', launch=True, config=None, threads=48, usecuda=['normal', 'nommq'], usevulkan=None, useclblast=None, usecpu=False, contextsize=32768, gpulayers=999, tensor_split=[1.0, 1.0], version=False, analyze='', maingpu=-1, batchsize=512, blasthreads=None, lora=None, loramult=1.0, noshift=False, nofastforward=False, useswa=False, ropeconfig=[0.0, 10000.0], overridenativecontext=0, usemmap=False, usemlock=False, noavx2=False, failsafe=False, debugmode=0, onready='', benchmark=None, prompt='', cli=False, genlimit=0, multiuser=1, multiplayer=False, websearch=False, remotetunnel=False, highpriority=False, foreground=False, preloadstory=None, savedatafile=None, quiet=False, ssl=None, nocertify=False, mmproj=None, mmprojcpu=False, visionmaxres=1024, draftmodel=None, draftamount=8, draftgpulayers=999, draftgpusplit=None, password=None, ratelimit=0, ignoremissing=False, chatcompletionsadapter='AutoGuess', jinja=False, jinja_tools=False, flashattention=True, lowvram=False, quantkv=1, forceversion=0, smartcontext=False, unpack='', exportconfig='', exporttemplate='', nomodel=False, moeexperts=-1, moecpu=0, defaultgenamt=768, nobostoken=False, enableguidance=False, maxrequestsize=32, overridekv=None, overridetensors=None, showgui=False, skiplauncher=False, singleinstance=False, hordemodelname='', hordeworkername='', hordekey='', hordemaxctx=0, hordegenlen=0, sdmodel='', sdthreads=23, sdclamped=0, sdclampedsoft=0, sdt5xxl='', sdclip1='', sdclip2='', sdphotomaker='', sdflashattention=False, sdoffloadcpu=False, sdvaecpu=False, sdclipgpu=False, sdconvdirect='off', sdvae='', sdvaeauto=False, sdquant=0, sdlora='', sdloramult=1.0, sdtiledvae=768, sdgendefaults='', whispermodel='', ttsmodel='', ttswavtokenizer='', ttsgpu=False, ttsmaxlen=4096, ttsthreads=0, embeddingsmodel='', embeddingsmaxctx=0, embeddingsgpu=False, admin=False, adminpassword='', admindir='', hordeconfig=None, sdconfig=None, noblas=False, nommap=False, sdnotile=False, testmemory=False)

Loading Text Model: /mnt/Data5TB/Soft/Models/LLM/Deviant-EXPERIMENTAL-V2-70B.i1-Q4_K_M.gguf

The reported GGUF Arch is: llama
Arch Category: 0

---

Identified as GGUF model.
Attempting to Load...

Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
CUDA MMQ: False

Initializing CUDA/HIP, please wait, the following step may take a few minutes (only for first launch)...

Applying Tensor Split...
ggml_cuda_init: found 2 ROCm devices:
Device 0: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
Device 1: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
llama_model_load_from_file_impl: using device ROCm0 (AMD Radeon Graphics) (0000:0d:00.0) - 32712 MiB free
llama_model_load_from_file_impl: using device ROCm1 (AMD Radeon Graphics) (0000:86:00.0) - 32722 MiB free
llama_model_loader: loaded meta data with 60 key-value pairs and 724 tensors from /mnt/Data5TB/Soft/Models/LLM/Deviant-EXPERIMENTAL-V2-70B.i1-Q4_K_M.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file size = 39.59 GiB (4.82 BPW)
init_tokenizer: initializing tokenizer for type 2
load: printing all EOG tokens:
load: - 128001 ('<|end_of_text|>')
load: - 128008 ('<|eom_id|>')
load: - 128009 ('<|eot_id|>')
load: special tokens cache size = 256
load: token to piece cache size = 0.7999 MB
print_info: arch = llama
print_info: vocab_only = 0
print_info: n_ctx_train = 131072
print_info: n_embd = 8192
print_info: n_embd_inp = 8192
print_info: n_layer = 80
print_info: n_head = 64
print_info: n_head_kv = 8
print_info: n_rot = 128
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 128
print_info: n_embd_head_v = 128
print_info: n_gqa = 8
print_info: n_embd_k_gqa = 1024
print_info: n_embd_v_gqa = 1024
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-05
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 28672
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 0
print_info: rope scaling = linear
print_info: freq_base_train = 500000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 131072
print_info: rope_finetuned = unknown
print_info: model type = 70B
print_info: model params = 70.55 B
print_info: general.name = Bigger Body 70b
print_info: vocab type = BPE
print_info: n_vocab = 128256
print_info: n_merges = 280147
print_info: BOS token = 128000 '<|begin_of_text|>'
print_info: EOS token = 128009 '<|eot_id|>'
print_info: EOT token = 128009 '<|eot_id|>'
print_info: EOM token = 128008 '<|eom_id|>'
print_info: PAD token = 128004 '<|finetune_right_pad_id|>'
print_info: LF token = 198 'Ċ'
print_info: EOG token = 128001 '<|end_of_text|>'
print_info: EOG token = 128008 '<|eom_id|>'
print_info: EOG token = 128009 '<|eot_id|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: relocated tensors: 1 of 803
load_tensors: offloading 80 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 81/81 layers to GPU
load_tensors: CPU model buffer size = 563.62 MiB
load_tensors: ROCm0 model buffer size = 20038.81 MiB
load_tensors: ROCm1 model buffer size = 19940.67 MiB
load_all_data: using async uploads for device ROCm0, buffer type ROCm0, backend ROCm0
..................................................load_all_data: using async uploads for device ROCm1, buffer type ROCm1, backend ROCm1
.................................................
Automatic RoPE Scaling: Using (scale:1.000, base:500000.0).
llama_context: constructing llama_context
llama_context: n_seq_max = 1
llama_context: n_ctx = 33024
llama_context: n_ctx_seq = 33024
llama_context: n_batch = 512
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = enabled
llama_context: kv_unified = true
llama_context: freq_base = 500000.0
llama_context: freq_scale = 1
llama_context: n_ctx_seq (33024) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
set_abort_callback: call
llama_context: ROCm_Host output buffer size = 0.49 MiB
llama_kv_cache: ROCm0 KV buffer size = 2809.78 MiB
llama_kv_cache: ROCm1 KV buffer size = 2672.72 MiB
llama_kv_cache: size = 5482.50 MiB ( 33024 cells, 80 layers, 1/1 seqs), K (q8_0): 2741.25 MiB, V (q8_0): 2741.25 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 3
llama_context: max_nodes = 5800
llama_context: pipeline parallelism enabled (n_copies=4)
llama_context: reserving full memory module
llama_context: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1
llama_context: ROCm0 compute buffer size = 570.29 MiB
llama_context: ROCm1 compute buffer size = 459.55 MiB
llama_context: ROCm_Host compute buffer size = 274.05 MiB
llama_context: graph nodes = 2487
llama_context: graph splits = 3
Threadpool set to 48 threads and 48 blasthreads...
attach_threadpool: call
Starting model warm up, please wait a moment...
Load Text Model OK: True
Chat completion heuristic: Llama 3.x
Embedded KoboldAI Lite loaded.
Embedded API docs loaded.
Llama.cpp UI loaded.

Active Modules: TextGeneration
Inactive Modules: ImageGeneration VoiceRecognition MultimodalVision MultimodalAudio NetworkMultiplayer ApiKeyPassword WebSearchProxy TextToSpeech VectorEmbeddings AdminControl
Enabled APIs: KoboldCppApi OpenAiApi OllamaApi
Starting Kobold API on port 5001 at http://localhost:5001/api/
Starting OpenAI Compatible API on port 5001 at http://localhost:5001/v1/
Starting llama.cpp secondary WebUI at http://localhost:5001/lcpp/

Please connect to custom endpoint at http://localhost:5001
Gtk-Message: 04:39:23.338: Not loading module "atk-bridge": The functionality is provided by GTK natively. Please try to not load it.

Input: {"n": 1, "max_context_length": 131072, "max_length": 768, "rep_pen": 1.05, "temperature": 0.75, "top_p": 0.92, "top_k": 100, "top_a": 0, "typical": 1, "tfs": 1, "rep_pen_range": 360, "rep_pen_slope": 0.7, "sampler_order": [6, 0, 1, 3, 4, 2, 5], "memory": "", "trim_stop": true, "genkey": "KCPP4693", "min_p": 0, "dynatemp_range": 0, "dynatemp_exponent": 1, "smoothing_factor": 0, "smoothing_curve": 1, "nsigma": 0, "banned_tokens": [], "render_special": false, "logprobs": false, "replace_instruct_placeholders": true, "presence_penalty": 0, "logit_bias": {}, "stop_sequence": ["{{[INPUT]}}", "{{[OUTPUT]}}"], "use_default_badwordsids": false, "bypass_eos": false, "prompt": "{{[INPUT]}}hi{{[OUTPUT]}}"}

!!! ====== !!!
(Warning! Request max_context_length=131072 exceeds allocated context size of 32768. It will be reduced to fit. Consider launching with increased --contextsize to avoid issues. This message will only show once per session.)
!!! ====== !!!

Processing Prompt (12 / 12 tokens)
Generating (12 / 768 tokens)
(EOS token triggered! ID:128009)
[04:39:34] CtxLimit:24/32768, Amt:12/768, Init:0.00s, Process:4.04s (2.97T/s), Generate:1.77s (6.78T/s), Total:5.81s
Output: Hi! How can I help you today? 😊

Input: {"n": 1, "max_context_length": 131072, "max_length": 768, "rep_pen": 1.05, "temperature": 0.75, "top_p": 0.92, "top_k": 100, "top_a": 0, "typical": 1, "tfs": 1, "rep_pen_range": 360, "rep_pen_slope": 0.7, "sampler_order": [6, 0, 1, 3, 4, 2, 5], "memory": "", "trim_stop": true, "genkey": "KCPP3146", "min_p": 0, "dynatemp_range": 0, "dynatemp_exponent": 1, "smoothing_factor": 0, "smoothing_curve": 1, "nsigma": 0, "banned_tokens": [], "render_special": false, "logprobs": false, "replace_instruct_placeholders": true, "presence_penalty": 0, "logit_bias": {}, "stop_sequence": ["{{[INPUT]}}", "{{[OUTPUT]}}"], "use_default_badwordsids": false, "bypass_eos": false, "prompt": "{{[INPUT]}}hi{{[OUTPUT]}}Hi! How can I help you today? 😊{{[INPUT]}}tell me a joke{{[OUTPUT]}}"}

Processing Prompt (14 / 14 tokens)
Generating (41 / 768 tokens)
(EOS token triggered! ID:128009)
[04:51:00] CtxLimit:78/32768, Amt:41/768, Init:0.00s, Process:2.90s (4.82T/s), Generate:4.50s (9.11T/s), Total:7.40s
Output: Why did the scarecrow win an award? Because he was outstanding in his field! 🌳😄

I hope that made you smile! Do you want to hear another one? 😊

i guess it's driver bug. a few weeks ago i used amdgpu driver. latest version doesn't work properly. missing tensile files, so i had to do something like this:
wget https://repo.radeon.com/amdgpu-install/6.4.2.1/ubuntu/noble/amdgpu-install_6.4.60402-1_all.deb

sudo apt install ./amdgpu-install_6.4.60402-1_all.deb

sudo apt update

sudo apt install amdgpu-dkms

sudo add-apt-repository --remove "https://repo.radeon.com/amdgpu/6.4.2.1/ubuntu noble"

sudo add-apt-repository --remove "https://repo.radeon.com/rocm/apt/6.4.2 noble"

sudo rm /etc/apt/sources.list.d/amdgpu.list

sudo rm /etc/apt/sources.list.d/rocm.list

sudo rm /etc/apt/sources.list.d/amdgpu-proprietary.list

wget https://repo.radeon.com/amdgpu-install/6.3.3/ubuntu/noble/amdgpu-install_6.3.60303-1_all.deb

sudo apt install ./amdgpu-install_6.3.60303-1_all.deb

sudo apt update

sudo usermod -a -G render,video $USER

sudo amdgpu-install --usecase=rocm,lrt,opencl,openclsdk,hip,hiplibsdk,dkms,mllib --rocmrelease=6.3.3 -y

Well. it worked, but no multi gpu. garbled output.
Now I am just using standard ubuntu driver with TheRock sdk, I mean, clean install of Ubuntu, no deb drivers. No proprietary drivers. Don’t using amdgpu-install. Just what is already installed.
(venv) maksim@maksim-ProLiant-DL380-Gen9:/mnt/Data5TB/koboldcpp$ pip list
Package Version

---

bitsandbytes 0.49.0.dev0
certifi 2025.11.12
charset-normalizer 3.4.4
customtkinter 5.2.2
darkdetect 0.8.0
einops 0.8.1
filelock 3.20.0
flash_attn 2.8.3
fsspec 2025.10.0
gguf 0.17.1
hf-xet 1.2.0
huggingface-hub 0.36.0
idna 3.11
Jinja2 3.1.6
MarkupSafe 3.0.3
mpmath 1.3.0
networkx 3.5
numpy 2.3.4
packaging 25.0
pillow 12.0.0
pip 24.0
protobuf 6.33.1
psutil 7.1.3
pytorch-triton-rocm 3.4.0+rocm7.11.0a20251124
PyYAML 6.0.3
regex 2025.11.3
requests 2.32.5
rocm 7.11.0a20251124
rocm-sdk-core 7.11.0a20251124
rocm-sdk-devel 7.11.0a20251124
rocm-sdk-libraries-gfx90X-dcgpu 7.11.0a20251124
safetensors 0.7.0
sentencepiece 0.2.1
setuptools 80.9.0
sympy 1.14.0
tokenizers 0.22.1
torch 2.8.0+rocm7.11.0a20251124
torchaudio 2.8.0a0+rocm7.11.0a20251124
torchvision 0.23.0a0+rocm7.11.0a20251124
tqdm 4.67.1
transformers 4.57.3
typing_extensions 4.15.0
urllib3 2.5.0

[Zubkover]

#cursedExperienceContinues

I tried to follow what @Maks4d suggested, but now I cannot even build llama.cpp
At the beginning it ran build until 97% (afair) and completely locked up my system with compile error.
After a reboot it errors out immediately :/

CMAKE_BUILD_TYPE=Release
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- GGML_SYSTEM_ARCH: x86
-- Including CPU backend
-- x86 detected
-- Adding CPU backend variant ggml-cpu: -march=native 
-- HIP and hipBLAS found
-- Including HIP backend
-- ggml version: 0.9.4
-- ggml commit:  db9783738
-- Configuring done (0.6s)
-- Generating done (0.4s)
-- Build files have been written to: /home/llm/llama.cpp/build
[  0%] Built target build_info
[  2%] Built target ggml-base
[  2%] Built target cpp-httplib
[  2%] Built target sha256
[  3%] Built target xxhash
[  3%] Built target sha1
[  4%] Built target llama-llava-cli
[  5%] Built target llama-gemma3-cli
[  6%] Built target llama-minicpmv-cli
[  6%] Built target llama-qwen2vl-cli
[ 28%] Built target ggml-hip
[ 32%] Built target ggml-cpu
[ 32%] Built target ggml
[ 32%] Built target llama-gguf-hash
[ 33%] Built target llama-gguf
[ 59%] Built target llama
[ 60%] Built target mtmd
[ 61%] Built target test-c
[ 65%] Built target common
[ 65%] Built target llama-simple
[ 66%] Built target llama-simple-chat
[ 67%] Built target llama-speculative
[ 67%] Built target llama-speculative-simple
[ 67%] Built target llama-finetune
[ 67%] Built target llama-gen-docs
[ 68%] Built target llama-logits
[ 68%] Built target llama-diffusion-cli
[ 68%] Built target llama-convert-llama2c-to-ggml
[ 69%] Built target llama-vdot
[ 71%] Built target llama-batched-bench
[ 71%] Built target llama-q8dot
[ 71%] Building CXX object tools/imatrix/CMakeFiles/llama-imatrix.dir/imatrix.cpp.o
[ 71%] Built target llama-gguf-split
[ 71%] Building CXX object tools/llama-bench/CMakeFiles/llama-bench.dir/llama-bench.cpp.o
[ 71%] Linking CXX executable ../../bin/llama-imatrix
/usr/bin/ld: warning: libhipblas.so.3, needed by ../../bin/libggml-hip.so.0.9.4, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: librocblas.so.5, needed by ../../bin/libggml-hip.so.0.9.4, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libamdhip64.so.7, needed by ../../bin/libggml-hip.so.0.9.4, not found (try using -rpath or -rpath-link)
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipGetDeviceCount@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `__hipRegisterFatBinary@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMemcpyAsync@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMemcpyPeerAsync@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipEventCreateWithFlags@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipblasGemmBatchedEx'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipHostUnregister@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `__hipRegisterFunction@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipDeviceSynchronize@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipGetErrorString@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipEventDestroy@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipSetDevice@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipHostFree@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMemset@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipDeviceDisablePeerAccess@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipOccupancyMaxActiveBlocksPerMultiprocessor@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipDeviceEnablePeerAccess@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMemAdvise@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipStreamWaitEvent@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `__hipPopCallConfiguration@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMemsetAsync@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMemcpy2DAsync@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipblasDestroy'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipDeviceCanAccessPeer@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipEventSynchronize@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipblasGemmStridedBatchedEx'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipLaunchKernel@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipFree@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipGetDevicePropertiesR0600@hip_6.0'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMallocManaged@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipStreamDestroy@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipHostMalloc@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipblasSetStream'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipblasSgemm'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipblasCreate'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipblasGemmEx'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `__hipPushCallConfiguration@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `__hipUnregisterFatBinary@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipStreamCreateWithFlags@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipHostRegister@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipEventRecord@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipGetLastError@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipGetDevice@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMemGetInfo@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMalloc@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipStreamSynchronize@hip_4.2'
collect2: error: ld returned 1 exit status
gmake[2]: *** [tools/imatrix/CMakeFiles/llama-imatrix.dir/build.make:104: bin/llama-imatrix] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:3975: tools/imatrix/CMakeFiles/llama-imatrix.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
[ 71%] Linking CXX executable ../../bin/llama-bench
/usr/bin/ld: warning: libhipblas.so.3, needed by ../../bin/libggml-hip.so.0.9.4, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: librocblas.so.5, needed by ../../bin/libggml-hip.so.0.9.4, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libamdhip64.so.7, needed by ../../bin/libggml-hip.so.0.9.4, not found (try using -rpath or -rpath-link)
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipEventRecord@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipDeviceSynchronize@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMemsetAsync@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `__hipPushCallConfiguration@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipblasCreate'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMalloc@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipOccupancyMaxActiveBlocksPerMultiprocessor@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipblasGemmEx'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipLaunchKernel@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMemcpyAsync@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMemcpy2DAsync@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipDeviceDisablePeerAccess@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `__hipPopCallConfiguration@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipGetLastError@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipblasSgemm'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipHostRegister@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipDeviceCanAccessPeer@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipGetDeviceCount@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipFree@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `__hipRegisterFatBinary@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipStreamSynchronize@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMemset@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipblasGemmBatchedEx'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipblasSetStream'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipStreamDestroy@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipblasGemmStridedBatchedEx'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMemcpyPeerAsync@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMemAdvise@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipGetDevicePropertiesR0600@hip_6.0'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `__hipRegisterFunction@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipGetDevice@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMemGetInfo@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipMallocManaged@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipblasDestroy'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipHostMalloc@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipEventDestroy@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipStreamWaitEvent@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipGetErrorString@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipHostUnregister@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipDeviceEnablePeerAccess@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipEventSynchronize@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipHostFree@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `__hipUnregisterFatBinary@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipEventCreateWithFlags@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipSetDevice@hip_4.2'
/usr/bin/ld: ../../bin/libggml-hip.so.0.9.4: undefined reference to `hipStreamCreateWithFlags@hip_4.2'
collect2: error: ld returned 1 exit status
gmake[2]: *** [tools/llama-bench/CMakeFiles/llama-bench.dir/build.make:104: bin/llama-bench] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:4007: tools/llama-bench/CMakeFiles/llama-bench.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2

[Maks4d]

@Zubkover I am successfully built and launched llamacpp with this commands:

git clone --recursive https://github.com/ggml-org/llama.cpp.git
cd ./llama.cpp/
python -m venv venv
source ./venv/bin/activate
pip install --index-url https://rocm.nightlies.amd.com/v2-staging/gfx90X-dcgpu/ --pre rocm[libraries,devel]==7.11.0a20251124 --force-reinstall
rocm-sdk init
pip install ninja cmake
export ROCM_PATH=$(rocm-sdk path --root)
export HIP_PLATFORM=amd
export HIP_PATH=$ROCM_PATH
export HIP_CLANG_PATH=$ROCM_PATH/llvm/bin
export HIP_INCLUDE_PATH=$ROCM_PATH/include
export HIP_LIB_PATH=$ROCM_PATH/lib
export HIP_DEVICE_LIB_PATH=$ROCM_PATH/lib/llvm/amdgcn/bitcode
export PATH="$ROCM_PATH/bin:$HIP_CLANG_PATH:$PATH"
export LD_LIBRARY_PATH="$HIP_LIB_PATH:$ROCM_PATH/lib:$ROCM_PATH/lib64:$ROCM_PATH/llvm/lib:${LD_LIBRARY_PATH:-}"
export LIBRARY_PATH="$HIP_LIB_PATH:$ROCM_PATH/lib:$ROCM_PATH/lib64:${LIBRARY_PATH:-}"
export CPATH="$HIP_INCLUDE_PATH:${CPATH:-}"
export PKG_CONFIG_PATH="$ROCM_PATH/lib/pkgconfig:${PKG_CONFIG_PATH:-}"
cmake -S . -B build -DGGML_HIP=ON -DLLAMA_CURL=OFF -DGPU_TARGETS=gfx906 -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -- -j 48
cd ./build/bin/
./llama-server -m /mnt/Data5TB/Soft/Models/LLM/Deviant-EXPERIMENTAL-V2-70B.i1-Q4_K_M.gguf --tensor-split 1,1 --n-gpu-layers 999

Errors usr/bin/ld: warning: libhipblas.so.3, needed by ../../bin/libggml-hip.so.0.9.4, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: librocblas.so.5, needed by ../../bin/libggml-hip.so.0.9.4, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libamdhip64.so.7, needed by ../../bin/libggml-hip.so.0.9.4, not found (try using -rpath or -rpath-link) happened because you need to export variables before build, and before launch llamacpp. Static build not working.
You must execute this commands to set up variables before build and if you close terminal, activate venv with installed rocm, used to build llamacpp, and then set up variables again:
export ROCM_PATH=$(rocm-sdk path --root)
export HIP_PLATFORM=amd
export HIP_PATH=$ROCM_PATH
export HIP_CLANG_PATH=$ROCM_PATH/llvm/bin
export HIP_INCLUDE_PATH=$ROCM_PATH/include
export HIP_LIB_PATH=$ROCM_PATH/lib
export HIP_DEVICE_LIB_PATH=$ROCM_PATH/lib/llvm/amdgcn/bitcode
export PATH="$ROCM_PATH/bin:$HIP_CLANG_PATH:$PATH"
export LD_LIBRARY_PATH="$HIP_LIB_PATH:$ROCM_PATH/lib:$ROCM_PATH/lib64:$ROCM_PATH/llvm/lib:${LD_LIBRARY_PATH:-}"
export LIBRARY_PATH="$HIP_LIB_PATH:$ROCM_PATH/lib:$ROCM_PATH/lib64:${LIBRARY_PATH:-}"
export CPATH="$HIP_INCLUDE_PATH:${CPATH:-}"
export PKG_CONFIG_PATH="$ROCM_PATH/lib/pkgconfig:${PKG_CONFIG_PATH:-}"

Well. Happed to me too. Locked up system. Don’t know what happened. I just rebooted my server and executed build commands again, and this time all builds successfully without problems. Strange. Didn’t happened to me before, because I am using koboldcpp most of the time. And with that fork I don’t need to set up all that variables again and again before launch. Just activating venv is enough.

[Zubkover]

@Maks4d Thanks for the detailed step-by-step!
I reinstalled Ubuntu (for the 7th time now XD) and ran slightly modified set of commands due to the clean install, diff in the bold:

**sudo apt install git**
git clone --recursive https://github.com/ggml-org/llama.cpp.git
cd ./llama.cpp/
**sudo apt install python3.12-venv**
**python3 -m venv venv**
source ./venv/bin/activate
pip install --index-url https://rocm.nightlies.amd.com/v2-staging/gfx90X-dcgpu/ --pre rocm[libraries,devel]==7.11.0a20251124 --force-reinstall
rocm-sdk init
pip install ninja cmake
export ROCM_PATH=$(rocm-sdk path --root)
export HIP_PLATFORM=amd
export HIP_PATH=$ROCM_PATH
export HIP_CLANG_PATH=$ROCM_PATH/llvm/bin
export HIP_INCLUDE_PATH=$ROCM_PATH/include
export HIP_LIB_PATH=$ROCM_PATH/lib
export HIP_DEVICE_LIB_PATH=$ROCM_PATH/lib/llvm/amdgcn/bitcode
export PATH="$ROCM_PATH/bin:$HIP_CLANG_PATH:$PATH"
export LD_LIBRARY_PATH="$HIP_LIB_PATH:$ROCM_PATH/lib:$ROCM_PATH/lib64:$ROCM_PATH/llvm/lib:${LD_LIBRARY_PATH:-}"
export LIBRARY_PATH="$HIP_LIB_PATH:$ROCM_PATH/lib:$ROCM_PATH/lib64:${LIBRARY_PATH:-}"
export CPATH="$HIP_INCLUDE_PATH:${CPATH:-}"
export PKG_CONFIG_PATH="$ROCM_PATH/lib/pkgconfig:${PKG_CONFIG_PATH:-}"
cmake -S . -B build -DGGML_HIP=ON -DLLAMA_CURL=OFF -DGPU_TARGETS=gfx906 -DCMAKE_BUILD_TYPE=Release

And now it fails even sooner, immediately at build attempt with:

cmake -S . -B build -DGGML_HIP=ON -DLLAMA_CURL=OFF -DGPU_TARGETS=gfx906 -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -- -j 2
CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!

😅 Running base HIP build command from readme results in same output

BTW - no need for the usermod here? 🤔

[Maks4d]

@Zubkover usermod is needed to be executed once. It just adds user to needed groups of system. Without it you can’t work with gpu properly. PyTorch and other stuff will not see CUDA(HIP) devices if $USER is not a member of render and video groups. About this new error. I guess you need to run this command sudo apt install build-essential -y Hopefully it should fix this error.

[Zubkover]

Ran additionally then:

python3 -m venv venv
sudo apt install build-essential -y
source ./venv/bin/activate
sudo ln -s ~/llama.cpp/build/bin/llama-server /usr/local/bin/llama-server
*redone all exports for a good measure*

But it resulted in build being CPU-only ???

cmake -S . -B build -DGGML_HIP=ON -DLLAMA_CURL=OFF -DGPU_TARGETS=gfx906 -DCMAKE_BUILD_TYPE=Release
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMAKE_BUILD_TYPE=Release
-- Found Git: /bin/git (found version "2.43.0")
-- The ASM compiler identification is GNU
-- Found assembler: /bin/cc
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- GGML_SYSTEM_ARCH: x86
-- Including CPU backend
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- x86 detected
-- Adding CPU backend variant ggml-cpu: -march=native 
-- The HIP compiler identification is Clang 22.0.0
-- Detecting HIP compiler ABI info
-- Detecting HIP compiler ABI info - done
-- Check for working HIP compiler: /home/llm/llama.cpp/venv/lib/python3.12/site-packages/_rocm_sdk_devel/llvm/bin/clang++ - skipped
-- Detecting HIP compile features
-- Detecting HIP compile features - done
-- HIP and hipBLAS found
-- Including HIP backend
-- ggml version: 0.9.4
-- ggml commit:  db9783738
-- Configuring done (4.0s)
-- Generating done (0.2s)
-- Build files have been written to: /home/llm/llama.cpp/build

and then attempting to run server

lama-server -hf unsloth/gpt-oss-20b-GGUF:Q4_K_M --tensor-split 1,1 --n-gpu-layers 999
ggml_cuda_init: failed to initialize ROCm: no ROCm-capable device is detected
warning: llama.cpp was compiled without support for GPU offload. Setting a tensor split has no effect.
warning: no usable GPU found, --gpu-layers option will be ignored
warning: one possible reason is that llama.cpp was compiled without GPU support
warning: consult docs/build.md for compilation instructions
error: failed to get manifest at https://huggingface.co/v2/unsloth/gpt-oss-20b-GGUF/manifests/Q4_K_M: 'https' scheme is not supported.
error: failed to get manifest (check your internet connection)

y no internet connection? sth w/ python venv permissions?

adding

sudo usermod -a -G render,video $LOGNAME
sudo reboot
python3 -m venv venv
source ./venv/bin/activate
cmake -S . -B build -DGGML_HIP=ON -DLLAMA_CURL=OFF -DGPU_TARGETS=gfx906 -DCMAKE_BUILD_TYPE=Release

Made me loose rocm and cmake on that reboot for whatever reason:

(venv) llm@llm:~/llama.cpp$ rocm-sdk
rocm-sdk: command not found

^noticed during exports setup

(venv) llm@llm:~/llama.cpp$ cmake -S . -B build -DGGML_HIP=ON -DLLAMA_CURL=OFF -DGPU_TARGETS=gfx906 -DCMAKE_BUILD_TYPE=Releas
Command 'cmake' not found, but can be installed with:
sudo snap install cmake  # version 4.2.0, or
sudo apt  install cmake  # version 3.27.8-1build1
See 'snap info cmake' for additional versions.

Am I missing something horrendously obvious in the flow @Maks4d ?
I should use venv just like I would use similar "env shell addons" like devbox... right? 😅

[Maks4d]

@Zubkover Yes, all tools are installed inside venv, so you need to active venv before. And then export hip variables(this long wall of export, i showed before). You lose cmake and rocm? Something broke. Delete venv end recreate it. Install rocm lib and devel, cmake inside venv. apt install cmake is older, i didn't know if it compatible with building llamacpp. Run rocm-sdk init, export variables and run build commad. You can try to use my build https://disk.yandex.ru/d/n0qKfC0PT09b5w There is some symlinks, so it can be broken. I created another build, without symlinks https://disk.yandex.ru/d/cGgbanlFnt_qaA It is somewhat static, but still need to activate venv and export variables.
I disabled -DLLAMA_CURL=OFF because some of curl libs is not installed on my system and build fails if I leave this option by default ON. My build can not download from hf. Only working with pre downloaded local gguf files. Instead of -hf I'm using -m /path/to/local/gguf

Maybe try kobold, if llamacpp not working?
###Run this, if you not already done this: sudo usermod -a -G render,video $LOGNAME
###Reboot here
###Kopy paste from here
sudo apt install git python3-tk python3-pip python3-venv python-is-python3 build-essential -y
git clone --recursive https://github.com/LostRuins/koboldcpp.git
cd ./koboldcpp/
python -m venv venv
source ./venv/bin/activate
pip install --index-url https://rocm.nightlies.amd.com/v2-staging/gfx90X-dcgpu/ --pre rocm[libraries,devel]==7.11.0a20251124 torch==2.8.0+rocm7.11.0a20251124 torchaudio==2.8.0a0+rocm7.11.0a20251124 torchvision==0.23.0a0+rocm7.11.0a20251124 --force-reinstall
rocm-sdk init
pip install -r ./requirements.txt
export ROCM_PATH=$(rocm-sdk path --root)
export HIP_PLATFORM=amd
export HIP_PATH=$ROCM_PATH
export HIP_CLANG_PATH=$ROCM_PATH/llvm/bin
export HIP_INCLUDE_PATH=$ROCM_PATH/include
export HIP_LIB_PATH=$ROCM_PATH/lib
export HIP_DEVICE_LIB_PATH=$ROCM_PATH/lib/llvm/amdgcn/bitcode
export PATH="$ROCM_PATH/bin:$HIP_CLANG_PATH:$PATH"
export LD_LIBRARY_PATH="$HIP_LIB_PATH:$ROCM_PATH/lib:$ROCM_PATH/lib64:$ROCM_PATH/llvm/lib:${LD_LIBRARY_PATH:-}"
export LIBRARY_PATH="$HIP_LIB_PATH:$ROCM_PATH/lib:$ROCM_PATH/lib64:${LIBRARY_PATH:-}"
export CPATH="$HIP_INCLUDE_PATH:${CPATH:-}"
export PKG_CONFIG_PATH="$ROCM_PATH/lib/pkgconfig:${PKG_CONFIG_PATH:-}"
make LLAMA_HIPBLAS=1 AMDGPU_TARGETS="gfx906" -j2
python ./koboldcpp.py
###To here. Try this set of commands.

###To launch kobold after closing terminal
source ./venv/bin/activate
python ./koboldcpp.py

###Honestly i don't know if AMDGPU_TARGETS="gfx906" really needed to be used in make command, or if it even affect anything. I run all this commands and launched koboldcpp gui succesfully. Context 32768, can try larger. Disable QuantMatMul, enable flash attention if you want. Backend HipBlas, GPU ID All, Gpu layers 999, Tokens tab -> Quantize KV Cache can be set to 8bit, to save vram. Moe Cpu Layers 0, Hardware tab -> Tensor split 1,1 Qwen3-Next-80B-A3B-Instruct-UD-Q4_K_XL.gguf launched, using 2 gpu. Downloaded model here https://huggingface.co/unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF