What happened?
All attempts to load a GLM 4.5 AIR based model after #1231 fail. It works fine in 8d952ff
Running Linux with 2x3090 GPUs, Ryzen 9 9950X3D, and 2x64GB of RAM.
Name and Version
llama-cli --version
version: 27 (9c1c74a)
built with cc (GCC) 15.2.1 20260103 for x86_64-pc-linux-gnu
What operating system are you seeing the problem on?
Linux
Relevant log output
llama-server \
--host 0.0.0.0 \
--port 8085 \
--model "/mnt/Speed/AI/Models/ApocalipseParty/iceblink-v3d-gguf/iceblink-v3d-Q8_0-FFN-Q5_K-Q5_K-Q8_0.gguf" \
-a IceblinkV3d \
-b 4096 \
-ub 4096 \
--threads 16 \
--ctx-size 24064 \
--n-gpu-layers 999 \
-ot "(9|[1-2][0-9]|3[0-9])\..*_exps.*=CPU" \
--no-mmap \
-fa on \
-sm graph \
-ts 180,191 \
-np 1 \
-smgs \
-cuda fusion=1
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24126 MiB
Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24103 MiB
=============================== NCCL main communicator initialized
INFO [ main] build info | tid="140138120986624" timestamp=1770306550 build=27 commit="9c1c74a"
INFO [ main] system info | tid="140138120986624" timestamp=1770306550 n_threads=16 n_threads_batch=-1 total_threads=32 system_info="AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | "
CUDA0: using device CUDA0 - 23743 MiB free
CUDA1: using device CUDA1 - 22565 MiB free
llama_model_loader: loaded meta data with 52 key-value pairs and 780 tensors from /mnt/Speed/AI/Models/ApocalipseParty/iceblink-v3d-gguf/iceblink-v3d-Q8_0-FFN-Q5_K-Q5_K-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = glm4moe
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Model
llama_model_loader: - kv 3: general.size_label str = 128x8.0B
llama_model_loader: - kv 4: general.base_model.count u32 = 2
llama_model_loader: - kv 5: general.base_model.0.name str = GLM 4.5 Air
llama_model_loader: - kv 6: general.base_model.0.organization str = Zai Org
llama_model_loader: - kv 7: general.base_model.0.repo_url str = https://huggingface.co/zai-org/GLM-4....
llama_model_loader: - kv 8: general.base_model.1.name str = Derestricted Iceblink
llama_model_loader: - kv 9: general.base_model.1.organization str = ApocalypseParty
llama_model_loader: - kv 10: general.base_model.1.repo_url str = https://huggingface.co/ApocalypsePart...
llama_model_loader: - kv 11: general.tags arr[str,2] = ["mergekit", "merge"]
llama_model_loader: - kv 12: glm4moe.block_count u32 = 46
llama_model_loader: - kv 13: glm4moe.context_length u32 = 131072
llama_model_loader: - kv 14: glm4moe.embedding_length u32 = 4096
llama_model_loader: - kv 15: glm4moe.feed_forward_length u32 = 10944
llama_model_loader: - kv 16: glm4moe.attention.head_count u32 = 96
llama_model_loader: - kv 17: glm4moe.attention.head_count_kv u32 = 8
llama_model_loader: - kv 18: glm4moe.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 19: glm4moe.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 20: glm4moe.expert_used_count u32 = 8
llama_model_loader: - kv 21: glm4moe.expert_group_count u32 = 1
llama_model_loader: - kv 22: glm4moe.expert_group_used_count u32 = 1
llama_model_loader: - kv 23: glm4moe.attention.key_length u32 = 128
llama_model_loader: - kv 24: glm4moe.attention.value_length u32 = 128
llama_model_loader: - kv 25: glm4moe.rope.dimension_count u32 = 64
llama_model_loader: - kv 26: glm4moe.expert_count u32 = 128
llama_model_loader: - kv 27: glm4moe.expert_feed_forward_length u32 = 1408
llama_model_loader: - kv 28: glm4moe.expert_shared_count u32 = 1
llama_model_loader: - kv 29: glm4moe.leading_dense_block_count u32 = 1
llama_model_loader: - kv 30: glm4moe.expert_gating_func u32 = 2
llama_model_loader: - kv 31: glm4moe.expert_weights_scale f32 = 1.000000
llama_model_loader: - kv 32: glm4moe.expert_weights_norm bool = true
llama_model_loader: - kv 33: glm4moe.nextn_predict_layers u32 = 0
llama_model_loader: - kv 34: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 35: tokenizer.ggml.pre str = glm4
llama_model_loader: - kv 36: tokenizer.ggml.tokens arr[str,151365] = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 37: tokenizer.ggml.token_type arr[i32,151365] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 38: tokenizer.ggml.merges arr[str,318088] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv 39: tokenizer.ggml.eos_token_id u32 = 151329
llama_model_loader: - kv 40: tokenizer.ggml.padding_token_id u32 = 151329
llama_model_loader: - kv 41: tokenizer.ggml.bos_token_id u32 = 151331
llama_model_loader: - kv 42: tokenizer.ggml.eot_token_id u32 = 151336
llama_model_loader: - kv 43: tokenizer.ggml.unknown_token_id u32 = 151329
llama_model_loader: - kv 44: tokenizer.ggml.eom_token_id u32 = 151338
llama_model_loader: - kv 45: tokenizer.chat_template str = [gMASK]<sop>\n{%- if tools -%}\n<|syste...
llama_model_loader: - kv 46: general.quantization_version u32 = 2
llama_model_loader: - kv 47: general.file_type u32 = 7
llama_model_loader: - kv 48: quantize.imatrix.file str = ../iceblink-v3d-imatrix.gguf
llama_model_loader: - kv 49: quantize.imatrix.dataset str = ../calibration_data.txt
llama_model_loader: - kv 50: quantize.imatrix.entries_count u32 = 502
llama_model_loader: - kv 51: quantize.imatrix.chunks_count u32 = 98
llama_model_loader: - type f32: 321 tensors
llama_model_loader: - type q8_0: 369 tensors
llama_model_loader: - type q5_K: 90 tensors
load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special_eom_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load: - 151329 ('<|endoftext|>')
load: - 151336 ('<|user|>')
load: - 151338 ('<|observation|>')
load: special tokens cache size = 36
load: token to piece cache size = 0.9713 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = glm4moe
llm_load_print_meta: n_ctx_train = 131072
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_layer = 46
llm_load_print_meta: n_head = 96
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_rot = 64
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_swa_pattern = 1
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 12
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 10944
llm_load_print_meta: n_expert = 128
llm_load_print_meta: n_expert_used = 8
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 2
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 131072
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = ?B
llm_load_print_meta: model ftype = Q8_0
llm_load_print_meta: model params = 106.851 B
llm_load_print_meta: model size = 82.596 GiB (6.640 BPW)
llm_load_print_meta: repeating layers = 81.369 GiB (6.618 BPW, 105.611 B parameters)
llm_load_print_meta: general.name = Model
print_info: vocab type = BPE
print_info: n_vocab = 151365
print_info: n_merges = 318088
print_info: BOS token = 151331 '[gMASK]'
print_info: EOS token = 151329 '<|endoftext|>'
print_info: EOT token = 151336 '<|user|>'
print_info: EOM token = 151338 '<|observation|>'
print_info: UNK token = 151329 '<|endoftext|>'
print_info: PAD token = 151329 '<|endoftext|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 151347 '<|code_prefix|>'
print_info: FIM SUF token = 151349 '<|code_suffix|>'
print_info: FIM MID token = 151348 '<|code_middle|>'
print_info: EOG token = 151329 '<|endoftext|>'
print_info: EOG token = 151336 '<|user|>'
print_info: EOG token = 151338 '<|observation|>'
print_info: max token length = 1024
llm_load_tensors: ggml ctx size = 3.48 MiB
Tensor blk.9.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.9.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.9.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.10.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.10.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.10.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.11.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.11.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.11.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.12.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.12.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.12.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.13.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.13.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.13.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.14.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.14.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.14.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.15.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.15.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.15.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.16.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.16.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.16.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.17.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.17.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.17.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.18.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.18.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.18.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.19.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.19.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.19.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.20.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.20.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.20.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.21.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.21.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.21.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.22.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.22.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.22.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.23.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.23.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.23.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.24.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.24.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.24.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.25.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.25.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.25.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.26.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.26.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.26.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.27.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.27.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.27.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.28.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.28.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.28.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.29.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.29.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.29.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.30.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.30.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.30.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.31.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.31.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.31.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.32.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.32.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.32.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.33.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.33.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.33.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.34.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.34.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.34.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.35.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.35.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.35.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.36.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.36.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.36.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.37.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.37.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.37.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.38.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.38.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.38.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.39.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.39.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.39.ffn_down_exps.weight buffer type overriden to CPU
================================ max_gpu = 0
Estimated model buffer size per device:
Device 0: 14581.48 MiB
Device 1: 15636.12 MiB
llm_load_tensors: offloading 46 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 47/47 layers to GPU
llm_load_tensors: CPU buffer size = 53196.00 MiB
llm_load_tensors: CUDA_Host buffer size = 628.22 MiB
llm_load_tensors: CUDA_Split buffer size = 30217.78 MiB
llm_load_tensors: CUDA0 buffer size = 628.24 MiB
................................................................................................... =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 0->1
~ggml_backend_cuda_context: have 0 graphs
.
===================================== llama_new_context_with_model: f16
llama_new_context_with_model: n_ctx = 24064
llama_new_context_with_model: n_batch = 4096
llama_new_context_with_model: n_ubatch = 4096
llama_new_context_with_model: flash_attn = 1
llama_new_context_with_model: attn_max_b = 0
llama_new_context_with_model: fused_moe = 1
llama_new_context_with_model: grouped er = 0
llama_new_context_with_model: fused_up_gate = 1
llama_new_context_with_model: fused_mmad = 1
llama_new_context_with_model: rope_cache = 0
llama_new_context_with_model: graph_reuse = 1
llama_new_context_with_model: k_cache_hadam = 0
llama_new_context_with_model: split_mode_graph_scheduling = 1
llama_new_context_with_model: reduce_type = f16
llama_new_context_with_model: sched_async = 0
llama_new_context_with_model: ser = -1, 0
llama_new_context_with_model: freq_base = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: cuda_params = fusion=1
=========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 0->1
=========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 1->0
llama_kv_cache_init: CUDA_Split KV buffer size = 4324.07 MiB
llama_kv_cache_init: KV cache size per device:
Device 0: 2162 MiB
Device 1: 2162 MiB
llama_new_context_with_model: KV self size = 4324.00 MiB, K (f16): 2162.00 MiB, V (f16): 2162.00 MiB
llama_new_context_with_model: CUDA_Host output buffer size = 0.58 MiB
/home/quair/Downloads/ik_llama.cpp/src/llama-hparams.h:191: fatal error
❌ ptrace: Operation not permitted.
❌ No stack.
❌ The program is not being run.
/home/quair/Downloads/ik_llama.cpp/IceblinkV3d_88.sh: line 19: 204332 Aborted (core dumped) llama-server --host 0.0.0.0 --port 8085 --model "/mnt/Speed/AI/Models/ApocalipseParty/iceblink-v3d-gguf/iceblink-v3d-Q8_0-FFN-Q5_K-Q5_K-Q8_0.gguf" -a IceblinkV3d -b 4096 -ub 4096 --threads 16 --ctx-size 24064 --n-gpu-layers 999 -ot "(9|[1-2][0-9]|3[0-9])\..*_exps.*=CPU" --no-mmap -fa on -sm graph -ts 180,191 -np 1 -smgs -cuda fusion=1
What happened?
All attempts to load a GLM 4.5 AIR based model after #1231 fail. It works fine in 8d952ff
Running Linux with 2x3090 GPUs, Ryzen 9 9950X3D, and 2x64GB of RAM.
Name and Version
llama-cli --version
version: 27 (9c1c74a)
built with cc (GCC) 15.2.1 20260103 for x86_64-pc-linux-gnu
What operating system are you seeing the problem on?
Linux
Relevant log output
llama-server \ --host 0.0.0.0 \ --port 8085 \ --model "/mnt/Speed/AI/Models/ApocalipseParty/iceblink-v3d-gguf/iceblink-v3d-Q8_0-FFN-Q5_K-Q5_K-Q8_0.gguf" \ -a IceblinkV3d \ -b 4096 \ -ub 4096 \ --threads 16 \ --ctx-size 24064 \ --n-gpu-layers 999 \ -ot "(9|[1-2][0-9]|3[0-9])\..*_exps.*=CPU" \ --no-mmap \ -fa on \ -sm graph \ -ts 180,191 \ -np 1 \ -smgs \ -cuda fusion=1 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 2 CUDA devices: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24126 MiB Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24103 MiB =============================== NCCL main communicator initialized INFO [ main] build info | tid="140138120986624" timestamp=1770306550 build=27 commit="9c1c74a" INFO [ main] system info | tid="140138120986624" timestamp=1770306550 n_threads=16 n_threads_batch=-1 total_threads=32 system_info="AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | " CUDA0: using device CUDA0 - 23743 MiB free CUDA1: using device CUDA1 - 22565 MiB free llama_model_loader: loaded meta data with 52 key-value pairs and 780 tensors from /mnt/Speed/AI/Models/ApocalipseParty/iceblink-v3d-gguf/iceblink-v3d-Q8_0-FFN-Q5_K-Q5_K-Q8_0.gguf (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = glm4moe llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Model llama_model_loader: - kv 3: general.size_label str = 128x8.0B llama_model_loader: - kv 4: general.base_model.count u32 = 2 llama_model_loader: - kv 5: general.base_model.0.name str = GLM 4.5 Air llama_model_loader: - kv 6: general.base_model.0.organization str = Zai Org llama_model_loader: - kv 7: general.base_model.0.repo_url str = https://huggingface.co/zai-org/GLM-4.... llama_model_loader: - kv 8: general.base_model.1.name str = Derestricted Iceblink llama_model_loader: - kv 9: general.base_model.1.organization str = ApocalypseParty llama_model_loader: - kv 10: general.base_model.1.repo_url str = https://huggingface.co/ApocalypsePart... llama_model_loader: - kv 11: general.tags arr[str,2] = ["mergekit", "merge"] llama_model_loader: - kv 12: glm4moe.block_count u32 = 46 llama_model_loader: - kv 13: glm4moe.context_length u32 = 131072 llama_model_loader: - kv 14: glm4moe.embedding_length u32 = 4096 llama_model_loader: - kv 15: glm4moe.feed_forward_length u32 = 10944 llama_model_loader: - kv 16: glm4moe.attention.head_count u32 = 96 llama_model_loader: - kv 17: glm4moe.attention.head_count_kv u32 = 8 llama_model_loader: - kv 18: glm4moe.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 19: glm4moe.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 20: glm4moe.expert_used_count u32 = 8 llama_model_loader: - kv 21: glm4moe.expert_group_count u32 = 1 llama_model_loader: - kv 22: glm4moe.expert_group_used_count u32 = 1 llama_model_loader: - kv 23: glm4moe.attention.key_length u32 = 128 llama_model_loader: - kv 24: glm4moe.attention.value_length u32 = 128 llama_model_loader: - kv 25: glm4moe.rope.dimension_count u32 = 64 llama_model_loader: - kv 26: glm4moe.expert_count u32 = 128 llama_model_loader: - kv 27: glm4moe.expert_feed_forward_length u32 = 1408 llama_model_loader: - kv 28: glm4moe.expert_shared_count u32 = 1 llama_model_loader: - kv 29: glm4moe.leading_dense_block_count u32 = 1 llama_model_loader: - kv 30: glm4moe.expert_gating_func u32 = 2 llama_model_loader: - kv 31: glm4moe.expert_weights_scale f32 = 1.000000 llama_model_loader: - kv 32: glm4moe.expert_weights_norm bool = true llama_model_loader: - kv 33: glm4moe.nextn_predict_layers u32 = 0 llama_model_loader: - kv 34: tokenizer.ggml.model str = gpt2 llama_model_loader: - kv 35: tokenizer.ggml.pre str = glm4 llama_model_loader: - kv 36: tokenizer.ggml.tokens arr[str,151365] = ["!", "\"", "#", "$", "%", "&", "'", ... llama_model_loader: - kv 37: tokenizer.ggml.token_type arr[i32,151365] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... llama_model_loader: - kv 38: tokenizer.ggml.merges arr[str,318088] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "... llama_model_loader: - kv 39: tokenizer.ggml.eos_token_id u32 = 151329 llama_model_loader: - kv 40: tokenizer.ggml.padding_token_id u32 = 151329 llama_model_loader: - kv 41: tokenizer.ggml.bos_token_id u32 = 151331 llama_model_loader: - kv 42: tokenizer.ggml.eot_token_id u32 = 151336 llama_model_loader: - kv 43: tokenizer.ggml.unknown_token_id u32 = 151329 llama_model_loader: - kv 44: tokenizer.ggml.eom_token_id u32 = 151338 llama_model_loader: - kv 45: tokenizer.chat_template str = [gMASK]<sop>\n{%- if tools -%}\n<|syste... llama_model_loader: - kv 46: general.quantization_version u32 = 2 llama_model_loader: - kv 47: general.file_type u32 = 7 llama_model_loader: - kv 48: quantize.imatrix.file str = ../iceblink-v3d-imatrix.gguf llama_model_loader: - kv 49: quantize.imatrix.dataset str = ../calibration_data.txt llama_model_loader: - kv 50: quantize.imatrix.entries_count u32 = 502 llama_model_loader: - kv 51: quantize.imatrix.chunks_count u32 = 98 llama_model_loader: - type f32: 321 tensors llama_model_loader: - type q8_0: 369 tensors llama_model_loader: - type q5_K: 90 tensors load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect load: special_eom_id is not in special_eog_ids - the tokenizer config may be incorrect load: printing all EOG tokens: load: - 151329 ('<|endoftext|>') load: - 151336 ('<|user|>') load: - 151338 ('<|observation|>') load: special tokens cache size = 36 load: token to piece cache size = 0.9713 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = glm4moe llm_load_print_meta: n_ctx_train = 131072 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_layer = 46 llm_load_print_meta: n_head = 96 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_rot = 64 llm_load_print_meta: n_swa = 0 llm_load_print_meta: n_swa_pattern = 1 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 12 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 10944 llm_load_print_meta: n_expert = 128 llm_load_print_meta: n_expert_used = 8 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 2 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 131072 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = ?B llm_load_print_meta: model ftype = Q8_0 llm_load_print_meta: model params = 106.851 B llm_load_print_meta: model size = 82.596 GiB (6.640 BPW) llm_load_print_meta: repeating layers = 81.369 GiB (6.618 BPW, 105.611 B parameters) llm_load_print_meta: general.name = Model print_info: vocab type = BPE print_info: n_vocab = 151365 print_info: n_merges = 318088 print_info: BOS token = 151331 '[gMASK]' print_info: EOS token = 151329 '<|endoftext|>' print_info: EOT token = 151336 '<|user|>' print_info: EOM token = 151338 '<|observation|>' print_info: UNK token = 151329 '<|endoftext|>' print_info: PAD token = 151329 '<|endoftext|>' print_info: LF token = 198 'Ċ' print_info: FIM PRE token = 151347 '<|code_prefix|>' print_info: FIM SUF token = 151349 '<|code_suffix|>' print_info: FIM MID token = 151348 '<|code_middle|>' print_info: EOG token = 151329 '<|endoftext|>' print_info: EOG token = 151336 '<|user|>' print_info: EOG token = 151338 '<|observation|>' print_info: max token length = 1024 llm_load_tensors: ggml ctx size = 3.48 MiB Tensor blk.9.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.9.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.9.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.10.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.10.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.10.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.11.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.11.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.11.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.12.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.12.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.12.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.13.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.13.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.13.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.14.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.14.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.14.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.15.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.15.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.15.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.16.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.16.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.16.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.17.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.17.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.17.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.18.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.18.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.18.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.19.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.19.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.19.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.20.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.20.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.20.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.21.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.21.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.21.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.22.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.22.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.22.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.23.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.23.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.23.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.24.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.24.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.24.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.25.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.25.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.25.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.26.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.26.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.26.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.27.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.27.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.27.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.28.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.28.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.28.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.29.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.29.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.29.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.30.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.30.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.30.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.31.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.31.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.31.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.32.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.32.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.32.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.33.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.33.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.33.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.34.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.34.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.34.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.35.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.35.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.35.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.36.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.36.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.36.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.37.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.37.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.37.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.38.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.38.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.38.ffn_down_exps.weight buffer type overriden to CPU Tensor blk.39.ffn_up_exps.weight buffer type overriden to CPU Tensor blk.39.ffn_gate_exps.weight buffer type overriden to CPU Tensor blk.39.ffn_down_exps.weight buffer type overriden to CPU ================================ max_gpu = 0 Estimated model buffer size per device: Device 0: 14581.48 MiB Device 1: 15636.12 MiB llm_load_tensors: offloading 46 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 47/47 layers to GPU llm_load_tensors: CPU buffer size = 53196.00 MiB llm_load_tensors: CUDA_Host buffer size = 628.22 MiB llm_load_tensors: CUDA_Split buffer size = 30217.78 MiB llm_load_tensors: CUDA0 buffer size = 628.24 MiB ................................................................................................... =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 0->1 ~ggml_backend_cuda_context: have 0 graphs . ===================================== llama_new_context_with_model: f16 llama_new_context_with_model: n_ctx = 24064 llama_new_context_with_model: n_batch = 4096 llama_new_context_with_model: n_ubatch = 4096 llama_new_context_with_model: flash_attn = 1 llama_new_context_with_model: attn_max_b = 0 llama_new_context_with_model: fused_moe = 1 llama_new_context_with_model: grouped er = 0 llama_new_context_with_model: fused_up_gate = 1 llama_new_context_with_model: fused_mmad = 1 llama_new_context_with_model: rope_cache = 0 llama_new_context_with_model: graph_reuse = 1 llama_new_context_with_model: k_cache_hadam = 0 llama_new_context_with_model: split_mode_graph_scheduling = 1 llama_new_context_with_model: reduce_type = f16 llama_new_context_with_model: sched_async = 0 llama_new_context_with_model: ser = -1, 0 llama_new_context_with_model: freq_base = 1000000.0 llama_new_context_with_model: freq_scale = 1 llama_new_context_with_model: cuda_params = fusion=1 =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 0->1 =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 1->0 llama_kv_cache_init: CUDA_Split KV buffer size = 4324.07 MiB llama_kv_cache_init: KV cache size per device: Device 0: 2162 MiB Device 1: 2162 MiB llama_new_context_with_model: KV self size = 4324.00 MiB, K (f16): 2162.00 MiB, V (f16): 2162.00 MiB llama_new_context_with_model: CUDA_Host output buffer size = 0.58 MiB /home/quair/Downloads/ik_llama.cpp/src/llama-hparams.h:191: fatal error ❌ ptrace: Operation not permitted. ❌ No stack. ❌ The program is not being run. /home/quair/Downloads/ik_llama.cpp/IceblinkV3d_88.sh: line 19: 204332 Aborted (core dumped) llama-server --host 0.0.0.0 --port 8085 --model "/mnt/Speed/AI/Models/ApocalipseParty/iceblink-v3d-gguf/iceblink-v3d-Q8_0-FFN-Q5_K-Q5_K-Q8_0.gguf" -a IceblinkV3d -b 4096 -ub 4096 --threads 16 --ctx-size 24064 --n-gpu-layers 999 -ot "(9|[1-2][0-9]|3[0-9])\..*_exps.*=CPU" --no-mmap -fa on -sm graph -ts 180,191 -np 1 -smgs -cuda fusion=1