Skip to content

Bug: GLM 4.5 AIR fails to load after #1231 #1237

Description

@Quairon-Nailo

What happened?

All attempts to load a GLM 4.5 AIR based model after #1231 fail. It works fine in 8d952ff

Running Linux with 2x3090 GPUs, Ryzen 9 9950X3D, and 2x64GB of RAM.

Name and Version

llama-cli --version
version: 27 (9c1c74a)
built with cc (GCC) 15.2.1 20260103 for x86_64-pc-linux-gnu

What operating system are you seeing the problem on?

Linux

Relevant log output

llama-server \
        --host 0.0.0.0 \
        --port 8085 \
        --model "/mnt/Speed/AI/Models/ApocalipseParty/iceblink-v3d-gguf/iceblink-v3d-Q8_0-FFN-Q5_K-Q5_K-Q8_0.gguf" \
        -a IceblinkV3d \
        -b 4096 \
        -ub 4096 \
        --threads 16 \
        --ctx-size 24064 \
        --n-gpu-layers 999 \
        -ot "(9|[1-2][0-9]|3[0-9])\..*_exps.*=CPU" \
        --no-mmap \
        -fa on \
        -sm graph \
        -ts 180,191 \
        -np 1 \
        -smgs \
        -cuda fusion=1
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24126 MiB
  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24103 MiB
=============================== NCCL main communicator initialized
INFO [                    main] build info | tid="140138120986624" timestamp=1770306550 build=27 commit="9c1c74a"
INFO [                    main] system info | tid="140138120986624" timestamp=1770306550 n_threads=16 n_threads_batch=-1 total_threads=32 system_info="AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | "
CUDA0: using device CUDA0 - 23743 MiB free
CUDA1: using device CUDA1 - 22565 MiB free
llama_model_loader: loaded meta data with 52 key-value pairs and 780 tensors from /mnt/Speed/AI/Models/ApocalipseParty/iceblink-v3d-gguf/iceblink-v3d-Q8_0-FFN-Q5_K-Q5_K-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = glm4moe
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Model
llama_model_loader: - kv   3:                         general.size_label str              = 128x8.0B
llama_model_loader: - kv   4:                   general.base_model.count u32              = 2
llama_model_loader: - kv   5:                  general.base_model.0.name str              = GLM 4.5 Air
llama_model_loader: - kv   6:          general.base_model.0.organization str              = Zai Org
llama_model_loader: - kv   7:              general.base_model.0.repo_url str              = https://huggingface.co/zai-org/GLM-4....
llama_model_loader: - kv   8:                  general.base_model.1.name str              = Derestricted Iceblink
llama_model_loader: - kv   9:          general.base_model.1.organization str              = ApocalypseParty
llama_model_loader: - kv  10:              general.base_model.1.repo_url str              = https://huggingface.co/ApocalypsePart...
llama_model_loader: - kv  11:                               general.tags arr[str,2]       = ["mergekit", "merge"]
llama_model_loader: - kv  12:                        glm4moe.block_count u32              = 46
llama_model_loader: - kv  13:                     glm4moe.context_length u32              = 131072
llama_model_loader: - kv  14:                   glm4moe.embedding_length u32              = 4096
llama_model_loader: - kv  15:                glm4moe.feed_forward_length u32              = 10944
llama_model_loader: - kv  16:               glm4moe.attention.head_count u32              = 96
llama_model_loader: - kv  17:            glm4moe.attention.head_count_kv u32              = 8
llama_model_loader: - kv  18:                     glm4moe.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  19:   glm4moe.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  20:                  glm4moe.expert_used_count u32              = 8
llama_model_loader: - kv  21:                 glm4moe.expert_group_count u32              = 1
llama_model_loader: - kv  22:            glm4moe.expert_group_used_count u32              = 1
llama_model_loader: - kv  23:               glm4moe.attention.key_length u32              = 128
llama_model_loader: - kv  24:             glm4moe.attention.value_length u32              = 128
llama_model_loader: - kv  25:               glm4moe.rope.dimension_count u32              = 64
llama_model_loader: - kv  26:                       glm4moe.expert_count u32              = 128
llama_model_loader: - kv  27:         glm4moe.expert_feed_forward_length u32              = 1408
llama_model_loader: - kv  28:                glm4moe.expert_shared_count u32              = 1
llama_model_loader: - kv  29:          glm4moe.leading_dense_block_count u32              = 1
llama_model_loader: - kv  30:                 glm4moe.expert_gating_func u32              = 2
llama_model_loader: - kv  31:               glm4moe.expert_weights_scale f32              = 1.000000
llama_model_loader: - kv  32:                glm4moe.expert_weights_norm bool             = true
llama_model_loader: - kv  33:               glm4moe.nextn_predict_layers u32              = 0
llama_model_loader: - kv  34:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  35:                         tokenizer.ggml.pre str              = glm4
llama_model_loader: - kv  36:                      tokenizer.ggml.tokens arr[str,151365]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  37:                  tokenizer.ggml.token_type arr[i32,151365]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  38:                      tokenizer.ggml.merges arr[str,318088]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  39:                tokenizer.ggml.eos_token_id u32              = 151329
llama_model_loader: - kv  40:            tokenizer.ggml.padding_token_id u32              = 151329
llama_model_loader: - kv  41:                tokenizer.ggml.bos_token_id u32              = 151331
llama_model_loader: - kv  42:                tokenizer.ggml.eot_token_id u32              = 151336
llama_model_loader: - kv  43:            tokenizer.ggml.unknown_token_id u32              = 151329
llama_model_loader: - kv  44:                tokenizer.ggml.eom_token_id u32              = 151338
llama_model_loader: - kv  45:                    tokenizer.chat_template str              = [gMASK]<sop>\n{%- if tools -%}\n<|syste...
llama_model_loader: - kv  46:               general.quantization_version u32              = 2
llama_model_loader: - kv  47:                          general.file_type u32              = 7
llama_model_loader: - kv  48:                      quantize.imatrix.file str              = ../iceblink-v3d-imatrix.gguf
llama_model_loader: - kv  49:                   quantize.imatrix.dataset str              = ../calibration_data.txt
llama_model_loader: - kv  50:             quantize.imatrix.entries_count u32              = 502
llama_model_loader: - kv  51:              quantize.imatrix.chunks_count u32              = 98
llama_model_loader: - type  f32:  321 tensors
llama_model_loader: - type q8_0:  369 tensors
llama_model_loader: - type q5_K:   90 tensors
load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special_eom_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 151329 ('<|endoftext|>')
load:   - 151336 ('<|user|>')
load:   - 151338 ('<|observation|>')
load: special tokens cache size = 36
load: token to piece cache size = 0.9713 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = glm4moe
llm_load_print_meta: n_ctx_train      = 131072
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_layer          = 46
llm_load_print_meta: n_head           = 96
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_swa_pattern    = 1
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 12
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 10944
llm_load_print_meta: n_expert         = 128
llm_load_print_meta: n_expert_used    = 8
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 131072
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = Q8_0
llm_load_print_meta: model params     = 106.851 B
llm_load_print_meta: model size       = 82.596 GiB (6.640 BPW) 
llm_load_print_meta: repeating layers = 81.369 GiB (6.618 BPW, 105.611 B parameters)
llm_load_print_meta: general.name     = Model
print_info: vocab type       = BPE
print_info: n_vocab          = 151365
print_info: n_merges         = 318088
print_info: BOS token        = 151331 '[gMASK]'
print_info: EOS token        = 151329 '<|endoftext|>'
print_info: EOT token        = 151336 '<|user|>'
print_info: EOM token        = 151338 '<|observation|>'
print_info: UNK token        = 151329 '<|endoftext|>'
print_info: PAD token        = 151329 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151347 '<|code_prefix|>'
print_info: FIM SUF token    = 151349 '<|code_suffix|>'
print_info: FIM MID token    = 151348 '<|code_middle|>'
print_info: EOG token        = 151329 '<|endoftext|>'
print_info: EOG token        = 151336 '<|user|>'
print_info: EOG token        = 151338 '<|observation|>'
print_info: max token length = 1024
llm_load_tensors: ggml ctx size =    3.48 MiB
Tensor blk.9.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.9.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.9.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.10.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.10.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.10.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.11.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.11.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.11.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.12.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.12.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.12.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.13.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.13.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.13.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.14.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.14.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.14.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.15.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.15.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.15.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.16.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.16.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.16.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.17.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.17.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.17.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.18.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.18.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.18.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.19.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.19.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.19.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.20.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.20.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.20.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.21.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.21.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.21.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.22.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.22.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.22.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.23.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.23.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.23.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.24.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.24.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.24.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.25.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.25.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.25.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.26.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.26.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.26.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.27.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.27.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.27.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.28.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.28.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.28.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.29.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.29.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.29.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.30.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.30.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.30.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.31.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.31.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.31.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.32.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.32.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.32.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.33.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.33.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.33.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.34.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.34.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.34.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.35.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.35.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.35.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.36.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.36.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.36.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.37.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.37.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.37.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.38.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.38.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.38.ffn_down_exps.weight buffer type overriden to CPU
Tensor blk.39.ffn_up_exps.weight buffer type overriden to CPU
Tensor blk.39.ffn_gate_exps.weight buffer type overriden to CPU
Tensor blk.39.ffn_down_exps.weight buffer type overriden to CPU
================================ max_gpu = 0
Estimated model buffer size per device:
    Device 0:  14581.48 MiB
    Device 1:  15636.12 MiB
llm_load_tensors: offloading 46 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 47/47 layers to GPU
llm_load_tensors:        CPU buffer size = 53196.00 MiB
llm_load_tensors:  CUDA_Host buffer size =   628.22 MiB
llm_load_tensors: CUDA_Split buffer size = 30217.78 MiB
llm_load_tensors:      CUDA0 buffer size =   628.24 MiB
................................................................................................... =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 0->1
~ggml_backend_cuda_context: have 0 graphs
.
===================================== llama_new_context_with_model: f16
llama_new_context_with_model: n_ctx         = 24064
llama_new_context_with_model: n_batch       = 4096
llama_new_context_with_model: n_ubatch      = 4096
llama_new_context_with_model: flash_attn    = 1
llama_new_context_with_model: attn_max_b    = 0
llama_new_context_with_model: fused_moe     = 1
llama_new_context_with_model: grouped er    = 0
llama_new_context_with_model: fused_up_gate = 1
llama_new_context_with_model: fused_mmad    = 1
llama_new_context_with_model: rope_cache    = 0
llama_new_context_with_model: graph_reuse   = 1
llama_new_context_with_model: k_cache_hadam = 0
llama_new_context_with_model: split_mode_graph_scheduling = 1
llama_new_context_with_model: reduce_type   = f16
llama_new_context_with_model: sched_async   = 0
llama_new_context_with_model: ser           = -1, 0
llama_new_context_with_model: freq_base     = 1000000.0
llama_new_context_with_model: freq_scale    = 1
llama_new_context_with_model: cuda_params   = fusion=1
 =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 0->1
 =========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 1->0
llama_kv_cache_init: CUDA_Split KV buffer size =  4324.07 MiB
llama_kv_cache_init: KV cache size per device:
    Device 0:  2162 MiB
    Device 1:  2162 MiB
llama_new_context_with_model: KV self size  = 4324.00 MiB, K (f16): 2162.00 MiB, V (f16): 2162.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.58 MiB
/home/quair/Downloads/ik_llama.cpp/src/llama-hparams.h:191: fatal error
❌ ptrace: Operation not permitted.
❌ No stack.
❌ The program is not being run.
/home/quair/Downloads/ik_llama.cpp/IceblinkV3d_88.sh: line 19: 204332 Aborted                    (core dumped) llama-server --host 0.0.0.0 --port 8085 --model "/mnt/Speed/AI/Models/ApocalipseParty/iceblink-v3d-gguf/iceblink-v3d-Q8_0-FFN-Q5_K-Q5_K-Q8_0.gguf" -a IceblinkV3d -b 4096 -ub 4096 --threads 16 --ctx-size 24064 --n-gpu-layers 999 -ot "(9|[1-2][0-9]|3[0-9])\..*_exps.*=CPU" --no-mmap -fa on -sm graph -ts 180,191 -np 1 -smgs -cuda fusion=1

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions