Feature Request: GLM 4.7 Flash graph parallel support

[Nepherpitou]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Support -sm graph for GLM 4.7 Flash model.

Motivation

Model generation speed degrades a lot on long contexts, making it barely usable with context longer than 60K because of advanced thinking.

Possible Implementation

No response

[ikawrakow]

The main issue is that GLM-4.7-Flash uses MLA (same as DeepSeek and Kimi models), and I don't see how one can effectively split the MLA calculation over 2 or more GPUs. This is why I have not added split mode graph support for these models. I can certainly add split mode graph for the FFN part, but that will not help performance degradation for long context.

If someone can see a good way how to split the MLA attention tensors, let me know.

The second issue with GLM-4.7-Flash is the bad GQA ratio of 20, which makes the FA slower than it could be if it was a power of 2. The zAI people, for whatever reason, insist on using such strange GQA ratios in their models (it is 12 in GLM-4.5/4.6/4.7/AIR). And this is why performance of zAI models generally degrades more with context length.

[Nepherpitou]

The main issue is that GLM-4.7-Flash uses MLA (same as DeepSeek and Kimi models), and I don't see how one can effectively split the MLA calculation over 2 or more GPUs. This is why I have not added split mode graph support for these models. I can certainly add split mode graph for the FFN part, but that will not help performance degradation for long context.

If someone can see a good way how to split the MLA attention tensors, let me know.

The second issue with GLM-4.7-Flash is the bad GQA ratio of 20, which makes the FA slower than it could be if it was a power of 2. The zAI people, for whatever reason, insist in using such strange GQA ratios in their models (it is 12 in GLM-4.5/4.6/4.7/AIR). And this is why performance of zAI models generally degrades more with context length.

Thanks for detailed reply, really appreciate it!

[ikawrakow]

@Nepherpitou

Please check out #1182. This is not split mode graph, but on my 2x3090 system I observe 56% better TG performance at a context of 64k tokens.

[marksverdhei]

Nice catch about FA2 @ikawrakow I might consider some model surgery for better params if the other fixes dont beat me to it. Maybe some pruning and merging. ironic that its slower that what it's supposed to be with flash attention given its name though

[ikawrakow]

@marksverdhei

ironic that its slower with flash attention given its name though

Just to clarify: FA is not slower than no-FA. But it is much slower than it would be if the GQA ratio was, say, 16. PR #1182 mitigates the issue to some extent.

[ikawrakow]

The only way I can see an efficient split mode graph variant for MLA models such as GLM-4.7-Flash, DeepSeek-V2+ or Kimi is to have the KV cache duplicated on each GPU (it is actually just K-cache and, due to MLA, quite small).

f16 K-cache for these models is 576 * 2 * n_layer per token, where n_layer is the number of layers in the model. For GLM-4.7-Flash this works out to about 53 kB per token, so 3.23 GiB for 64k tokens. I.e., with 2 GPUs, instead of using 3.22 GiB total of KV cache, or 1.61 GiB per GPU, it will be 3.22 GiB per GPU. Would that be an acceptable VRAM sacrifice to get the better split mode graph performance?

[RBAncient]

(Lurker chiming in) Add me to the camp of those for which it would be worth the VRAM cost.

And thanks for your efforts!

[marksverdhei]

The only way I can see an efficient split mode graph variant for MLA models such as GLM-4.7-Flash, DeepSeek-V2+ or Kimi is to have the KV cache duplicated on each GPU (it is actually just K-cache and, due to MLA, quite small).

f16 K-cache for these models is 576 * 2 * n_layer per token, where n_layer is the number of layers in the model. For GLM-4.7-Flash this works out to about 53 kB per token, so 3.23 GiB for 64k tokens. I.e., with 2 GPUs, instead of using 3.22 GiB total of KV cache, or 1.61 GiB per GPU, it will be 3.22 GiB per GPU. Would that be an acceptable VRAM sacrifice to get the better split mode graph performance?

[ikawrakow]

OK, I did try. The split strategy I had in mind results in a lower performance than split mode layer.

[Nepherpitou]

The only way I can see an efficient split mode graph variant for MLA models such as GLM-4.7-Flash, DeepSeek-V2+ or Kimi is to have the KV cache duplicated on each GPU (it is actually just K-cache and, due to MLA, quite small).

f16 K-cache for these models is 576 * 2 * n_layer per token, where n_layer is the number of layers in the model. For GLM-4.7-Flash this works out to about 53 kB per token, so 3.23 GiB for 64k tokens. I.e., with 2 GPUs, instead of using 3.22 GiB total of KV cache, or 1.61 GiB per GPU, it will be 3.22 GiB per GPU. Would that be an acceptable VRAM sacrifice to get the better split mode graph performance?

Well... what am I doing wrong? Q8 model takes ~23Gb of VRAM on each GPU, 92Gb total for 200K context. But as I can see, it must be close to 8Gb + 3Gb on each GPU, total of 42Gb.

Details

./llamacpp/cpu/llama-server --model /mnt/data/llm-data/models/zai-org/GLM-4.7-Flash-Q8_0.gguf -ngl 99
INFO [                    main] build info | tid="134541809524736" timestamp=1769851587 build=4159 commit="373f043d"
INFO [                    main] system info | tid="134541809524736" timestamp=1769851587 n_threads=64 n_threads_batch=-1 total_threads=128 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | "
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes, VRAM: 24077 MiB
  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB
  Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB
  Device 3: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB
CUDA0: using device CUDA0 - 23683 MiB free
CUDA1: using device CUDA1 - 23860 MiB free
CUDA2: using device CUDA2 - 23860 MiB free
CUDA3: using device CUDA3 - 23860 MiB free
llama_model_loader: loaded meta data with 59 key-value pairs and 844 tensors from /mnt/data/llm-data/models/zai-org/GLM-4.7-Flash-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = deepseek2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                      general.sampling.temp f32              = 1.000000
llama_model_loader: - kv   3:                               general.name str              = Glm-4.7-Flash
llama_model_loader: - kv   4:                           general.basename str              = Glm-4.7-Flash
llama_model_loader: - kv   5:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   6:                         general.size_label str              = 64x2.6B
llama_model_loader: - kv   7:                            general.license str              = mit
llama_model_loader: - kv   8:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv   9:                   general.base_model.count u32              = 1
llama_model_loader: - kv  10:                  general.base_model.0.name str              = GLM 4.7 Flash
llama_model_loader: - kv  11:          general.base_model.0.organization str              = Zai Org
llama_model_loader: - kv  12:              general.base_model.0.repo_url str              = https://huggingface.co/zai-org/GLM-4....
llama_model_loader: - kv  13:                               general.tags arr[str,2]       = ["unsloth", "text-generation"]
llama_model_loader: - kv  14:                          general.languages arr[str,2]       = ["en", "zh"]
llama_model_loader: - kv  15:                      deepseek2.block_count u32              = 47
llama_model_loader: - kv  16:                   deepseek2.context_length u32              = 202752
llama_model_loader: - kv  17:                 deepseek2.embedding_length u32              = 2048
llama_model_loader: - kv  18:              deepseek2.feed_forward_length u32              = 10240
llama_model_loader: - kv  19:             deepseek2.attention.head_count u32              = 20
llama_model_loader: - kv  20:          deepseek2.attention.head_count_kv u32              = 1
llama_model_loader: - kv  21:                   deepseek2.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  22: deepseek2.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  23:                deepseek2.expert_used_count u32              = 4
llama_model_loader: - kv  24:               deepseek2.expert_group_count u32              = 1
llama_model_loader: - kv  25:          deepseek2.expert_group_used_count u32              = 1
llama_model_loader: - kv  26:               deepseek2.expert_gating_func u32              = 2
llama_model_loader: - kv  27:        deepseek2.leading_dense_block_count u32              = 1
llama_model_loader: - kv  28:                       deepseek2.vocab_size u32              = 154880
llama_model_loader: - kv  29:            deepseek2.attention.q_lora_rank u32              = 768
llama_model_loader: - kv  30:           deepseek2.attention.kv_lora_rank u32              = 512
llama_model_loader: - kv  31:             deepseek2.attention.key_length u32              = 576
llama_model_loader: - kv  32:           deepseek2.attention.value_length u32              = 512
llama_model_loader: - kv  33:         deepseek2.attention.key_length_mla u32              = 256
llama_model_loader: - kv  34:       deepseek2.attention.value_length_mla u32              = 256
llama_model_loader: - kv  35:       deepseek2.expert_feed_forward_length u32              = 1536
llama_model_loader: - kv  36:                     deepseek2.expert_count u32              = 64
llama_model_loader: - kv  37:              deepseek2.expert_shared_count u32              = 1
llama_model_loader: - kv  38:             deepseek2.expert_weights_scale f32              = 1.800000
llama_model_loader: - kv  39:              deepseek2.expert_weights_norm bool             = true
llama_model_loader: - kv  40:             deepseek2.rope.dimension_count u32              = 64
llama_model_loader: - kv  41:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  42:                         tokenizer.ggml.pre str              = glm4
llama_model_loader: - kv  43:                      tokenizer.ggml.tokens arr[str,154880]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  44:                  tokenizer.ggml.token_type arr[i32,154880]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  45:                      tokenizer.ggml.merges arr[str,321649]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  46:                tokenizer.ggml.eos_token_id u32              = 154820
llama_model_loader: - kv  47:            tokenizer.ggml.padding_token_id u32              = 154821
llama_model_loader: - kv  48:                tokenizer.ggml.bos_token_id u32              = 154822
llama_model_loader: - kv  49:                tokenizer.ggml.eot_token_id u32              = 154827
llama_model_loader: - kv  50:            tokenizer.ggml.unknown_token_id u32              = 154820
llama_model_loader: - kv  51:                tokenizer.ggml.eom_token_id u32              = 154829
llama_model_loader: - kv  52:                    tokenizer.chat_template str              = [gMASK]<sop>\n{%- if tools -%}\n<|syste...
llama_model_loader: - kv  53:               general.quantization_version u32              = 2
llama_model_loader: - kv  54:                          general.file_type u32              = 7
llama_model_loader: - kv  55:                      quantize.imatrix.file str              = GLM-4.7-Flash-GGUF/imatrix_unsloth.gguf
llama_model_loader: - kv  56:                   quantize.imatrix.dataset str              = unsloth_calibration_GLM-4.7-Flash.txt
llama_model_loader: - kv  57:             quantize.imatrix.entries_count u32              = 607
llama_model_loader: - kv  58:              quantize.imatrix.chunks_count u32              = 85
llama_model_loader: - type  f32:  281 tensors
llama_model_loader: - type q8_0:  563 tensors
================= Adjusted mainline llama.cpp MLA tensors to ik_llama.cpp
load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special_eom_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 154820 ('<|endoftext|>')
load:   - 154827 ('<|user|>')
load:   - 154829 ('<|observation|>')
load: special tokens cache size = 36
load: token to piece cache size = 0.9811 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = deepseek2
llm_load_print_meta: n_ctx_train      = 202752
llm_load_print_meta: n_embd           = 2048
llm_load_print_meta: n_layer          = 47
llm_load_print_meta: n_head           = 20
llm_load_print_meta: n_head_kv        = 20
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_swa_pattern    = 1
llm_load_print_meta: n_embd_head_k    = 256
llm_load_print_meta: n_embd_head_v    = 256
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 5120
llm_load_print_meta: n_embd_v_gqa     = 5120
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 10240
llm_load_print_meta: n_expert         = 64
llm_load_print_meta: n_expert_used    = 4
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 202752
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 30B.A3B
llm_load_print_meta: model ftype      = Q8_0
llm_load_print_meta: model params     = 29.943 B
llm_load_print_meta: model size       = 29.647 GiB (8.505 BPW) 
llm_load_print_meta: repeating layers = 29.019 GiB (8.505 BPW, 29.309 B parameters)
llm_load_print_meta: general.name     = Glm-4.7-Flash
llm_load_print_meta: n_layer_dense_lead   = 1
llm_load_print_meta: n_lora_q             = 768
llm_load_print_meta: n_lora_kv            = 512
llm_load_print_meta: n_ff_exp             = 1536
llm_load_print_meta: n_expert_shared      = 1
llm_load_print_meta: expert_weights_scale = 1.8
llm_load_print_meta: expert_weights_norm  = 1
llm_load_print_meta: expert_gating_func   = sigmoid
llm_load_print_meta: rope_yarn_log_mul    = 0.0000
print_info: vocab type       = BPE
print_info: n_vocab          = 154880
print_info: n_merges         = 321649
print_info: BOS token        = 154822 '[gMASK]'
print_info: EOS token        = 154820 '<|endoftext|>'
print_info: EOT token        = 154827 '<|user|>'
print_info: EOM token        = 154829 '<|observation|>'
print_info: UNK token        = 154820 '<|endoftext|>'
print_info: PAD token        = 154821 '[MASK]'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 154838 '<|code_prefix|>'
print_info: FIM SUF token    = 154840 '<|code_suffix|>'
print_info: FIM MID token    = 154839 '<|code_middle|>'
print_info: EOG token        = 154820 '<|endoftext|>'
print_info: EOG token        = 154827 '<|user|>'
print_info: EOG token        = 154829 '<|observation|>'
print_info: max token length = 1024
llm_load_tensors: ggml ctx size =   10.30 MiB
llm_load_tensors: offloading 47 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 48/48 layers to GPU
llm_load_tensors:        CPU buffer size =   321.41 MiB
llm_load_tensors:      CUDA0 buffer size =  7171.26 MiB
llm_load_tensors:      CUDA1 buffer size =  7729.57 MiB
llm_load_tensors:      CUDA2 buffer size =  7729.57 MiB
llm_load_tensors:      CUDA3 buffer size =  7406.85 MiB
....................................................................................................
============ llm_prepare_mla: need to compute 47 wkv_b tensors
Computed blk.0.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA0
Computed blk.1.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA0
Computed blk.2.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA0
Computed blk.3.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA0
Computed blk.4.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA0
Computed blk.5.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA0
Computed blk.6.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA0
Computed blk.7.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA0
Computed blk.8.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA0
Computed blk.9.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA0
Computed blk.10.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA0
Computed blk.11.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA0
Computed blk.12.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA1
Computed blk.13.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA1
Computed blk.14.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA1
Computed blk.15.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA1
Computed blk.16.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA1
Computed blk.17.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA1
Computed blk.18.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA1
Computed blk.19.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA1
Computed blk.20.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA1
Computed blk.21.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA1
Computed blk.22.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA1
Computed blk.23.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA1
Computed blk.24.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA2
Computed blk.25.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA2
Computed blk.26.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA2
Computed blk.27.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA2
Computed blk.28.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA2
Computed blk.29.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA2
Computed blk.30.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA2
Computed blk.31.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA2
Computed blk.32.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA2
Computed blk.33.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA2
Computed blk.34.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA2
Computed blk.35.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA2
Computed blk.36.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA3
Computed blk.37.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA3
Computed blk.38.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA3
Computed blk.39.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA3
Computed blk.40.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA3
Computed blk.41.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA3
Computed blk.42.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA3
Computed blk.43.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA3
Computed blk.44.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA3
Computed blk.45.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA3
Computed blk.46.attn_kv_b.weight as 512 x 8960 and stored in buffer CUDA3
===================================== llama_new_context_with_model: f16
llama_new_context_with_model: n_ctx         = 202752
llama_new_context_with_model: n_batch       = 2048
llama_new_context_with_model: n_ubatch      = 512
llama_new_context_with_model: flash_attn    = 1
llama_new_context_with_model: mla_attn      = 3
llama_new_context_with_model: attn_max_b    = 0
llama_new_context_with_model: fused_moe     = 1
llama_new_context_with_model: grouped er    = 0
llama_new_context_with_model: fused_up_gate = 1
llama_new_context_with_model: fused_mmad    = 1
llama_new_context_with_model: rope_cache    = 0
llama_new_context_with_model: graph_reuse   = 1
llama_new_context_with_model: k_cache_hadam = 0
llama_new_context_with_model: split_mode_graph_scheduling = 0
llama_new_context_with_model: reduce_type   = f16
llama_new_context_with_model: sched_async   = 0
llama_new_context_with_model: ser           = -1, 0
llama_new_context_with_model: freq_base     = 1000000.0
llama_new_context_with_model: freq_scale    = 1
llama_kv_cache_init:      CUDA0 KV buffer size =  2673.00 MiB
llama_kv_cache_init:      CUDA1 KV buffer size =  2673.00 MiB
llama_kv_cache_init:      CUDA2 KV buffer size =  2673.00 MiB
llama_kv_cache_init:      CUDA3 KV buffer size =  2450.25 MiB
llama_new_context_with_model: KV self size  = 10469.25 MiB, c^KV (f16): 10469.25 MiB, kv^T: not used
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.59 MiB
llama_new_context_with_model: pipeline parallelism enabled (n_copies=1)
llama_new_context_with_model:      CUDA0 compute buffer size = 10625.15 MiB
llama_new_context_with_model:      CUDA1 compute buffer size = 10625.15 MiB
llama_new_context_with_model:      CUDA2 compute buffer size = 10625.15 MiB
llama_new_context_with_model:      CUDA3 compute buffer size = 10625.15 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =   202.01 MiB
llama_new_context_with_model: graph nodes  = 2526
llama_new_context_with_model: graph splits = 5
XXXXXXXXXXXXXXXXXXXXX Setting only active experts offload
INFO [                    init] initializing slots | tid="134541809524736" timestamp=1769851599 n_slots=1
INFO [                    init] new slot | tid="134541809524736" timestamp=1769851599 id_slot=0 n_ctx_slot=202752
srv          init: Exclude reasoning tokens when selecting slot based on similarity: start: <think>, end: </think>
use `--reasoning-tokens none` to disable.
prompt cache is enabled, size limit: 8192 MiB
use `--cache-ram 0` to disable the prompt cache
INFO [                    main] model loaded | tid="134541809524736" timestamp=1769851599
INFO [                    main] chat template | tid="134541809524736" timestamp=1769851599 chat_template="[gMASK]<sop>\n{%- if tools -%}\n<|system|>\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{% for tool in tools %}\n{{ tool | tojson(ensure_ascii=False) }}\n{% endfor %}\n</tools>\n\nFor each function call, output the function name and arguments within the following XML format:\n<tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>{%- endif -%}\n{%- macro visible_text(content) -%}\n    {%- if content is string -%}\n        {{- content }}\n    {%- elif content is iterable and content is not mapping -%}\n        {%- for item in content -%}\n            {%- if item is mapping and item.type == 'text' -%}\n                {{- item.text }}\n            {%- elif item is string -%}\n                {{- item }}\n            {%- endif -%}\n        {%- endfor -%}\n    {%- else -%}\n        {{- content }}\n    {%- endif -%}\n{%- endmacro -%}\n{%- set ns = namespace(last_user_index=-1) %}\n{%- for m in messages %}\n    {%- if m.role == 'user' %}\n        {% set ns.last_user_index = loop.index0 -%}\n    {%- endif %}\n{%- endfor %}\n{% for m in messages %}\n{%- if m.role == 'user' -%}<|user|>{{ visible_text(m.content) }}\n{%- elif m.role == 'assistant' -%}\n<|assistant|>\n{%- set reasoning_content = '' %}\n{%- set content = visible_text(m.content) %}\n{%- if m.reasoning_content is string %}\n    {%- set reasoning_content = m.reasoning_content %}\n{%- else %}\n    {%- if '</think>' in content %}\n        {%- set reasoning_content = content.split('</think>')[0].rstrip('\\n').split('<think>')[-1].lstrip('\\n') %}\n        {%- set content = content.split('</think>')[-1].lstrip('\\n') %}\n    {%- endif %}\n{%- endif %}\n{%- if ((clear_thinking is defined and not clear_thinking) or loop.index0 > ns.last_user_index) and reasoning_content -%}\n{{ '<think>' + reasoning_content.strip() +  '</think>'}}\n{%- else -%}\n{{ '</think>' }}\n{%- endif -%}\n{%- if content.strip() -%}\n{{ content.strip() }}\n{%- endif -%}\n{% if m.tool_calls %}\n{% for tc in m.tool_calls %}\n{%- if tc.function %}\n    {%- set tc = tc.function %}\n{%- endif %}\n{{- '<tool_call>' + tc.name -}}\n{% set _args = tc.arguments %}{% for k, v in _args.items() %}<arg_key>{{ k }}</arg_key><arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>{% endfor %}</tool_call>{% endfor %}\n{% endif %}\n{%- elif m.role == 'tool' -%}\n{%- if m.content is string -%}\n{%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n    {{- '<|observation|>' }}\n{%- endif %}\n{{- '<tool_response>' }}\n{{- m.content }}\n{{- '</tool_response>' }}\n{%- else -%}\n<|observation|>{% for tr in m.content %}\n<tool_response>{{ tr.output if tr.output is defined else tr }}</tool_response>{% endfor -%}\n{% endif -%}\n{%- elif m.role == 'system' -%}\n<|system|>{{ visible_text(m.content) }}\n{%- endif -%}\n{%- endfor -%}\n{%- if add_generation_prompt -%}\n    <|assistant|>{{- '</think>' if (enable_thinking is defined and not enable_thinking) else '<think>' -}}\n{%- endif -%}"
INFO [                    main] chat template | tid="134541809524736" timestamp=1769851599 chat_example="[gMASK]<sop><|system|>\nYou are a helpful assistant<|user|>\nHello<|assistant|>\nHi there<|user|>\nHow are you?<|assistant|>" built_in=true
INFO [                    main] HTTP server listening | tid="134541809524736" timestamp=1769851599 n_threads_http="127" port="8080" hostname="127.0.0.1"
INFO [              slots_idle] all slots are idle | tid="134541809524736" timestamp=1769851599
^CINFO [              slots_idle] all slots are idle | tid="134541809524736" timestamp=1769851617
~ggml_backend_cuda_context: have 1 graphs
~ggml_backend_cuda_context: have 1 graphs
~ggml_backend_cuda_context: have 1 graphs
~ggml_backend_cuda_context: have 1 graphs

[Nepherpitou]

Well, I'm stupid. All this time running model without -mla 1

[saood06]

Well... what am I doing wrong? Q8 model takes ~23Gb of VRAM on each GPU, 92Gb total for 200K context. But as I can see, it must be close to 8Gb + 3Gb on each GPU, total of 42Gb.

You could try --attn-max-batch N (if you aren't sure on what N, try powers of 2 starting from 64) see (#237) which would really reduce your compute buffers down (at the cost of an extremely small amount of performance)

Well, I'm stupid. All this time running model without -mla 1

The logs show mla is set to 3 (which is good).

[saood06]

The other thing you could do is try Q8 K cache (-ctk q8_0) with -khad (see #1033) to reduce the KV size by basically half, and with negligible quality loss.

Edit: Forgot -khad requires attention head size to be a power of 2, so I don't think it can be used here but Q8 cache is still usually worth the tradeoff.