Bug: Garbled output with --split-mode layer on asymmetric multi-GPU setup (V100 32G+16G)

[netqer]

What happened?

Hello,

I am encountering an issue where llama-server produces garbled/invalid output when using --split-mode layer on a dual-GPU setup with asymmetric VRAM. However, using --split-mode graph works perfectly with the same hardware and model.

Environment:

OS: Linux (Ubuntu)
GPUs: 2x NVIDIA V100 SXM2
GPU 0: 16GB VRAM
GPU 1: 32GB VRAM
CUDA Version: 12.8
Model:

Path: /home/xx/models/qwen35_gguf/unsloth/Qwen3___5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q6_K_XL.gguf
Type: GGUF (Q6_K_XL)
Steps to Reproduce:

Working Command (--split-mode graph):
Running the following command produces normal, coherent text output.

./llama-server -m "/home/xx/models/qwen35_gguf/unsloth/Qwen3___5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q6_K_XL.gguf" --jinja -ngl 99 --threads 50 --ctx-size 32684 --temp 0.6 --min-p 0.0 --top-p 0.95 --top-k 20 --presence-penalty 1.0 --host 0.0.0.0 --split-mode graph

========================================================================

Failing Command (--split-mode layer):
Running the following command starts the server successfully, but the generated output is completely garbled (mojibake).

./llama-server -m "/home/xx/models/qwen35_gguf/unsloth/Qwen3___5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q6_K_XL.gguf" --jinja -ngl 99 --threads 50 --ctx-size 32684 --temp 0.6 --min-p 0.0 --top-p 0.95 --top-k 20 --presence-penalty 1.0 --host 0.0.0.0 --split-mode layer
Actual Output (Garbled):
When using --split-mode layer, the response looks like this:

%#,&151&-".)4.35,2-2#,!*#+2'43%(13"#&)-20#*50,)*"%&'1#,%(&4+2%.#(5.-5!&'2-352+!32&'"2,05.&+3(1# Expected Output: Normal natural language text similar to what is produced when using --split-mode graph.

Additional Context:

The VRAM configuration is asymmetric (32GB + 16GB). I suspect the layer splitting logic might be miscalculating memory usage or tensor distribution across the uneven GPUs when --split-mode layer is selected.
The server does not crash; it simply generates invalid tokens.
-ngl 99 is used to offload all layers to GPU.

Thank you.

Name and Version

version: 4347 (233225d)
built with cc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 for x86_64-linux-gnu

What operating system are you seeing the problem on?

No response

Relevant log output

[ikawrakow]

I think it is a miracle that there is a way you can make it work on a Volta GPU. This project does not have the resources of llama.cpp and therefore there is no one here who has racks of computers with I don't know how many different GPUs. I consider anything older than Turing to be not officially supported.

At some level it would be interesting to understand why it runs with split mode graph, but it does not with split mode layer. I don't think it has anything to do with the GPUs being asymmetric in terms of VRAM (but you could post the full log so we can see which layer went to which GPU). It is much more likely that the issue is due to Unsloth's unfortunate decision to prepare and promote models where they have left some of the tensors as f16. This is an exceedingly bad idea for a model that has been trained using bf16 precision. If you feel like it, you can download the Q6_K model, and let us know how that goes with split mode layer and graph.

There is this note on the ik_llama.cpp main page concerning Unsloth *_XL models:

[netqer]

Thank you for your response. I really appreciate your work.

Following your suggestion, I switched to the Q8_0 model, but the issue persists.

Below are the logs for Q8_0 --split-mode layer:

xx@lnx:~/program/ik_llama.cpp/build/bin$ ./llama-server -m "/home/xx/models/qwen35_gguf/unsloth/Qwen3___5-35B-A3B-GGUF/Qwen3.5-35B-A3B-Q8_0.gguf" --jinja -ngl 99 --threads 50 --ctx-size 32684 --temp 0.6 --min-p 0.0 --top-p 0.95 --top-k 20 --presence-penalty 1.0 --host 0.0.0.0 --split-mode layer
INFO [ main] build info | tid="128286737227776" timestamp=1774367160 build=4347 commit="233225db"
INFO [ main] system info | tid="128286737227776" timestamp=1774367160 n_threads=50 n_threads_batch=-1 total_threads=104 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI = 1 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | "
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
Device 0: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes, VRAM: 16144 MiB
Device 1: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
=============================== NCCL main communicator initialized
CUDA0: using device CUDA0 - 15710 MiB free
CUDA1: using device CUDA1 - 32060 MiB free
llama_model_loader: loaded meta data with 52 key-value pairs and 733 tensors from /home/xx/models/qwen35_gguf/unsloth/Qwen3___5-35B-A3B-GGUF/Qwen3.5-35B-A3B-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen35moe
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.sampling.top_k i32 = 20
llama_model_loader: - kv 3: general.sampling.top_p f32 = 0.950000
llama_model_loader: - kv 4: general.sampling.temp f32 = 1.000000
llama_model_loader: - kv 5: general.name str = Qwen3.5-35B-A3B
llama_model_loader: - kv 6: general.basename str = Qwen3.5-35B-A3B
llama_model_loader: - kv 7: general.quantized_by str = Unsloth
llama_model_loader: - kv 8: general.size_label str = 35B-A3B
llama_model_loader: - kv 9: general.license str = apache-2.0
llama_model_loader: - kv 10: general.license.link str = https://huggingface.co/Qwen/Qwen3.5-3...
llama_model_loader: - kv 11: general.repo_url str = https://huggingface.co/unsloth
llama_model_loader: - kv 12: general.base_model.count u32 = 1
llama_model_loader: - kv 13: general.base_model.0.name str = Qwen3.5 35B A3B
llama_model_loader: - kv 14: general.base_model.0.organization str = Qwen
llama_model_loader: - kv 15: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen3.5-3...
llama_model_loader: - kv 16: general.tags arr[str,2] = ["unsloth", "image-text-to-text"]
llama_model_loader: - kv 17: qwen35moe.block_count u32 = 40
llama_model_loader: - kv 18: qwen35moe.context_length u32 = 262144
llama_model_loader: - kv 19: qwen35moe.embedding_length u32 = 2048
llama_model_loader: - kv 20: qwen35moe.attention.head_count u32 = 16
llama_model_loader: - kv 21: qwen35moe.attention.head_count_kv u32 = 2
llama_model_loader: - kv 22: qwen35moe.rope.dimension_sections arr[i32,4] = [11, 11, 10, 0]
llama_model_loader: - kv 23: qwen35moe.rope.freq_base f32 = 10000000.000000
llama_model_loader: - kv 24: qwen35moe.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 25: qwen35moe.expert_count u32 = 256
llama_model_loader: - kv 26: qwen35moe.expert_used_count u32 = 8
llama_model_loader: - kv 27: qwen35moe.attention.key_length u32 = 256
llama_model_loader: - kv 28: qwen35moe.attention.value_length u32 = 256
llama_model_loader: - kv 29: qwen35moe.expert_feed_forward_length u32 = 512
llama_model_loader: - kv 30: qwen35moe.expert_shared_feed_forward_length u32 = 512
llama_model_loader: - kv 31: qwen35moe.ssm.conv_kernel u32 = 4
llama_model_loader: - kv 32: qwen35moe.ssm.state_size u32 = 128
llama_model_loader: - kv 33: qwen35moe.ssm.group_count u32 = 16
llama_model_loader: - kv 34: qwen35moe.ssm.time_step_rank u32 = 32
llama_model_loader: - kv 35: qwen35moe.ssm.inner_size u32 = 4096
llama_model_loader: - kv 36: qwen35moe.full_attention_interval u32 = 4
llama_model_loader: - kv 37: qwen35moe.rope.dimension_count u32 = 64
llama_model_loader: - kv 38: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 39: tokenizer.ggml.pre str = qwen35
llama_model_loader: - kv 40: tokenizer.ggml.tokens arr[str,248320] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 41: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 42: tokenizer.ggml.merges arr[str,247587] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 43: tokenizer.ggml.eos_token_id u32 = 248046
llama_model_loader: - kv 44: tokenizer.ggml.padding_token_id u32 = 248055
llama_model_loader: - kv 45: tokenizer.chat_template str = {%- set image_count = namespace(value...
llama_model_loader: - kv 46: general.quantization_version u32 = 2
llama_model_loader: - kv 47: general.file_type u32 = 7
llama_model_loader: - kv 48: quantize.imatrix.file str = Qwen3.5-35B-A3B-GGUF/imatrix_unsloth....
llama_model_loader: - kv 49: quantize.imatrix.dataset str = unsloth_calibration_Qwen3.5-35B-A3B.txt
llama_model_loader: - kv 50: quantize.imatrix.entries_count u32 = 510
llama_model_loader: - kv 51: quantize.imatrix.chunks_count u32 = 76
llama_model_loader: - type f32: 301 tensors
llama_model_loader: - type q8_0: 432 tensors
load: printing all EOG tokens:
load: - 248044 ('<|endoftext|>')
load: - 248046 ('<|im_end|>')
load: - 248063 ('<|fim_pad|>')
load: - 248064 ('<|repo_name|>')
load: - 248065 ('<|file_sep|>')
load: special tokens cache size = 33
load: token to piece cache size = 1.7581 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = qwen35moe
llm_load_print_meta: n_ctx_train = 262144
llm_load_print_meta: n_embd = 2048
llm_load_print_meta: n_layer = 40
llm_load_print_meta: n_head = 16
llm_load_print_meta: n_head_kv = 2
llm_load_print_meta: n_rot = 64
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_swa_pattern = 1
llm_load_print_meta: n_embd_head_k = 256
llm_load_print_meta: n_embd_head_v = 256
llm_load_print_meta: n_gqa = 8
llm_load_print_meta: n_embd_k_gqa = 512
llm_load_print_meta: n_embd_v_gqa = 512
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-06
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 0
llm_load_print_meta: n_expert = 256
llm_load_print_meta: n_expert_used = 8
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 40
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 262144
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: mrope sections = [11, 11, 10, 0]
llm_load_print_meta: ssm_d_conv = 4
llm_load_print_meta: ssm_d_inner = 4096
llm_load_print_meta: ssm_d_state = 128
llm_load_print_meta: ssm_dt_rank = 32
llm_load_print_meta: ssm_n_group = 16
llm_load_print_meta: model type = 35B.A3B
llm_load_print_meta: model ftype = Q8_0
llm_load_print_meta: model params = 34.661 B
llm_load_print_meta: model size = 34.358 GiB (8.515 BPW)
llm_load_print_meta: repeating layers = 33.352 GiB (8.516 BPW, 33.643 B parameters)
llm_load_print_meta: general.name = Qwen3.5-35B-A3B
print_info: vocab type = BPE
print_info: n_vocab = 248320
print_info: n_merges = 247587
print_info: BOS token = 11 ','
print_info: EOS token = 248046 '<|im_end|>'
print_info: EOT token = 248046 '<|im_end|>'
print_info: PAD token = 248055 '<|vision_pad|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 248060 '<|fim_prefix|>'
print_info: FIM SUF token = 248062 '<|fim_suffix|>'
print_info: FIM MID token = 248061 '<|fim_middle|>'
print_info: FIM PAD token = 248063 '<|fim_pad|>'
print_info: FIM REP token = 248064 '<|repo_name|>'
print_info: FIM SEP token = 248065 '<|file_sep|>'
print_info: EOG token = 248044 '<|endoftext|>'
print_info: EOG token = 248046 '<|im_end|>'
print_info: EOG token = 248063 '<|fim_pad|>'
print_info: EOG token = 248064 '<|repo_name|>'
print_info: EOG token = 248065 '<|file_sep|>'
print_info: max token length = 256
======================================= HAVE_FANCY_SIMD is defined
------------------- Layer sizes:
Layer 0: 855.47, 2.09, 857.56 56.00 MiB
Layer 1: 855.47, 2.09, 857.56 56.00 MiB
Layer 2: 855.47, 2.09, 857.56 56.00 MiB
Layer 3: 848.84, 63.84, 912.67 56.00 MiB
Layer 4: 855.47, 2.09, 857.56 56.00 MiB
Layer 5: 855.47, 2.09, 857.56 56.00 MiB
Layer 6: 855.47, 2.09, 857.56 56.00 MiB
Layer 7: 848.84, 63.84, 912.67 56.00 MiB
Layer 8: 855.47, 2.09, 857.56 56.00 MiB
Layer 9: 855.47, 2.09, 857.56 56.00 MiB
Layer 10: 855.47, 2.09, 857.56 56.00 MiB
Layer 11: 848.84, 63.84, 912.67 56.00 MiB
Layer 12: 855.47, 2.09, 857.56 56.00 MiB
Layer 13: 855.47, 2.09, 857.56 56.00 MiB
Layer 14: 855.47, 2.09, 857.56 56.00 MiB
Layer 15: 848.84, 63.84, 912.67 56.00 MiB
Layer 16: 855.47, 2.09, 857.56 56.00 MiB
Layer 17: 855.47, 2.09, 857.56 56.00 MiB
Layer 18: 855.47, 2.09, 857.56 56.00 MiB
Layer 19: 848.84, 63.84, 912.67 56.00 MiB
Layer 20: 855.47, 2.09, 857.56 56.00 MiB
Layer 21: 855.47, 2.09, 857.56 56.00 MiB
Layer 22: 855.47, 2.09, 857.56 56.00 MiB
Layer 23: 848.84, 63.84, 912.67 56.00 MiB
Layer 24: 855.47, 2.09, 857.56 56.00 MiB
Layer 25: 855.47, 2.09, 857.56 56.00 MiB
Layer 26: 855.47, 2.09, 857.56 56.00 MiB
Layer 27: 848.84, 63.84, 912.67 56.00 MiB
Layer 28: 855.47, 2.09, 857.56 56.00 MiB
Layer 29: 855.47, 2.09, 857.56 56.00 MiB
Layer 30: 855.47, 2.09, 857.56 56.00 MiB
Layer 31: 848.84, 63.84, 912.67 56.00 MiB
Layer 32: 855.47, 2.09, 857.56 56.00 MiB
Layer 33: 855.47, 2.09, 857.56 56.00 MiB
Layer 34: 855.47, 2.09, 857.56 56.00 MiB
Layer 35: 848.84, 63.84, 912.67 56.00 MiB
Layer 36: 855.47, 2.09, 857.56 56.00 MiB
Layer 37: 855.47, 2.09, 857.56 56.00 MiB
Layer 38: 855.47, 2.09, 857.56 56.00 MiB
Layer 39: 848.84, 63.84, 912.67 56.00 MiB
Layer 40: 515.31, 429.00, 944.31 MiB (output layer)

Total : 34152.46, 1130.17, 35282.64 MiB
Setting default device in layer 0 to 0
Setting default device in layer 1 to 0
Setting default device in layer 2 to 0
Setting default device in layer 3 to 0
Setting default device in layer 4 to 0
Setting default device in layer 5 to 0
Setting default device in layer 6 to 0
Setting default device in layer 7 to 0
Setting default device in layer 8 to 0
Setting default device in layer 9 to 0
Setting default device in layer 10 to 0
Setting default device in layer 11 to 0
Setting default device in layer 12 to 0
Setting default device in layer 13 to 0
Setting default device in layer 14 to 1
Setting default device in layer 15 to 1
Setting default device in layer 16 to 1
Setting default device in layer 17 to 1
Setting default device in layer 18 to 1
Setting default device in layer 19 to 1
Setting default device in layer 20 to 1
Setting default device in layer 21 to 1
Setting default device in layer 22 to 1
Setting default device in layer 23 to 1
Setting default device in layer 24 to 1
Setting default device in layer 25 to 1
Setting default device in layer 26 to 1
Setting default device in layer 27 to 1
Setting default device in layer 28 to 1
Setting default device in layer 29 to 1
Setting default device in layer 30 to 1
Setting default device in layer 31 to 1
Setting default device in layer 32 to 1
Setting default device in layer 33 to 1
Setting default device in layer 34 to 1
Setting default device in layer 35 to 1
Setting default device in layer 36 to 1
Setting default device in layer 37 to 1
Setting default device in layer 38 to 1
Setting default device in layer 39 to 1
Setting default device in layer 40 to 1
llm_load_tensors: ggml ctx size = 3.20 MiB
llm_load_tensors: offloading 40 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 41/41 layers to GPU
llm_load_tensors: CPU buffer size = 515.31 MiB
llm_load_tensors: CUDA0 buffer size = 11956.68 MiB
llm_load_tensors: CUDA1 buffer size = 22711.11 MiB
....................................................................................................
llama_init_from_model: n_ctx = 32768
llama_init_from_model: n_batch = 2048
llama_init_from_model: n_ubatch = 512
llama_init_from_model: flash_attn = 1
llama_init_from_model: attn_max_b = 0
llama_init_from_model: fused_moe = 1
llama_init_from_model: grouped er = 0
llama_init_from_model: fused_up_gate = 1
llama_init_from_model: fused_mmad = 1
llama_init_from_model: rope_cache = 0
llama_init_from_model: graph_reuse = 1
llama_init_from_model: k_cache_hadam = 0
llama_init_from_model: split_mode_graph_scheduling = 0
llama_init_from_model: reduce_type = f16
llama_init_from_model: sched_async = 0
llama_init_from_model: ser = -1, 0
llama_init_from_model: freq_base = 10000000.0
llama_init_from_model: freq_scale = 1
=========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 0->1
=========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 1->0
llama_kv_cache_init: CUDA0 KV buffer size = 215.03 MiB
llama_kv_cache_init: CUDA1 KV buffer size = 487.78 MiB
llama_init_from_model: KV self size = 640.00 MiB, K (f16): 320.00 MiB, V (f16): 320.00 MiB
llama_init_from_model: CUDA_Host output buffer size = 0.95 MiB
llama_init_from_model: pipeline parallelism enabled (n_copies=1)
llama_init_from_model: CUDA0 compute buffer size = 84.17 MiB
llama_init_from_model: CUDA1 compute buffer size = 489.00 MiB
llama_init_from_model: CUDA_Host compute buffer size = 36.01 MiB
llama_init_from_model: graph nodes = 2615
llama_init_from_model: graph splits = 3
llama_init_from_model: enabling only_active_experts scheduling
INFO [ init] initializing slots | tid="128286737227776" timestamp=1774367184 n_slots=1
INFO [ init] new slot | tid="128286737227776" timestamp=1774367184 id_slot=0 n_ctx_slot=32768
srv init: Exclude reasoning tokens when selecting slot based on similarity: start: , end:
use --reasoning-tokens none to disable.
no implementations specified for speculative decoding
slot init: id 0 | task -1 | speculative decoding context not initialized
prompt cache is enabled, size limit: 8192 MiB
use --cache-ram 0 to disable the prompt cache
init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

'
srv init: init: chat template, thinking = 1
INFO [ main] model loaded | tid="128286737227776" timestamp=1774367184
INFO [ main] HTTP server listening | tid="128286737227776" timestamp=1774367184 n_threads_http="103" port="8080" hostname="0.0.0.0"
INFO [ slots_idle] all slots are idle | tid="128286737227776" timestamp=1774367184
======== Prompt cache: cache size: 0, n_keep: 0, n_discarded_prompt: 0, cache_ram_n_min: 0, f_keep: 0.00, cache_ram_similarity: 0.50
banned strings is not supported by recurrent model, it will be disabled.
WARN [ launch_slot_with_task] %s
| ="ctx_shift is not supported by recurrent model, it will be disabled"
INFO [ launch_slot_with_task] slot is processing task | tid="128286737227776" timestamp=1774367215 id_slot=0 id_task=0
======== Cache: cache_size = 0, n_past0 = 0, n_past1 = 0, n_past_prompt1 = 0, n_past2 = 0, n_past_prompt2 = 0
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="128286737227776" timestamp=1774367215 id_slot=0 id_task=0 p0=0
slot create_check: id 0 | task 0 | created context checkpoint 1 of 32 (pos_min = 33, pos_max = 33, size = 62.814 MiB, took 65.28 ms)
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="128286737227776" timestamp=1774367215 id_slot=0 id_task=0 p0=34
INFO [ log_server_request] request | tid="128281244332032" timestamp=1774367222 remote_addr="192.168.1.19" remote_port=59213 status=200 method="POST" path="/v1/chat/completions" params={}
srv stop: cancel task, id_task = 0
INFO [ release_slots] slot released | tid="128286737227776" timestamp=1774367223 id_slot=0 id_task=0 n_ctx=32768 n_past=162 n_system_tokens=0 n_cache_tokens=162 truncated=false
INFO [ slots_idle] all slots are idle | tid="128286737227776" timestamp=1774367223

---

**********Below are the logs for Q8_0 --split-mode graph:

xx@lnx:~/program/ik_llama.cpp/build/bin$ ./llama-server -m "/home/xx/models/qwen35_gguf/unsloth/Qwen3___5-35B-A3B-GGUF/Qwen3.5-35B-A3B-Q8_0.gguf" --jinja -ngl 99 --threads 50 --ctx-size 32684 --temp 0.6 --min-p 0.0 --top-p 0.95 --top-k 20 --presence-penalty 1.0 --host 0.0.0.0 --split-mode graph
INFO [ main] build info | tid="136757751341056" timestamp=1774367616 build=4347 commit="233225db"
INFO [ main] system info | tid="136757751341056" timestamp=1774367616 n_threads=50 n_threads_batch=-1 total_threads=104 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI = 1 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | "
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
Device 0: Tesla V100-SXM2-16GB, compute capability 7.0, VMM: yes, VRAM: 16144 MiB
Device 1: Tesla V100-SXM2-32GB, compute capability 7.0, VMM: yes, VRAM: 32494 MiB
=============================== NCCL main communicator initialized
CUDA0: using device CUDA0 - 15710 MiB free
CUDA1: using device CUDA1 - 32060 MiB free
llama_model_loader: loaded meta data with 52 key-value pairs and 733 tensors from /home/xx/models/qwen35_gguf/unsloth/Qwen3___5-35B-A3B-GGUF/Qwen3.5-35B-A3B-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen35moe
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.sampling.top_k i32 = 20
llama_model_loader: - kv 3: general.sampling.top_p f32 = 0.950000
llama_model_loader: - kv 4: general.sampling.temp f32 = 1.000000
llama_model_loader: - kv 5: general.name str = Qwen3.5-35B-A3B
llama_model_loader: - kv 6: general.basename str = Qwen3.5-35B-A3B
llama_model_loader: - kv 7: general.quantized_by str = Unsloth
llama_model_loader: - kv 8: general.size_label str = 35B-A3B
llama_model_loader: - kv 9: general.license str = apache-2.0
llama_model_loader: - kv 10: general.license.link str = https://huggingface.co/Qwen/Qwen3.5-3...
llama_model_loader: - kv 11: general.repo_url str = https://huggingface.co/unsloth
llama_model_loader: - kv 12: general.base_model.count u32 = 1
llama_model_loader: - kv 13: general.base_model.0.name str = Qwen3.5 35B A3B
llama_model_loader: - kv 14: general.base_model.0.organization str = Qwen
llama_model_loader: - kv 15: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen3.5-3...
llama_model_loader: - kv 16: general.tags arr[str,2] = ["unsloth", "image-text-to-text"]
llama_model_loader: - kv 17: qwen35moe.block_count u32 = 40
llama_model_loader: - kv 18: qwen35moe.context_length u32 = 262144
llama_model_loader: - kv 19: qwen35moe.embedding_length u32 = 2048
llama_model_loader: - kv 20: qwen35moe.attention.head_count u32 = 16
llama_model_loader: - kv 21: qwen35moe.attention.head_count_kv u32 = 2
llama_model_loader: - kv 22: qwen35moe.rope.dimension_sections arr[i32,4] = [11, 11, 10, 0]
llama_model_loader: - kv 23: qwen35moe.rope.freq_base f32 = 10000000.000000
llama_model_loader: - kv 24: qwen35moe.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 25: qwen35moe.expert_count u32 = 256
llama_model_loader: - kv 26: qwen35moe.expert_used_count u32 = 8
llama_model_loader: - kv 27: qwen35moe.attention.key_length u32 = 256
llama_model_loader: - kv 28: qwen35moe.attention.value_length u32 = 256
llama_model_loader: - kv 29: qwen35moe.expert_feed_forward_length u32 = 512
llama_model_loader: - kv 30: qwen35moe.expert_shared_feed_forward_length u32 = 512
llama_model_loader: - kv 31: qwen35moe.ssm.conv_kernel u32 = 4
llama_model_loader: - kv 32: qwen35moe.ssm.state_size u32 = 128
llama_model_loader: - kv 33: qwen35moe.ssm.group_count u32 = 16
llama_model_loader: - kv 34: qwen35moe.ssm.time_step_rank u32 = 32
llama_model_loader: - kv 35: qwen35moe.ssm.inner_size u32 = 4096
llama_model_loader: - kv 36: qwen35moe.full_attention_interval u32 = 4
llama_model_loader: - kv 37: qwen35moe.rope.dimension_count u32 = 64
llama_model_loader: - kv 38: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 39: tokenizer.ggml.pre str = qwen35
llama_model_loader: - kv 40: tokenizer.ggml.tokens arr[str,248320] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 41: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 42: tokenizer.ggml.merges arr[str,247587] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 43: tokenizer.ggml.eos_token_id u32 = 248046
llama_model_loader: - kv 44: tokenizer.ggml.padding_token_id u32 = 248055
llama_model_loader: - kv 45: tokenizer.chat_template str = {%- set image_count = namespace(value...
llama_model_loader: - kv 46: general.quantization_version u32 = 2
llama_model_loader: - kv 47: general.file_type u32 = 7
llama_model_loader: - kv 48: quantize.imatrix.file str = Qwen3.5-35B-A3B-GGUF/imatrix_unsloth....
llama_model_loader: - kv 49: quantize.imatrix.dataset str = unsloth_calibration_Qwen3.5-35B-A3B.txt
llama_model_loader: - kv 50: quantize.imatrix.entries_count u32 = 510
llama_model_loader: - kv 51: quantize.imatrix.chunks_count u32 = 76
llama_model_loader: - type f32: 301 tensors
llama_model_loader: - type q8_0: 432 tensors
load: printing all EOG tokens:
load: - 248044 ('<|endoftext|>')
load: - 248046 ('<|im_end|>')
load: - 248063 ('<|fim_pad|>')
load: - 248064 ('<|repo_name|>')
load: - 248065 ('<|file_sep|>')
load: special tokens cache size = 33
load: token to piece cache size = 1.7581 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = qwen35moe
llm_load_print_meta: n_ctx_train = 262144
llm_load_print_meta: n_embd = 2048
llm_load_print_meta: n_layer = 40
llm_load_print_meta: n_head = 16
llm_load_print_meta: n_head_kv = 2
llm_load_print_meta: n_rot = 64
llm_load_print_meta: n_swa = 0
llm_load_print_meta: n_swa_pattern = 1
llm_load_print_meta: n_embd_head_k = 256
llm_load_print_meta: n_embd_head_v = 256
llm_load_print_meta: n_gqa = 8
llm_load_print_meta: n_embd_k_gqa = 512
llm_load_print_meta: n_embd_v_gqa = 512
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-06
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 0
llm_load_print_meta: n_expert = 256
llm_load_print_meta: n_expert_used = 8
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 40
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 262144
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: mrope sections = [11, 11, 10, 0]
llm_load_print_meta: ssm_d_conv = 4
llm_load_print_meta: ssm_d_inner = 4096
llm_load_print_meta: ssm_d_state = 128
llm_load_print_meta: ssm_dt_rank = 32
llm_load_print_meta: ssm_n_group = 16
llm_load_print_meta: model type = 35B.A3B
llm_load_print_meta: model ftype = Q8_0
llm_load_print_meta: model params = 34.661 B
llm_load_print_meta: model size = 34.358 GiB (8.515 BPW)
llm_load_print_meta: repeating layers = 33.352 GiB (8.516 BPW, 33.643 B parameters)
llm_load_print_meta: general.name = Qwen3.5-35B-A3B
print_info: vocab type = BPE
print_info: n_vocab = 248320
print_info: n_merges = 247587
print_info: BOS token = 11 ','
print_info: EOS token = 248046 '<|im_end|>'
print_info: EOT token = 248046 '<|im_end|>'
print_info: PAD token = 248055 '<|vision_pad|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 248060 '<|fim_prefix|>'
print_info: FIM SUF token = 248062 '<|fim_suffix|>'
print_info: FIM MID token = 248061 '<|fim_middle|>'
print_info: FIM PAD token = 248063 '<|fim_pad|>'
print_info: FIM REP token = 248064 '<|repo_name|>'
print_info: FIM SEP token = 248065 '<|file_sep|>'
print_info: EOG token = 248044 '<|endoftext|>'
print_info: EOG token = 248046 '<|im_end|>'
print_info: EOG token = 248063 '<|fim_pad|>'
print_info: EOG token = 248064 '<|repo_name|>'
print_info: EOG token = 248065 '<|file_sep|>'
print_info: max token length = 256
======================================= HAVE_FANCY_SIMD is defined
------------------- Layer sizes:
Layer 0: 855.47, 2.09, 857.56 56.00 MiB
Layer 1: 855.47, 2.09, 857.56 56.00 MiB
Layer 2: 855.47, 2.09, 857.56 56.00 MiB
Layer 3: 848.84, 63.84, 912.67 56.00 MiB
Layer 4: 855.47, 2.09, 857.56 56.00 MiB
Layer 5: 855.47, 2.09, 857.56 56.00 MiB
Layer 6: 855.47, 2.09, 857.56 56.00 MiB
Layer 7: 848.84, 63.84, 912.67 56.00 MiB
Layer 8: 855.47, 2.09, 857.56 56.00 MiB
Layer 9: 855.47, 2.09, 857.56 56.00 MiB
Layer 10: 855.47, 2.09, 857.56 56.00 MiB
Layer 11: 848.84, 63.84, 912.67 56.00 MiB
Layer 12: 855.47, 2.09, 857.56 56.00 MiB
Layer 13: 855.47, 2.09, 857.56 56.00 MiB
Layer 14: 855.47, 2.09, 857.56 56.00 MiB
Layer 15: 848.84, 63.84, 912.67 56.00 MiB
Layer 16: 855.47, 2.09, 857.56 56.00 MiB
Layer 17: 855.47, 2.09, 857.56 56.00 MiB
Layer 18: 855.47, 2.09, 857.56 56.00 MiB
Layer 19: 848.84, 63.84, 912.67 56.00 MiB
Layer 20: 855.47, 2.09, 857.56 56.00 MiB
Layer 21: 855.47, 2.09, 857.56 56.00 MiB
Layer 22: 855.47, 2.09, 857.56 56.00 MiB
Layer 23: 848.84, 63.84, 912.67 56.00 MiB
Layer 24: 855.47, 2.09, 857.56 56.00 MiB
Layer 25: 855.47, 2.09, 857.56 56.00 MiB
Layer 26: 855.47, 2.09, 857.56 56.00 MiB
Layer 27: 848.84, 63.84, 912.67 56.00 MiB
Layer 28: 855.47, 2.09, 857.56 56.00 MiB
Layer 29: 855.47, 2.09, 857.56 56.00 MiB
Layer 30: 855.47, 2.09, 857.56 56.00 MiB
Layer 31: 848.84, 63.84, 912.67 56.00 MiB
Layer 32: 855.47, 2.09, 857.56 56.00 MiB
Layer 33: 855.47, 2.09, 857.56 56.00 MiB
Layer 34: 855.47, 2.09, 857.56 56.00 MiB
Layer 35: 848.84, 63.84, 912.67 56.00 MiB
Layer 36: 855.47, 2.09, 857.56 56.00 MiB
Layer 37: 855.47, 2.09, 857.56 56.00 MiB
Layer 38: 855.47, 2.09, 857.56 56.00 MiB
Layer 39: 848.84, 63.84, 912.67 56.00 MiB
Layer 40: 515.31, 429.00, 944.31 MiB (output layer)

Total : 34152.46, 1130.17, 35282.64 MiB
Setting default device in layer 0 to 0
Setting default device in layer 1 to 0
Setting default device in layer 2 to 0
Setting default device in layer 3 to 0
Setting default device in layer 4 to 0
Setting default device in layer 5 to 0
Setting default device in layer 6 to 0
Setting default device in layer 7 to 0
Setting default device in layer 8 to 0
Setting default device in layer 9 to 0
Setting default device in layer 10 to 0
Setting default device in layer 11 to 0
Setting default device in layer 12 to 0
Setting default device in layer 13 to 0
Setting default device in layer 14 to 1
Setting default device in layer 15 to 1
Setting default device in layer 16 to 1
Setting default device in layer 17 to 1
Setting default device in layer 18 to 1
Setting default device in layer 19 to 1
Setting default device in layer 20 to 1
Setting default device in layer 21 to 1
Setting default device in layer 22 to 1
Setting default device in layer 23 to 1
Setting default device in layer 24 to 1
Setting default device in layer 25 to 1
Setting default device in layer 26 to 1
Setting default device in layer 27 to 1
Setting default device in layer 28 to 1
Setting default device in layer 29 to 1
Setting default device in layer 30 to 1
Setting default device in layer 31 to 1
Setting default device in layer 32 to 1
Setting default device in layer 33 to 1
Setting default device in layer 34 to 1
Setting default device in layer 35 to 1
Setting default device in layer 36 to 1
Setting default device in layer 37 to 1
Setting default device in layer 38 to 1
Setting default device in layer 39 to 1
Setting default device in layer 40 to 1
llm_load_tensors: ggml ctx size = 4.27 MiB
================================ max_gpu = 0
Estimated model buffer size per device:
Device 0: 10822.15 MiB
Device 1: 23411.29 MiB
No tensors in buffer type CUDA0
llm_load_tensors: offloading 40 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 41/41 layers to GPU
llm_load_tensors: CPU buffer size = 515.31 MiB
llm_load_tensors: CUDA_Split buffer size = 34233.97 MiB
llm_load_tensors: CUDA1 buffer size = 515.32 MiB
...................................................................................................
llama_init_from_model: n_ctx = 32768
llama_init_from_model: n_batch = 2048
llama_init_from_model: n_ubatch = 512
llama_init_from_model: flash_attn = 1
llama_init_from_model: attn_max_b = 0
llama_init_from_model: fused_moe = 1
llama_init_from_model: grouped er = 0
llama_init_from_model: fused_up_gate = 1
llama_init_from_model: fused_mmad = 1
llama_init_from_model: rope_cache = 0
llama_init_from_model: graph_reuse = 1
llama_init_from_model: k_cache_hadam = 0
llama_init_from_model: split_mode_graph_scheduling = 0
llama_init_from_model: reduce_type = f16
llama_init_from_model: sched_async = 0
llama_init_from_model: ser = -1, 0
llama_init_from_model: freq_base = 10000000.0
llama_init_from_model: freq_scale = 1
=========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 0->1
=========================== ggml_cuda_set_peer_access: Enabling Peer Access between Devices 1->0
llama_kv_cache_init: CUDA_Split KV buffer size = 702.82 MiB
llama_kv_cache_init: KV cache size per device:
Device 0: 339.105 MiB
Device 1: 363.707 MiB
llama_init_from_model: KV self size = 640.00 MiB, K (f16): 320.00 MiB, V (f16): 320.00 MiB
llama_init_from_model: CUDA_Host output buffer size = 0.95 MiB
llama_init_from_model: CUDA0 compute buffer size = 78.50 MiB
llama_init_from_model: CUDA1 compute buffer size = 493.00 MiB
llama_init_from_model: CUDA_Host compute buffer size = 36.01 MiB
llama_init_from_model: graph nodes = 5386
llama_init_from_model: graph splits = 241
llama_init_from_model: enabling only_active_experts scheduling
INFO [ init] initializing slots | tid="136757751341056" timestamp=1774367650 n_slots=1
INFO [ init] new slot | tid="136757751341056" timestamp=1774367650 id_slot=0 n_ctx_slot=32768
srv init: Exclude reasoning tokens when selecting slot based on similarity: start: , end:
use --reasoning-tokens none to disable.
no implementations specified for speculative decoding
slot init: id 0 | task -1 | speculative decoding context not initialized
prompt cache is enabled, size limit: 8192 MiB
use --cache-ram 0 to disable the prompt cache
init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

'
srv init: init: chat template, thinking = 1
INFO [ main] model loaded | tid="136757751341056" timestamp=1774367650
INFO [ main] HTTP server listening | tid="136757751341056" timestamp=1774367650 n_threads_http="103" port="8080" hostname="0.0.0.0"
INFO [ slots_idle] all slots are idle | tid="136757751341056" timestamp=1774367650
======== Prompt cache: cache size: 0, n_keep: 0, n_discarded_prompt: 0, cache_ram_n_min: 0, f_keep: 0.00, cache_ram_similarity: 0.50
banned strings is not supported by recurrent model, it will be disabled.
WARN [ launch_slot_with_task] %s
| ="ctx_shift is not supported by recurrent model, it will be disabled"
INFO [ launch_slot_with_task] slot is processing task | tid="136757751341056" timestamp=1774367677 id_slot=0 id_task=0
======== Cache: cache_size = 0, n_past0 = 0, n_past1 = 0, n_past_prompt1 = 0, n_past2 = 0, n_past_prompt2 = 0
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="136757751341056" timestamp=1774367677 id_slot=0 id_task=0 p0=0
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="136757751341056" timestamp=1774367677 id_slot=0 id_task=0 p0=9
slot create_check: id 0 | task 0 | created context checkpoint 1 of 32 (pos_min = 511, pos_max = 511, size = 62.818 MiB, took 62.12 ms)
slot print_timing: id 0 | task 0 |
prompt eval time = 117.94 ms / 14 tokens ( 8.42 ms per token, 118.70 tokens per second)
eval time = 7868.28 ms / 613 tokens ( 12.84 ms per token, 77.91 tokens per second)
total time = 7986.22 ms / 627 tokens
INFO [ log_server_request] request | tid="136752270487552" timestamp=1774367685 remote_addr="192.168.1.19" remote_port=56337 status=200 method="POST" path="/v1/chat/completions" params={}
INFO [ release_slots] slot released | tid="136757751341056" timestamp=1774367685 id_slot=0 id_task=0 n_ctx=32768 n_past=627 n_system_tokens=0 n_cache_tokens=627 truncated=false
INFO [ slots_idle] all slots are idle | tid="136757751341056" timestamp=1774367685
slot create_check: id 0 | task 0 | created context checkpoint 2 of 32 (pos_min = 625, pos_max = 625, size = 62.819 MiB, took 62.12 ms)
INFO [ log_server_request] request | tid="136751509164032" timestamp=1774367697 remote_addr="192.168.1.19" remote_port=53118 status=200 method="OPTIONS" path="/v1/chat/completions" params={}
======== Prompt cache: cache size: 627, n_keep: 0, n_discarded_prompt: 0, cache_ram_n_min: 0, f_keep: 0.00, cache_ram_similarity: 0.50
updating prompt cache

• saving prompt with length 627, total state size = 75.045 MiB
  prompt cache save took 155.03 ms
• looking for better prompt, base f_keep = 0.003, sim = 0.006, n_keep = 0, n_discarded_prompt = 0
• cache state: 1 prompts, 200.682 MiB (limits: 8192.000 MiB, 0 tokens, 25594 est)
  • prompt 0x5ef6a6d8fb20: 627 tokens, 0 discarded, checkpoints: 2, 200.682 MiB
    prompt cache load took 0.20 ms
    banned strings is not supported by recurrent model, it will be disabled.
    INFO [ launch_slot_with_task] slot is processing task | tid="136757751341056" timestamp=1774367697 id_slot=0 id_task=615
    ======== Cache: cache_size = 627, n_past0 = 1, n_past1 = 1, n_past_prompt1 = 1, n_past2 = 1, n_past_prompt2 = 1
    Common part does not match fully
    slot apply_checkp: id 0 | task 615 | n_past = 1, slot.prompt.tokens.size() = 627, seq_id = 0, pos_min = 625
    slot apply_checkp: id 0 | task 615 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see kv-cache : add SWA support ggml-org/llama.cpp#13194 (comment))
    slot apply_checkp: id 0 | task 615 | erased invalidated context checkpoint (pos_min = 511, pos_max = 511, size = 62.818 MiB)
    slot apply_checkp: id 0 | task 615 | erased invalidated context checkpoint (pos_min = 625, pos_max = 625, size = 62.819 MiB)
    INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="136757751341056" timestamp=1774367697 id_slot=0 id_task=615 p0=0
    slot create_check: id 0 | task 615 | created context checkpoint 1 of 32 (pos_min = 24, pos_max = 24, size = 62.814 MiB, took 60.23 ms)
    INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="136757751341056" timestamp=1774367697 id_slot=0 id_task=615 p0=25
    slot print_timing: id 0 | task 615 |
    prompt eval time = 186.71 ms / 30 tokens ( 6.22 ms per token, 160.68 tokens per second)
    eval time = 29.48 ms / 3 tokens ( 9.83 ms per token, 101.77 tokens per second)
    total time = 216.19 ms / 33 tokens
    INFO [ log_server_request] request | tid="136751509164032" timestamp=1774367697 remote_addr="192.168.1.19" remote_port=53118 status=200 method="POST" path="/v1/chat/completions" params={}
    INFO [ release_slots] slot released | tid="136757751341056" timestamp=1774367697 id_slot=0 id_task=615 n_ctx=32768 n_past=33 n_system_tokens=0 n_cache_tokens=33 truncated=false
    INFO [ slots_idle] all slots are idle | tid="136757751341056" timestamp=1774367697
    slot create_check: id 0 | task 615 | created context checkpoint 2 of 32 (pos_min = 31, pos_max = 31, size = 62.814 MiB, took 62.15 ms)
    INFO [ log_server_request] request | tid="136751500771328" timestamp=1774367698 remote_addr="192.168.1.19" remote_port=54803 status=200 method="OPTIONS" path="/v1/chat/completions" params={}
    ======== Prompt cache: cache size: 33, n_keep: 0, n_discarded_prompt: 0, cache_ram_n_min: 0, f_keep: 0.39, cache_ram_similarity: 0.50
    updating prompt cache
• saving prompt with length 33, total state size = 63.439 MiB
  prompt cache save took 141.40 ms
• looking for better prompt, base f_keep = 0.394, sim = 0.151, n_keep = 0, n_discarded_prompt = 0
• cache state: 2 prompts, 389.750 MiB (limits: 8192.000 MiB, 0 tokens, 13872 est)
  • prompt 0x5ef6a6d8fb20: 627 tokens, 0 discarded, checkpoints: 2, 200.682 MiB
  • prompt 0x5ef6a6db04b0: 33 tokens, 0 discarded, checkpoints: 2, 189.068 MiB
    prompt cache load took 0.17 ms
    banned strings is not supported by recurrent model, it will be disabled.
    INFO [ launch_slot_with_task] slot is processing task | tid="136757751341056" timestamp=1774367698 id_slot=0 id_task=620
    ======== Cache: cache_size = 33, n_past0 = 13, n_past1 = 13, n_past_prompt1 = 13, n_past2 = 13, n_past_prompt2 = 13
    slot apply_checkp: id 0 | task 620 | n_past = 13, slot.prompt.tokens.size() = 33, seq_id = 0, pos_min = 31
    slot apply_checkp: id 0 | task 620 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see kv-cache : add SWA support ggml-org/llama.cpp#13194 (comment))
    slot apply_checkp: id 0 | task 620 | erased invalidated context checkpoint (pos_min = 24, pos_max = 24, size = 62.814 MiB)
    slot apply_checkp: id 0 | task 620 | erased invalidated context checkpoint (pos_min = 31, pos_max = 31, size = 62.814 MiB)
    INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="136757751341056" timestamp=1774367698 id_slot=0 id_task=620 p0=0
    slot create_check: id 0 | task 620 | created context checkpoint 1 of 32 (pos_min = 133, pos_max = 133, size = 62.815 MiB, took 60.51 ms)
    INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="136757751341056" timestamp=1774367699 id_slot=0 id_task=620 p0=134
    slot print_timing: id 0 | task 620 |
    prompt eval time = 504.08 ms / 139 tokens ( 3.63 ms per token, 275.75 tokens per second)
    eval time = 1611.24 ms / 128 tokens ( 12.59 ms per token, 79.44 tokens per second)
    total time = 2115.33 ms / 267 tokens
    INFO [ log_server_request] request | tid="136751500771328" timestamp=1774367700 remote_addr="192.168.1.19" remote_port=54803 status=200 method="POST" path="/v1/chat/completions" params={}
    INFO [ release_slots] slot released | tid="136757751341056" timestamp=1774367700 id_slot=0 id_task=620 n_ctx=32768 n_past=267 n_system_tokens=0 n_cache_tokens=267 truncated=false
    INFO [ slots_idle] all slots are idle | tid="136757751341056" timestamp=1774367700
    slot create_check: id 0 | task 620 | created context checkpoint 2 of 32 (pos_min = 265, pos_max = 265, size = 62.816 MiB, took 60.75 ms)

[ikawrakow]

So, we see that the split between the GPUs works fine, so something else goes wrong. My hypothesis that is overflow due to the f16 tensors is not correct if it also happens with the Q8_0 model. In that case I have no idea why it works with split mode graph and does not with split mode layer.

My suggestion: with a Volta GPU take the win when it works. When it doesn't, just use llama.cpp.

[Ph0rk0z]

IIRC, volta has no DP4a, poor int8 support and had some stuff going on with flash attention in mainline where it needed special handling. 32gb volta cards are now being liquidated like the P40 was which is why people are going to start showing up with them.

There's SXM servers that can have up to 8 also SXM to PCIE adapters. Former has fast link between GPUs. All of them have pretty decent memory speeds since they are HBM.

[firecoperana]

If it is layer split issue, you can use --tensor-split to assign fixed amount of layers to each GPU. I tested with my two volta gpus, and it ran just fine. It might be a model issue, because I have unsloth's first uploaded version of Qwen3.5-35B-A3B-UD-Q4_K_XL. Huggingface has commit history where you can download their old models.

[magikRUKKOLA]

There's SXM servers that can have up to 8 also SXM to PCIE adapters.

32GBx8 ? Ha. Can you point the exact model so I can check the specs?

[EDIT]: NVIDIA Tesla V100 32GB SXM3? NVIDIA Tesla V100 32GB HBM2 SXM3?

And there are some motherboards with 8 HBM2 slots? Or one would have to use the PCIe converter?

Well, these GPUs are cheap. One cost around the same as RTX 3090. So 8 of them will have the same VRAM as [~11] x 3090. How much is the motherboard?

[Ph0rk0z]

here is an example: https://www.ebay.com/itm/168230833634 but there are other SXM servers. Keep in mind these idle like a mofo from what I saw.

[magikRUKKOLA]

@Ph0rk0z

Keep in mind these idle like a mofo from what I saw.

Idle? Meaning not as performant as RTX 3090?

[Ph0rk0z]

Meaning they are like P100's power management except idle power draw is probably 30-40w per GPU. Sitting, doing nothing.

[magikRUKKOLA]

@netqer

Hello! Can you run:

/usr/share/doc/nvidia-cuda-toolkit/examples/bin/x86_64/linux/release/p2pBandwidthLatencyTest

? you got PCIe or HBM2?

[netqer]

@netqer

Hello! Can you run:

/usr/share/doc/nvidia-cuda-toolkit/examples/bin/x86_64/linux/release/p2pBandwidthLatencyTest

? you got PCIe or HBM2?

No such file or directory

[magikRUKKOLA]

@netqer

Its from cuda-toolkit. Do you have it?

apt search cuda-toolkit | grep installed

[EDIT]: Apparently its over here: https://github.com/NVIDIA/cuda-samples

[Ph0rk0z]

In my case I had to d/l and compile the examples. Never saw it installed by default. Also NCCL tests like AlltoAll

[magikRUKKOLA]

@Ph0rk0z

Woops. Yeah, you're right. Its here: https://github.com/NVIDIA/cuda-samples