What happened?
After recent commits (will try to bissect but it takes about 30 min to load model, so this is kinda hard to do - preliminary it's autoparser's commit) Kimi 2.6 (using this ubergarm's quant) started to crush after prompt processing:
INFO [ release_slots] slot released | tid="124898033876992" timestamp=1777209274 id_slot=0 id_task=0 n_ctx=153600 n_past=1120 n_system_tokens=0 n_cache_tokens=1120 truncated=false
INFO [ slots_idle] all slots are idle | tid="124898033876992" timestamp=1777209274
INFO [ log_server_request] request | tid="124451154210816" timestamp=1777209274 remote_addr="192.168.0.61" remote_port=58336 status=200 method="POST" path="/v1/chat/completions" params={}
INFO [ log_server_request] request | tid="124451145818112" timestamp=1777209363 remote_addr="192.168.0.61" remote_port=35352 status=200 method="HEAD" path="/" params={}
======== Prompt cache: cache size: 0, n_keep: 0, n_discarded_prompt: 0, cache_ram_n_min: 0, f_keep: 0.00, cache_ram_similarity: 0.50
INFO [ launch_slot_with_task] slot is processing task | tid="124898033876992" timestamp=1777209378 id_slot=1 id_task=981
======== Cache: cache_size = 0, n_past0 = 0, n_past1 = 0, n_past_prompt1 = 0, n_past2 = 0, n_past_prompt2 = 0
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209378 id_slot=1 id_task=981 p0=0
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209392 id_slot=1 id_task=981 p0=2048
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209401 id_slot=1 id_task=981 p0=4096
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209405 id_slot=1 id_task=981 p0=6144
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209417 id_slot=1 id_task=981 p0=8192
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209428 id_slot=1 id_task=981 p0=10240
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209439 id_slot=1 id_task=981 p0=12288
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209450 id_slot=1 id_task=981 p0=14336
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209461 id_slot=1 id_task=981 p0=16384
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209472 id_slot=1 id_task=981 p0=18432
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209484 id_slot=1 id_task=981 p0=20480
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209495 id_slot=1 id_task=981 p0=22528
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209507 id_slot=1 id_task=981 p0=24576
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209518 id_slot=1 id_task=981 p0=26624
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209529 id_slot=1 id_task=981 p0=28672
INFO [ batch_pending_prompt] kv cache rm [p0, end) | tid="124898033876992" timestamp=1777209540 id_slot=1 id_task=981 p0=30720
begin: ngram_mod occupancy = 30202/4194304 (0.01)
=============================== Failed to sample token
Data has been stored in probabilities.txt
Create an issue with full log and attach probabilities.txt to the issue
Crashing now
/home/drros/ik_llama.cpp/src/llama-sampling.cpp:744: Fatal error
^CAborted (core dumped)
content of probabilities.txt:
candidates->size: 40
max = nan
sump = nan
r = 3567895402
probabilities:
0 38 nan nan
1 22 nan nan
2 10 nan nan
3 34 nan nan
4 26 nan nan
5 18 nan nan
6 20 nan nan
7 4 nan nan
8 24 nan nan
9 12 nan nan
10 32 nan nan
11 28 nan nan
12 16 nan nan
13 36 nan nan
14 8 nan nan
15 39 nan nan
16 9 nan nan
17 21 nan nan
18 1 nan nan
19 23 nan nan
20 11 nan nan
21 25 nan nan
22 5 nan nan
23 27 nan nan
24 13 nan nan
25 31 nan nan
26 29 nan nan
27 15 nan nan
28 33 nan nan
29 7 nan nan
30 35 nan nan
31 17 nan nan
32 37 nan nan
33 3 nan nan
34 19 nan nan
35 0 nan nan
36 2 nan nan
37 6 nan nan
38 14 nan nan
39 30 nan nan
command I'm running with: ./build/bin/llama-server --model /mnt/ds1nfs/codellamaweights/kimi-k-2-6-smol-iq3/kimi-k-2-6-smol-iq3/Kimi-K2.6-smol-IQ3_KS-00001-of-00010.gguf --merge-qkv --ctx-size $((150*1024)) -mla 3 -amb 512 --parallel 2 -b 2048 -ub 2048 --host 0.0.0.0 --port 30000 --no-mmap --jinja --mlock -ngl 999 -ncmoe 55 --chat-template-file ~/kimi-k-2-6.template2 --spec-type ngram-mod --spec-ngram-size-n 24 --draft-min 4 --draft-max 64 --spec-autotune -ctk q8_0 -ctv q8_0 -khad -vhad -cram 16384 -ts 100,106,100 --temp 1.0 --top-p 0.95
Steps to reproduce: start server, use claude code with it, after first prompt it fails, or use long enough message in web interface.
Hardware: AMD Epyc 9274f \ 384Gb 4800 mt\s ddr5 \ 3*RTX PRO 4000 Blackwell
Name and Version
./build/bin/llama-cli --version
version: 4437 (3a945af)
built with cc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 for x86_64-linux-gnu
What operating system are you seeing the problem on?
Linux
Relevant log output
What happened?
After recent commits (will try to bissect but it takes about 30 min to load model, so this is kinda hard to do - preliminary it's autoparser's commit) Kimi 2.6 (using this ubergarm's quant) started to crush after prompt processing:
content of probabilities.txt:
command I'm running with:
./build/bin/llama-server --model /mnt/ds1nfs/codellamaweights/kimi-k-2-6-smol-iq3/kimi-k-2-6-smol-iq3/Kimi-K2.6-smol-IQ3_KS-00001-of-00010.gguf --merge-qkv --ctx-size $((150*1024)) -mla 3 -amb 512 --parallel 2 -b 2048 -ub 2048 --host 0.0.0.0 --port 30000 --no-mmap --jinja --mlock -ngl 999 -ncmoe 55 --chat-template-file ~/kimi-k-2-6.template2 --spec-type ngram-mod --spec-ngram-size-n 24 --draft-min 4 --draft-max 64 --spec-autotune -ctk q8_0 -ctv q8_0 -khad -vhad -cram 16384 -ts 100,106,100 --temp 1.0 --top-p 0.95Steps to reproduce: start server, use claude code with it, after first prompt it fails, or use long enough message in web interface.
Hardware: AMD Epyc 9274f \ 384Gb 4800 mt\s ddr5 \ 3*RTX PRO 4000 Blackwell
Name and Version
./build/bin/llama-cli --version
version: 4437 (3a945af)
built with cc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 for x86_64-linux-gnu
What operating system are you seeing the problem on?
Linux
Relevant log output