Bug: iqk/fa/iqk_fa_templates.h:1175: GGML_ASSERT(S > 0) failed

[rofl0r]

What happened?

after a while, llama-server crashes with the above assert. i compiled RelWithDebInfo to be able to debug it and ran it in gdb, however some odd stuff happens (it appears ik_llama has its own signal handler which tries to ptrace, which fails), so i cannot get a backtrace.

ik_llama.cpp/ggml/src/./iqk/fa/iqk_fa_templates.h:1175: GGML_ASSERT(S
 > 0) failed
[Detaching after fork from child process 1519249]
[Detaching after fork from child process 1519250]
Could not attach to process.  If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
warning: process 1515739 is already traced by process 1515725
ptrace: Operation not permitted.
No stack.
The program is not being run.

Thread 1659 "llama-server" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffb8b762640 (LWP 1517399)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140718353294912) at ./nptl/pthread_kill.c:44
44  ./nptl/pthread_kill.c: No such file or directory.
(gdb) Could not attach to process.  If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
warning: process 1515739 is already traced by process 1515725
ptrace: Operation not permitted.
No stack.
The program is not being run.

Name and Version

/ik_llama.cpp[main]$ build/bin/llama-cli --version
version: 4551 (3f40e73)
built with cc (Ubuntu 11.4.0-1ubuntu122.04.3) 11.4.0 for x86_64-linux-gnu

What operating system are you seeing the problem on?

Linux

Relevant log output

command line used for llama-server is:

build/bin/llama-server --model ~/models/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled.IQ4_XS.gguf --spec-type mtp --draft-max 3 --draft-p-min 0.0 --spec-autotune -cnv --color --jinja --special -smgs -sas -mea 256 --temp 0.7 -t 6 --parallel 6 --cpu-moe --merge-up-gate-experts --flash-attn on --mla-use 3 --mlock --run-time-repack --no-kv-offload --host 127.0.0.1 --port 18081 --timeout 36000 --reasoning-budget 2048

it's a CPU-only build (no GPU exists).

last lines before crash is:

INFO [    batch_pending_prompt] kv cache rm [p0, end) | tid="140737325548096" ti
mestamp=1780318796 id_slot=2 id_task=20748 p0=0
llama_decode_internal: qwen3next mixed-sequence batch contains repeated seq_id v
alues; falling back to single-token chunking
[New Thread 0x7ffa1663a640 (LWP 1519244)]
[New Thread 0x7ff9f0b8c640 (LWP 1519245)]
[New Thread 0x7ff9f038b640 (LWP 1519246)]
[New Thread 0x7ff9efb8a640 (LWP 1519247)]
[New Thread 0x7ffa16e3b640 (LWP 1519248)]
[Thread 0x7ffa1663a640 (LWP 1519244) exited]
[Thread 0x7ff9efb8a640 (LWP 1519247) exited]
[Thread 0x7ff9f038b640 (LWP 1519246) exited]
[Thread 0x7ff9f0b8c640 (LWP 1519245) exited]
[Thread 0x7ffa16e3b640 (LWP 1519248) exited]

[mb8565]

Same assert here, on a different model and a config that I think widens the trigger.

iqk/fa/iqk_fa_templates.h:1175: GGML_ASSERT(S > 0) failed same build (4551 / 3f40e73) and same line as you. Mine is GPT-OSS-120B Q4_K_M, and the part worth flagging: I'm running -np 1 (single slot), so this isn't only a concurrency thing. It fires on plain prefix reuse, a new request shares a token prefix with the slot's cached content, the server logs kv cache rm [p0, end), and the next decode asserts. Your "mixed-sequence batch contains repeated seq_id values; falling back to single-token chunking" looks like the same thing seen from the batch-assembly side.

Environment

• ik_llama.cpp build 4551 (3f40e73)
• Built with gcc (Debian) 14.2.0
• CMAKE_BUILD_TYPE=Release, GGML_NATIVE=ON, GGML_CUDA=OFF. Worth noting: Release still trips it GGML_ASSERT is a hard abort, not gated by build type, so you don't need a debug build to reproduce.
• CPU-only: 2× Intel Xeon E5-4669 v4 (Broadwell), 88 threads. SIMD: AVX2 / FMA / F16C, no AVX512 so the AVX2 FA path hits it too.
• OS: Debian 13 (trixie).
• Server flags: --ctx-size 65536 -np 1 --no-mmap --cache-ram 0 --flash-attn on --batch-size 4096 --ubatch-size 4096 --threads 22 --threads-batch 44.

It's been around a while and the line just tracks the code (it was iqk_fa_templates.h:1157 on build 4411). Models I've personally hit it on: GPT-OSS-120B (single-slot prefix-reuse and -np 2), GLM-4.7-Flash (-np 2), Gemma4-MoE (repeated same-slot requests). It's the kernel, not the model.

[ikawrakow]

It's been around a while and the line just tracks the code (it was iqk_fa_templates.h:1157 on build 4411). Models I've personally hit it on: GPT-OSS-120B (single-slot prefix-reuse and -np 2), GLM-4.7-Flash (-np 2), Gemma4-MoE (repeated same-slot requests). It's the kernel, not the model.

Me, on the other hand, have never hit it on any of those models

It is not the kernel. If anything, something goes wrong with the attention mask (i.e., KV cache handling), and we end up with an empty mask. Hitting the assert with a not-empty mask and no NaNs in the self-attention input is impossible.

[mb8565]

Agree on the mechanism: the row comes back fully masked, the kernel accumulates zero blocks, S stays 0, and that is what the assert catches. So the KV cache and mask bookkeeping, not the kernel.

I could not reduce this to a minimal retest (a plain two-request prefix overlap trims cleanly and does not assert), so here is the real crash. GPT-OSS-120B, build 4551 / 3f40e73, CPU AVX2, -np 1. A 13858 token cache, then a re-plan that shares a 5293 prefix and then diverges:

Cache: n_past0 = 5293, n_past1 = 5293, n_past2 = 5294
kv cache rm [p0, end) id_task=11299 p0=5293
iqk_fa_templates.h:1175: GGML_ASSERT(S > 0) failed (on every OMP thread)

#4 ggml_abort
#5 FlashQKV<64, 8, 64>::normalize_and_store_1row<FlashMS<8, 64> >
#6 compute_helper<64, 64, 64, 128, HelperF16, HelperF16, ...>
#9 iqk_flash_attn_noalibi
#10 ggml_compute_forward_flash_attn_ext_f16
#19 llama_decode
#20 server_context::process_batch_tokens / update_slots

It aborts in normalize_and_store_1row with S=0, consistent with the fully masked row. Also n_past2 is one ahead of n_past0/1 here (5294 vs 5293); in a run that does not crash the three are equal. That one token skew, right after the partial trim, is my best guess for where the masked row comes from, and it only appears after the rm.

At -np 1 the mask_effective_nk1() union scan from #1880 has a single slot to reconcile, so I do not think it reaches this single fully masked row case.

Workaround: --flash-attn off avoids the iqk path (--cache-ram 0 does not help). Happy to pull a coredump or the full per-layer n_past if that would help narrow it down.

Build: 4551 (3f40e73), Release, GGML_CUDA=OFF, 2x E5-4669 v4 (AVX2, no AVX512), Debian 13.

[ikawrakow]

It looks like this off-by-one is causing an empty mask for one token. The solution would be to fix that. It would be useful if you could provide a simple reproduction that we can use.

[ikawrakow]

If you run with #1923, can you trigger the empty mask error?

[mb8565]

Reproduced it reliably. Our real planner workload fires it on the first attempt, FA on, -np 1: the architecture prompt grows the cache to ~15K tokens, then a re-decomposition request shares that long prefix and diverges. (A generic synthetic with the same cache-count state never fired, so the real token geometry is what matters.)

What I directly observe: after kv cache rm [p0=10463, end) on the divergent request, GGML_ASSERT(S > 0) fires at iqk_fa_templates.h:1175. That is a query row with S=0, an empty mask for that row, consistent with your diagnosis. Backtrace, identical to our production crash:

    normalize_and_store_1row <- iqk_flash_helper <- iqk_flash_attn_noalibi <- ggml_compute_forward_flash_attn_ext_f16 <- llama_decode

#1923's "found empty attention mask" did not print (0 hits, no nek1/first_k): the backtrace runs through iqk_flash_helper (the general path), but #1923's check is in the neq1==1 SWA decode fast-path, so it sits off this crash's path. (The reprocessed prefill batch with neq1 > 1 is my read of why; the backtrace plus the absent print are the evidence.)

Scoping note: this fire had n_past0 = n_past1 = n_past2 = 10463 (no cache-count skew) and still crashed, while the production crash had n_past2 one ahead. So that skew is a correlate; the common cause in both is the empty-masked row from the kv cache rm.

The repro uses our pipeline prompts and needs a gpt-oss-class model (interleaved SWA); gpt-oss-20b should hit the same path. Happy to share the script and full backtrace.

[rofl0r]

in case it helps, i've been using hermes agent which sends a huge system prompt. attached is a plaintext network trace from wireshark's "follow tcp stream".

hermes_convo.txt

[coreyoconnor]

I've encountered this with #1923 and mainline. I'm using

llama-server --model unsloth-gemma-4-26B-A4B-it-qat/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --spec-autotune \
                                      -cnv --color --jinja --special \
                                      -sm layer -smgs -sas -mea 256 --split-mode-f32 \
                                      --temp 0.7 -t 8 --parallel 4 \
                                      --cpu-moe --merge-up-gate-experts \
                                      --flash-attn on --mla-use 3 \
                                      --mlock --run-time-repack --no-kv-offload --log-enable

See https://point.free/blog/gemma-4-on-a-2016-xeon/ for how I got those args.

Also encountered using the base google gemma-4-26B-A4B-it model.

[coreyoconnor]

Smaller command that reproduces the assertion.

llama-server --model unsloth-gemma-4-26B-A4B-it-qat/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --spec-autotune \                                                                          
                                          -cnv --color --jinja --special \                                                                                                                                                                                                                    
                                          --temp 0.7 -t 8 --parallel 4 \                                                                                                                                                                                                                      
                                          --cpu-moe --merge-up-gate-experts \                                                                                                                                                                                                                 
                                          --flash-attn on --mla-use 3 \                                                                        
                                          --run-time-repack --no-kv-offload

[mb8565]

@coreyoconnor you ran #1923 and still saw the assert. Did the "found empty attention mask: nek1=..., first_k=..." line print for you, or did it go straight to the 1175 assert with no extra message? You are on Gemma-4 (SWA) and #1923's check is in the neq1==1 SWA-decode fast path. On my GPT-OSS crash that print never fired (the crash comes through the general prefill helper, iqk_flash_helper), so #1923 was off my path. If your path does hit it, you would have the nek1/first_k values ikawrakow asked for in this thread.

[ikawrakow]

Can you test #1935 and let me know if the assert persists? Thanks.

[coreyoconnor]

If your path does hit it, you would have the nek1/first_k values ikawrakow asked for in this thread.

Shoot. I do not have the output from that run still. I will try #1935 first and then #1923 again after.

[coreyoconnor]

For #1935 I ran into a crash that looks to be different. This seems to be a context shift failure:

INFO [           context_shift] slot context shift | tid="139346420570496" timestamp=1780944852 id_slot=0 id_task=6180 n_keep=1 n_left=65534 n_discard=32767 n_ctx=262144 n_past
=65535 n_system_tokens=0 n_cache_tokens=65535                                                                                                                                   
/build/source/ggml/src/ggml.c:20128: /build/source/ggml/src/ggml.c:20128: /build/source/ggml/src/ggml.c:20128: GGML_ASSERT(n_dims <= ne0) failed                                
/build/source/ggml/src/ggml.c:20128: GGML_ASSERT(n_dims <= ne0) failed                                                                                                          
/build/source/ggml/src/ggml.c:20128: GGML_ASSERT(n_dims <= ne0) failed                                                                                                          
GGML_ASSERT(n_dims <= ne0) failed                                                                                                                                               
GGML_ASSERT(n_dims <= ne0) failed/build/source/ggml/src/ggml.c:20128: /build/source/ggml/src/ggml.c:20128: GGML_ASSERT(n_dims <= ne0) failed
GGML_ASSERT(n_dims <= ne0) failed                                                                                                                                               
/build/source/ggml/src/ggml.c:20128: GGML_ASSERT(n_dims <= ne0) failed                                                                                                          

Haven't encountered the assert in iqk_fa_templates.h yet

[coreyoconnor]

Double shoot. I looked at the prior failure and it was also a context shift error. I think that was #1923 . But it's not clear from my logs.

[mb8565]

Tested #1935 on our case (the GPT-OSS-120B Q4_K_M planner crash, CPU, -np 1, no speculative decoding). The assert is gone.

I built #1935 and ran our faithful repro: a long architecture-plan call grows the cache to ~12-15K tokens, then a re-decompose call shares a long prefix and diverges, which fires the kv cache rm prefix trim that crashed the old build. Three runs back to back, all clean. The trim fired 20 times across the runs (one at p0=11519), with no GGML_ASSERT(S>0), no ggml_abort, and no "found empty attention mask". On our unpatched build (3f40e73) the same repro hits the assert.

So "disable mask optimization" resolves the -np 1 re-plan-trim path too, not just the parallel + spec-decode configs the others reported. Thanks for digging into this.