Parallel llama-server crashes when prompt cache/context checkpoints are enabled

[JakeATX]

Summary

I’m testing the feature/turboquant-kv-cache branch locally and found a reproducible llama-server parallel-serving crash that appears tied to prompt cache / context checkpoint handling rather than TurboQuant V-cache alone.

The key clue: a q8/q8 control also crashes with prompt cache/checkpoints enabled, while q8/turbo3 passes at both --parallel 2 and --parallel 3 when prompt cache/checkpoints are disabled.

Dataset/report with raw logs, telemetry, summaries, and charts:

https://huggingface.co/datasets/sjakek/nex-n2-mini-tq3-eval/tree/main/parallel_smoke_60k

Environment

• Repo: TheTom/llama-cpp-turboquant
• Branch: feature/turboquant-kv-cache
• Local commit: 35ac80d55 (Merge pull request #182 from TheTom/tom/catchup-from-feature)
• Binary: build-cuda/bin/llama-server
• GPU: RTX 3090 Ti
• CPU: Ryzen 7 5800X3D
• Model: Nex-N2-mini-UD-Q3_K_XL.gguf

Base server shape:

llama-server \
  -m Nex-N2-mini-UD-Q3_K_XL.gguf \
  --host 127.0.0.1 --port 8080 \
  -t 8 -tb 8 \
  -c 61440 \
  -b 2048 -ub 512 \
  -ngl 99 \
  -fa on \
  --parallel 2 \
  --jinja --reasoning off --no-webui

Requests were concurrent /completion calls with about 1K prompt tokens and n_predict=1000 per stream, temperature=0, top_k=1, ignore_eos=true, timings_per_token=true, cache_prompt=false in the request payload.

Repro matrix

All tests used total context -c 61440, with slot ctx split by --parallel.

Treatment	Parallel	Result	Notes
-ctk q8_0 -ctv q8_0 + prompt cache/checkpoints	2	crash	CUDA error
-ctk q8_0 -ctv q8_0 + prompt cache/checkpoints	3	crash	CUDA error
-ctk q8_0 -ctv turbo3 + prompt cache/checkpoints	2	crash	CUDA error
-ctk q8_0 -ctv turbo4 + prompt cache/checkpoints	2	crash	CUDA error
q8/turbo3 + --no-cont-batching but cache/checkpoints still on	2	crash	CUDA error
q8/turbo3 + -b 512 -ub 128 but cache/checkpoints still on	2	crash	CUDA error
q8/turbo3 + -fa off but cache/checkpoints still on	2	crash	CUDA error
q8/turbo3 + prompt cache/checkpoints disabled	2	pass	~196.7 aggregate decode tok/s
q8/turbo3 + prompt cache/checkpoints disabled	3	pass	~215.9 aggregate decode tok/s
conservative: q8/turbo3, no prompt cache/checkpoints, no cont batching, -b 512 -ub 128	2	pass	slower prefill
conservative: same	3	pass	slower prefill, ~419.7 aggregate decode tok/s

The stable flags were:

--no-cache-prompt --cache-ram 0 --ctx-checkpoints 0 --no-cache-idle-slots

Failure signature

The server exits quickly after concurrent requests start. The client sees:

RemoteDisconnected('Remote end closed connection without response')

Representative server log excerpt from a larger parallel run:

E cmd_child_to_router:error:/home/jake-k/TheTom-llama-cpp-turboquant/ggml/src/ggml-cuda/ggml-cuda.cu:105: CUDA error
CUDA error: invalid argument
E   current device: 0, in function ggml_cuda_kernel_launch at /home/jake-k/TheTom-llama-cpp-turboquant/ggml/src/ggml-cuda/common.cuh:1644
E   cudaGetLastError()

In the 60K smoke matrix, failures happen very early, before successful completion timings are returned. VRAM is not close to exhausted; peak total GPU usage is around 17.1-17.5 GiB on a 24 GiB card.

Why I think this is prompt-cache/checkpoint related

• q8/q8 fails with prompt cache/checkpoints enabled, so this does not look specific to turbo3.
• q8/turbo3 succeeds at p2 and p3 when prompt cache, cache RAM, idle-slot cache, and ctx checkpoints are all disabled.
• Toggling continuous batching, smaller batch/ubatch, or Flash Attention off did not fix it while prompt cache/checkpoints remained enabled.
• Earlier larger-context failures occurred around overlapping prompt processing / checkpoint creation.

Artifacts

Raw data and logs are in the HF dataset folder:

• parallel_smoke_matrix.csv
• parallel_smoke_matrix_compact.csv
• per-treatment server.log
• per-treatment telemetry.csv/json
• per-stream completion result JSON
• charts for pass/fail, VRAM, decode throughput, and p3 timeline

Direct folder:

https://huggingface.co/datasets/sjakek/nex-n2-mini-tq3-eval/tree/main/parallel_smoke_60k

Question: is prompt cache / context checkpointing expected to be safe with llama-server --parallel N on this branch, or should parallel serving currently disable these flags? If this is intended to work, I can run a narrower follow-up matrix separating --cache-prompt, --cache-ram, --cache-idle-slots, and --ctx-checkpoints individually.