[Feature Request] Add host-memory prompt caching (Context caching in RAM)

[Pento95]

On Oct 9, 2025 this llama.cpp pull request has been merged in mainline: ggml-org#16391

I think it would be extremely usefull to add it to koboldcpp! expecially, having a GUI option would make a lot of sense.

What is the problem?

Currently, KoboldCpp faces performance bottlenecks in two key scenarios when the KV cache must be dropped and rebuilt:

• Multi-tasking in Front-ends (e.g., SillyTavern, lite..): When a user switches from a long chat conversation to a different task (like "generate image prompt for SD"), the server must clear the main chat's KV cache. When the user returns to the chat, the entire chat history must be re-processed from scratch, causing a long wait.

• AI Horde Worker Inefficiency: As a worker, KoboldCpp processes jobs from many different users. Each new job likely has a completely different context, forcing the server to constantly drop and rebuild the KV cache. This leads to high "ingest" times and reduces the worker's overall throughput and efficiency.

What is the proposed solution?

The llama.cpp backend has implemented a native feature (server_prompt_cache) that solves this exact problem. It functions as a server-side "Context Shift" or KV cache manager.

This feature allows the server to intelligently cache multiple KV states in memory.

What are the benefits of adding this?

• Instant Context Switching (SillyTavern): For front-end users, this would make the experience seamless. The server could cache the main chat state, process the image prompt, and then instantly restore the chat state, eliminating re-processing delays.

• Massively Improved AI Horde Performance: This is a major benefit for Horde workers. The server could hold multiple user contexts in the cache. When a new job arrives that matches a cached context (e.g., a follow-up request from the same user), the worker can skip the entire processing/ingest phase and begin generation immediately. This would dramatically increase the worker's job speed and efficiency.

• Better Resource Utilization: Instead of wasting cycles constantly re-processing prompts, the server can use its VRAM and memory to hold ready-to-use contexts.

how does it work
use 8192 MiB of host RAM for caching prompts (default value)
-cram 8192

use as much host RAM is available (i.e. no limit)
-cram -1

disable prompt caching in RAM
-cram 0

This is a high-demand feature in other backends as well, like in Ollama Issue Tracker: ollama/ollama#8577

[LostRuins]

Prompt caching actually does exist in koboldcpp, under the admin endpoint. The problem is when to trigger it and when to automatically switching back to the correct "cache slot". This is probably doable heuristically but would require some deep diving to figure out

[Pento95]

I'd like to clarify my original request after doing more research into the codebase.

I initially didn't realize that the prompt caching feature (server_prompt_cache with -cram parameter) is only implemented in llama-server (tools/server/server.cpp), not in the core llama.cpp library.

Since KoboldCpp uses the core llama.cpp library directly (via expose.cpp/gpttype_adapter.cpp) rather than running llama-server as a separate process, the RAM-based prompt caching functionality isn't currently available to KoboldCpp's architecture.

The existing /api/admin/save_state and /load_state endpoints save KV cache to disk, which would be too slow for the use case I described (multi-tasking between conversations, AI Horde workers switching between users). With prompts that can be 1-2GB each, disk I/O would create significant overhead even on NVMe SSDs, and loading them back into RAM for similarity matching would negate the performance benefits.

Ideal solution: In-RAM prompt caching with automatic context switching, similar to what llama-server's server_prompt_cache` provides.

Possible implementation paths:

1. Port server_prompt_cache logic from the server into the core llama.cpp library
2. Implement a lighter-weight RAM-based cache manager in KoboldCpp's Python layer

Would any of these approaches align with KoboldCpp's design philosophy?
I'm happy to contribute if there's a preferred direction!

[LostRuins]

They don't actually cache to disk, it's still written to ram. You can give it a try by enabling admin mode and then triggered a save then load state. Nothing is written to disk

[Pento95]

Hi @LostRuins, thank you for the detailed explanation and my apologies for the noise and the mistake in where the kv cache state is written with the admin setting.

I did end up posting a comment on the llama.cpp discussion asking if they would consider moving their new server_prompt_cache logic into the core libllama.
It's almost certainly considered application-layer logic and is unlikely to happen, but I figured it was worth asking.

In the (very likely) event the llama.cpp team confirms this logic belongs in the application-layer and won't be moved to libllama, perhaps we could still use their implementation as inspiration.

I noticed you mentioned:

"The problem is when to trigger it and when to automatically switching back to the correct "cache slot". This is probably doable heuristically but would require some deep diving to figure out"

Since their llama-server implementation has already solved this exact heuristic problem (deciding which prompt matches which cache, etc.), we could potentially adapt their logic to implement this feature on Koboldcpp. It might save us from having to figure out that complex switching logic from scratch.

[LostRuins]

I wonder how they determine when to update each of the caches.

[Pento95]

You're right to question how they determine cache updates. That's the exact issue. My original suggestion to adapt llama-server's logic was flawed, because its heuristic is built on a foundation that doesn't work for our primary use cases.

The "Prefix Matching" Trap

llama-server's logic (in -cram) is based on Strict Prefix Matching. It only swaps a cache if the new prompt starts with the same N tokens as a cached prompt.

This fails in both of our key scenarios:

• AI Horde (Truncation): A client truncates the start of the context. The prefix breaks. llama-server sees a full cache miss and re-prefills 16k tokens.

• SillyTavern (Task Switching): A user switches from their "Main Chat" to a separate "Image Gen Prompt" task. This is functionally a new "user." When they switch back, the prefix is broken, and llama-server again forces a full re-prefill.

In both cases, their heuristic is unusable for us.

The Ideal Solution: A Two-Layer "Smart Cache"

The optimal solution needs to combine KoboldCpp's existing "diff" logic with a new, smarter RAM swapping system.

• Layer 1: "Smart Swap" (RAM Cache - The Filing Cabinet): This handles the coarse change between different contexts (e.g., User A vs. User B, or Main Chat vs. Image Gen).

• Layer 2: "Smart Shift" (VRAM Cache - The Active Desk): This is what KoboldCpp already does brilliantly, handling truncation within a single, active context.

Example Workflow

Here’s how this hybrid system would work in practice:

1. User is in "Main Chat".

• VRAM ("Active Desk") holds the "Main Chat" cache.
• RAM ("Filing Cabinet") is empty.

2. User switches to "Image Gen" task. (This is functionally the same as a new AI Horde user arriving).

• Layer 1 (Smart Swap): The server sees this is not a continuation/shift.
• Action: It hibernates the "Main Chat" cache (VRAM -> RAM). It then loads/prefills the "Image Gen" context into VRAM.
• Status: VRAM = "Image Gen", RAM = "Main Chat".

3. User switches back to "Main Chat". (This is the critical step).

• Layer 1 (Smart Swap): The server sees this is not a continuation of "Image Gen".
• Action: It hibernates "Image Gen" (VRAM -> RAM).
• Crucially (Delta # 2): It now runs a "diff" (the Smart Context logic) on the new prompt against all caches stored in the RAM "Filing Cabinet".
• Cache Hit! It finds a >90% match with the "Main Chat" cache.
• Action: It "thaws" (restores) the "Main Chat" cache (RAM -> VRAM).

4, Final Step:

• Layer 2 (Smart Shift): The server (now with the correct cache loaded into VRAM) executes its normal "Smart Context" logic to process the few new or truncated tokens from the previous turn.

This two-layer system would give us the best of all worlds: near-instant switching between users/tasks (Layer 1) and efficient handling of truncation within those tasks (Layer 2).

Next Steps

I will fork the project and attempt a draft implementation of this "Layer 1 Smart Swapping" logic. If I can get a functional proof-of-concept, I will open a draft PR for discussion.

(yes, i used an LLM to help me write this, my english sucks)