Feature Request: Compress cache-ram

[frenzybiscuit]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Is there some way to compress the cache-ram? Currently with Gemma 4 31B a 51k prompt will use 35GB of RAM to cache.

Can't we use lz4 or something to compress this down in the system RAM?

Motivation

RAM limitations.

Possible Implementation

No response

[ikawrakow]

Is this quantized cache or f16 cache? Is it per checkpoint, total with X checkpoints, or is it just to cache a prompt of 51k tokens?

[frenzybiscuit]

Is this quantized cache or f16 cache? Is it per checkpoint, total with X checkpoints, or is it just to cache a prompt of 51k tokens?

K is f16, v is q8 on context quantization.

Checkpoints are set to 8 total.

Complete command:

  "Default":
    cmd: |
      ik_llama.cpp/build/bin/llama-server --host 127.0.0.1 --port ${PORT}
      --model gguf/mero_artemis-31B-v0.3.1_hb16-q6_k.gguf
      --ctx-size 90000
      --jinja
      --cache-type-k f16
      --cache-type-v q8_0
      --split-mode graph
      --gpu-layers 200
      --flash-attn on
      --parallel 1   
      --peg
      --reasoning-budget 1280
      --reasoning on
      --reasoning-budget-message "... thinking budget exceeded, let's answer now"
      --max-gpu 4
      --ubatch-size 1024
      --batch-size 1024
      --threads 24   
      --ctx-checkpoints 8
      --cache-ram 102400
      -ts 20,20,20,15

[frenzybiscuit]

This is what I see:

 - saving prompt with length 52693, total state size = 34670.236 MiB

[ikawrakow]

Well, if the KV cache was compressible, why not just use a quantized cache?

[usrlocalben]

it would change the world economy if LZ was effective.

[usrlocalben]

I ran across this lz4 kv cache while looking at the various DSv4 impls. Claims 1.5-1.7 compression ratio. I'm not sure if slop or if I stand corrected. 😅

[saood06]

I ran across this lz4 kv cache while looking at the various DSv4 impls. Claims 1.5-1.7 compression ratio. I'm not sure if slop or if I stand corrected. 😅

Would be interesting if true. I know I did some experimenting with something very similar but with MLA cache data as part of #558 and I couldn't get compression. I wasn't using lz4, I was using zstd but it was with zstd's dictionary compression.

[ikawrakow]

I ran across this lz4 kv cache while looking at the various DSv4 impls. Claims 1.5-1.7 compression ratio. I'm not sure if slop or if I stand corrected. 😅

Would be interesting if true. I know I did some experimenting with something very similar but with MLA cache data as part of #558 and I couldn't get compression. I wasn't using lz4, I was using zstd but it was with zstd's dictionary compression.

I haven't looked very closely so perhaps I'm missing something, but as far as I can tell the KV cache is stored as f32 in DwarfStar4. See for instance here, or better, here. Given this, 1.75X compression in the best case scenario is rather disappointing. By just going to f16 or bf16 one would save a factor of 2 (and f32 -> bf16 conversion is way faster than lz4 compression).

[magikRUKKOLA]

iven this, 1.75X compression in the best case scenario is rather disappointing. By just going to f16 or bf16 one would save a factor of 2 (and f32 -> bf16 conversion is way faster than lz4 compression).

JFYI. I noticed that the BF-16 tensors of the Kimi-K2.6 are indeed getting compressed. Here is the overall stat for all the other quants (which are not getting compressed at all):

 zfs get all | grep comp
zfsdata       compressratio           1.44x                   -
zfsdata       compression             zstd-9                  local
zfsdata       refcompressratio        1.00x                   -
zfsdata/data  compressratio           1.44x                   -
zfsdata/data  compression             zstd-9                  inherited from zfsdata
zfsdata/data  refcompressratio        1.44x                   -

But the BF-16 specifically got compressed from 1.9TB to 1.1TB.

At the same time, why bother if one could use q8_0. Or q6_0. Its almost lossless for the KV-cache anyways.

[usrlocalben]

@magikRUKKOLA I may be stating the obvious but as @ikawrakow observed above the compression comes from low entropy. b/f16-as-f32, or in kimi's case, int4-as-bf16 means there will be relatively low entropy and LZ etc. will find it.

put another way, it just proves that the original width was a waste for the information represented.