Misc. bug: llama-server vram usage gradually increasing each run until OOM

[SarcasticBaka29]

Name and Version

version: 9261 (ad27757) │
built with GNU 14.2.0 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server --device CUDA0 --host 0.0.0.0 --port ${PORT} -ngl 999  -fa on  --jinja --no-mmap -cram 2048 --ctx-checkpoints 1 --no-warmup -np 1
      --model ${host_model_dir}/Qwen3.6-27B-MTP/Qwen3.6-27B-UD-Q4_K_XL.gguf
      --no-mmproj
      --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.0 
      --spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64
      --spec-type ngram-map-k4v --spec-ngram-map-k4v-size-n 16 --spec-ngram-map-k4v-size-m 24 --spec-ngram-map-k4v-min-hits 1
      --ctx-size 80000
      -n 32768
      --cache-type-k q8_0
      --cache-type-v q8_0
      --temp 0.6
      --presence-penalty 0.0
      --repeat-penalty 1.0
      --min-p 0.0
      --top-k 20
      --top-p 0.95
      --fit off
      --reasoning on
      --reasoning-budget -1
      --chat-template-kwargs '{"preserve_thinking":true}'

Problem description & steps to reproduce

Let me say upfront that I'm not even sure if this is actually a bug or normal behavior I was not aware of. I'm currently running the MTP enabled Qwen3.6-27B-UD-Q4_K_XL on a 22GB 2080TI GPU using the CUDA backend and I can just about fit 80k context which uses 21.05GB after the model is fully loaded.

The issue I'm facing is that once I start running prompts the VRAM usage will start steadily increasing until it eventually OOMs.

To give specific numbers : VRAM usage immediately jumps from 21.05 to 21.5 when launching the first prompt, then it will slowly but surely increase as tokens are being generated, reaching 21.75 at 5k tokens and hits the OOM error at around 10k tokens.

The additional VRAM is not freed after the generation fully finishes either. It just keeps adding up for every new prompt I run.b

I thought this might be an MTP related bug since I had never noticed this before and I'm aware that MTP in general requires more VRAM so I tried the non mtp version of the same Qwen3.6-27B model and although the steady increase in VRAM usage does still happen, it does tend to go up much slower.

I thought this might be related to checkpoints but setting the number of checkpoints to 1 makes no difference whatsoever.

Thanks for any help.

First Bad Commit

No response

Relevant log output

Logs

[paradigm]

I'm also seeing this. I suspect it occurred after 3a9c1b8, as I either did not run into or did not notice the issue there.

[ddh0]

I see this same issue too, I can't tell what causes it.

EDIT: I do notice that when it does happen, VRAM increases over time during token generation. But I don't know which combination of options/models/features cause the issue to appear... :/

[paradigm]

Relevant MR has merged: #23461

Unfortunately the issue remains for me.

[SarcasticBaka29]

Relevant MR has merged: #23461

Unfortunately the issue remains for me.

Same here. Just rebuilt and retested and the same behavior occurs.

[am17an]

Can you identify which parameters cause this? Perhaps remove all -spec-type and check, and then add draft-mtp and then later add ngram-mod etc. I tried reproducing this and I can't

[SarcasticBaka29]

Can you identify which parameters cause this? Perhaps remove all -spec-type and check, and then add draft-mtp and then later add ngram-mod etc. I tried reproducing this and I can't

I tried to test the various scenarios using the same Qwen3.6-27-MTP quant and my results were as follows:

• No speculative decoding of any kind : Vram usage stays pretty much the exact same after 10k tokens generated.
• Using "--spec-type draft-mtp" only : Immediate 500mb increase on the first prompt (pp buffer?) then keeps increasing relatively slowly as tokens are generated.
• Using "ngram-mod" and "ngram-map-k4v" only : initial increase of about 150mb after about 1k tokens generated followed by a steady increase at about the same rate as MTP.
• Using both speculative decoding methods together aka the worst of both worlds scenario: Getting the initial 500mb bump on pp followed by the second 150mb increase after about 1k tokens then a steady increase at a faster rate than when using each method individually.

Hope this will be helpful to you.

[rafalhao]

Hi, I'm having the same issue, somewhere along the lines of commits in the master branch, the VRAM usage shot up straight away on the first prompt, something like 400 to 500 MB, I also run out of VRAM much faster leading to OOM errors well before my context window is full.

Before I didn't have this issue, meaning launching llama-server with the same set of flags, wouldn't cause this problem. (I was using draft-mtp both before and at the moment).

I'm using Ubuntu Server 26.04 LTS RTX 3090.

If in the meantime this issue is fixed, I'll signal it here.

[paradigm]

I attempted to find the minimum set of flags which reliably reproduce the issue:

• server : free draft/MTP resources on sleep to fix VRAM leak #23461 appears to have resolved the original MTP memory leak I was running into. At the time of my original report I was under the impression I had reproduced this without --split-mode tensor.
• After initially reproducing an issue with MTP but before server : free draft/MTP resources on sleep to fix VRAM leak #23461 merged, I added in --split-mode tensor, which introduced a separate, currently known memory leak that I didn't detect due to the preexisting MTP leak covering it.
• Post-server : free draft/MTP resources on sleep to fix VRAM leak #23461, removing --split-mode tensor resolves the issue, irrelevant of MTP flags.

Thus I'm withdrawing my reported ability to reproduce a memory leak with the OP's described set of flags.

I encourage others here that reproduce the issue to follow am17an's advice and attempt to find a minimum set of flags that reproduce the issue.

[SarcasticBaka29]

Since the issue still persists and makes using any kind of speculative decoding impossible, I decided to do some more testing by switching to the vulkan backend to see if there's any difference. Unfortunately the same exact behavior is observed so whatever the issue is, it's not cuda specific.

Edit: Also tried Bartowski's Q4_K_M quant instead of unsloth's thinking it might be a quantization issue but I'm still seeing the same thing happen.

Edit 2: Further testing indicates that this issue is in some way related to kv-cache quantization. When I get rid of the --cache-type-k q8_0 and --cache-type-v q8_0 parameters (Thus having to lower my context to 45000), the initial +650mb bump in vram usage on first prompt still occurs, but after that vram does not grow significantly as tokens are generated like it did before. It only goes up by 11mb after generating 30k tokens.

So the issue I'm facing seems to perhaps stem from a combination of speculative decoding and kv-cache quantization.

[noodleprincss-ai]

I am also able to reproduce this on Windows 11 with llama.cpp build 9275 using qwen3.6-27b Q6K on an RTX 5090 with CUDA 13.1 with the following launch parameters (this is where I discovered it being an issue) - also if it matters, the GGUF I'm using is the Unsloth MTP variety (https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF?show_file_info=Qwen3.6-27B-Q6_K.gguf):

& "C:\llama.cpp\b9275\build\bin\Release\llama-server.exe" `
  --model "C:\llama.cpp\models\Qwen3.6-27B-Q6_K.gguf" `
  --mmproj "C:\llama.cpp\models\Qwen-3.6-27B-mmproj-BF16.gguf" `
  --image-min-tokens 1024 `
  --no-mmproj-offload `
  --n-gpu-layers 999 `
  --ctx-size 131072 `
  --fit off `
  --parallel 1 `
  --threads 16 `
  --temp 0.7 `
  --top-p 0.95 `
  --min-p 0.00 `
  --top-k 20 `
  --presence_penalty 0.0 `
  --chat-template-kwargs '{\"preserve_thinking\": true}' `
  --flash-attn on `
  --spec-type draft-mtp `
  --spec-draft-n-max 3 `
  --cache-type-k-draft q4_0 `
  --cache-type-v-draft q4_0 `
  --batch-size 4096 `
  --ubatch-size 1024 `
  --cache-type-k q8_0 `
  --cache-type-v q8_0 `
  --checkpoint-every-n-tokens 16384 `
  --cache-ram 0 `
  --ctx-checkpoints 2 `
  --no-mmap `
  --host 0.0.0.0 `
  --port 8083 `
  --log-file $logFile

I do not have any --split-mode tensor setup for this system and the symptom I see is a regular growth of 4~6MiB chunks of VRAM being utilized during both prompt processing and token generation. (When deeper in the context window, it can jump by 4 to approx 32MiB per prompt, but early in context the 4~16MiB-per-growth seems to be mostly consistent).

Observations running a quick AI-written script to test multiple turns using a fairly standard "the quick brown fox..." type seed and then a "please continue" for further turns (pasted at the bottom since I can't attach .ps1 files) -- each turn sees a + delta VRAM change. This is the condition that eventually leads to an OOM:

Idle VRAM: 28734 MiB
Turn 1 (msgs= 3): 28734 -> 29752 MiB (delta +1018)
Turn 2 (msgs= 5): 29752 -> 29772 MiB (delta +20)
Turn 3 (msgs= 7): 29772 -> 29780 MiB (delta +8)
Turn 4 (msgs= 9): 29780 -> 29796 MiB (delta +16)
Turn 5 (msgs= 11): 29796 -> 29812 MiB (delta +16)
Turn 6 (msgs= 13): 29812 -> 29828 MiB (delta +16)
Turn 7 (msgs= 15): 29828 -> 29844 MiB (delta +16)
Turn 8 (msgs= 17): 29844 -> 29852 MiB (delta +8)
Turn 9 (msgs= 19): 29852 -> 29864 MiB (delta +12)
Turn 10 (msgs= 21): 29864 -> 29876 MiB (delta +12)
[...eventually this will OOM with enough turns... which is the issue at hand...]

---

[No recreate] Using a minimal launch parameter setup does NOT reproduce the issue:

& "C:\llama.cpp\b9275\build\bin\Release\llama-server.exe" `
  --model "C:\llama.cpp\models\Qwen3.6-27B-Q6_K.gguf" `
  --n-gpu-layers 999 `
  --ctx-size 131072 `
  --threads 16 `
  --host 0.0.0.0 `
  --port 8083 `
  --log-file $logFile

Minimal Scenario Observation: In this one, I see 30624MiB initially, then +88MiB on the first turn to 30712MiB and then +0 for all subsequent test turns.

Idle VRAM: 30624 MiB
Turn 1: 30624 -> 30712 MiB (delta +88)
Turn 2: 30712 -> 30712 MiB (delta +0)
Turn 3: 30712 -> 30712 MiB (delta +0)
Turn 4: 30712 -> 30712 MiB (delta +0)
Turn 5: 30712 -> 30712 MiB (delta +0)
Turn 6: 30712 -> 30712 MiB (delta +0)
Turn 7: 30712 -> 30712 MiB (delta +0)
Turn 8: 30712 -> 30712 MiB (delta +0)
Turn 9: 30712 -> 30712 MiB (delta +0)
Turn 10: 30712 -> 30712 MiB (delta +0)

---

[No recreate] MTP Spec Dec ONLY - Adding in speculative decoding (MTP) only, using this launch parameter setup (note: I had to reduce the context window to fit to VRAM size on my 5090 because the cache is not quantized with these launch parameters, so it overruns with the 'W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 999, abort' error):

& "C:\llama.cpp\b9275\build\bin\Release\llama-server.exe" `
  --model "C:\llama.cpp\models\Qwen3.6-27B-Q6_K.gguf" `
  --n-gpu-layers 999 `
  --ctx-size 50000 `
  --threads 16 `
  --spec-type draft-mtp `
  --spec-draft-n-max 3 `
  --cache-type-k-draft q4_0 `
  --cache-type-v-draft q4_0 `
  --host 0.0.0.0 `
  --port 8083 `
  --log-file $logFile

MTP Spec Dec Observation: Initial load RAM utilization is 28066MiB, turn 1 is +626MiB, all subsequent turns are +0.

Idle VRAM: 28066 MiB
Turn 1: 28066 -> 28692 MiB (delta +626)
Turn 2: 28692 -> 28692 MiB (delta +0)
Turn 3: 28692 -> 28692 MiB (delta +0)
Turn 4: 28692 -> 28692 MiB (delta +0)
Turn 5: 28692 -> 28692 MiB (delta +0)
Turn 6: 28692 -> 28692 MiB (delta +0)
Turn 7: 28692 -> 28692 MiB (delta +0)
Turn 8: 28692 -> 28692 MiB (delta +0)
Turn 9: 28692 -> 28692 MiB (delta +0)
Turn 10: 28692 -> 28692 MiB (delta +0)

---

[No recreate] Context Checkpoints ONLY - Next I tried to check and see if the context checkpoints was the isolated cause of the issue and used this setup (note: I left the context window at 50000):

& "C:\llama.cpp\b9275\build\bin\Release\llama-server.exe" `
  --model "C:\llama.cpp\models\Qwen3.6-27B-Q6_K.gguf" `
  --n-gpu-layers 999 `
  --ctx-size 50000 `
  --threads 16 `
  --checkpoint-every-n-tokens 16384 `
  --ctx-checkpoints 2 `
  --cache-ram 0 `
  --host 0.0.0.0 `
  --port 8083 `
  --log-file $logFile

Context Checkpoints Observation: I see 25568MiB initially, then +86MiB on the first turn, +2MiB on the second turn, and then +0 for all subsequent test turns. No recreate.

PS C:\llama.cpp> pwsh -File C:\llama.cpp\vram-leak-harness-appending.ps1 -Tag B-checkpoints -Turns 20 -Tokens 2048
Idle VRAM: 25568 MiB
Turn 1 (msgs= 3): 25568 -> 25654 MiB (delta +86)
Turn 2 (msgs= 5): 25654 -> 25656 MiB (delta +2)
Turn 3 (msgs= 7): 25656 -> 25656 MiB (delta +0)
Turn 4 (msgs= 9): 25656 -> 25656 MiB (delta +0)
Turn 5 (msgs= 11): 25656 -> 25656 MiB (delta +0)
Turn 6 (msgs= 13): 25656 -> 25656 MiB (delta +0)
Turn 7 (msgs= 15): 25656 -> 25656 MiB (delta +0)
Turn 8 (msgs= 17): 25656 -> 25656 MiB (delta +0)
Turn 9 (msgs= 19): 25656 -> 25656 MiB (delta +0)
Turn 10 (msgs= 21): 25656 -> 25656 MiB (delta +0)
[...remainder of turns truncated...]

---

[Recreate?!] KV Cache Quantization - Next I tested KV cache quantization using this script (note: the context window went back up to 131072 here because it fits in VRAM now and I left the remainder of the header env variables here in case someone wants to use this verbatim):

$env:PATH = "C:\llama.cpp\b9275\build\bin\Release;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\bin;" + $env:PATH
$env:CUDA_PATH = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1"

$logFile = "C:\llama.cpp\logs\serve-D-kvq-$(Get-Date -Format yyyyMMdd-HHmmss).log"
New-Item -ItemType Directory -Force -Path (Split-Path $logFile) | Out-Null

& "C:\llama.cpp\b9275\build\bin\Release\llama-server.exe" `
  --model "C:\llama.cpp\models\Qwen3.6-27B-Q6_K.gguf" `
  --n-gpu-layers 999 `
  --ctx-size 131072 `
  --threads 16 `
  --cache-type-k q8_0 `
  --cache-type-v q8_0 `
  --host 0.0.0.0 `
  --port 8083 `
  --log-file $logFile

KV Cache Quantization Observation: I definitely see VRAM usage creep up here (recreate?!). I'm not sure if this is expected or not, but I believe it is likely not since VRAM usage should be stable. One thing that is important to note is that I see the VRAM usage creep up right at the turn boundary (so I think it probably has to do with prompt processing side on the new turn). It is mildly interesting that this recreates without the MTP enablement...but perhaps it is worse when combined with MTP? I will test this out on the same script next, but wanted to share findings and possibly combine MTP with different KV cache quantization settings to see if there's an effect.

PS C:\llama.cpp> pwsh -File C:\llama.cpp\vram-leak-harness-appending.ps1 -Tag B-checkpoints -Turns 10 -Tokens 4096
Idle VRAM: 26882 MiB
Turn 1 (msgs= 3): 26882 -> 26906 MiB (delta +24)
Turn 2 (msgs= 5): 26906 -> 26912 MiB (delta +6)
Turn 3 (msgs= 7): 26912 -> 26918 MiB (delta +6)
Turn 4 (msgs= 9): 26918 -> 26926 MiB (delta +8)
Turn 5 (msgs= 11): 26926 -> 26934 MiB (delta +8)
Turn 6 (msgs= 13): 26934 -> 26940 MiB (delta +6)
Turn 7 (msgs= 15): 26940 -> 26946 MiB (delta +6)
Turn 8 (msgs= 17): 26946 -> 26952 MiB (delta +6)
Turn 9 (msgs= 19): 26952 -> 26958 MiB (delta +6)
Turn 10 (msgs= 21): 26958 -> 26958 MiB (delta +0)

---

Appendix:

Script used to generate the test data above (powershell):

param(
  [string]$Endpoint = "http://127.0.0.1:8083/v1/chat/completions",
  [int]$Turns       = 30,
  [int]$Tokens      = 512,
  [string]$Tag      = "run"
)

$stamp   = Get-Date -Format yyyyMMdd-HHmmss
$outDir  = "C:\llama.cpp\logs\vram-$Tag-$stamp"
New-Item -ItemType Directory -Force -Path $outDir | Out-Null
$vramCsv  = Join-Path $outDir "vram.csv"
$turnsCsv = Join-Path $outDir "turns.csv"

$smi = Start-Process -FilePath "nvidia-smi" `
  -ArgumentList "--query-gpu=timestamp,memory.used,memory.free","--format=csv,noheader,nounits","-l","1" `
  -RedirectStandardOutput $vramCsv -NoNewWindow -PassThru

function Get-VramMiB {
  $line = & nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | Select-Object -First 1
  [int]($line.Trim())
}

# Seed prompt — long-ish, ~1.5k tokens, so 30 turns covers ~45k+ tokens of context.
$seed = "You are an assistant. Discuss the following passage in detail. " + ("The quick brown fox jumps over the lazy dog. " * 40)

$messages = @(@{ role = "user"; content = $seed })

"turn,ctx_messages,vram_before_mib,vram_after_mib,delta_mib" | Out-File -FilePath $turnsCsv -Encoding ascii

Start-Sleep -Seconds 3
$baseline = Get-VramMiB
Write-Host "Idle VRAM: $baseline MiB"

for ($i = 1; $i -le $Turns; $i++) {
  $before = Get-VramMiB
  $body = @{
    model       = "local"
    max_tokens  = $Tokens
    temperature = 0
    messages    = $messages
  } | ConvertTo-Json -Depth 10

  try {
    $resp = Invoke-RestMethod -Uri $Endpoint -Method Post -ContentType "application/json" -Body $body -TimeoutSec 600
    $assistantReply = $resp.choices[0].message.content
    $messages += @{ role = "assistant"; content = $assistantReply }
    $messages += @{ role = "user"; content = "Please continue elaborating with more detail and examples." }
  } catch {
    Write-Warning "Turn $i request failed: $($_.Exception.Message)"
  }

  Start-Sleep -Seconds 1
  $after = Get-VramMiB
  $delta = $after - $before
  "$i,$($messages.Count),$before,$after,$delta" | Add-Content -Path $turnsCsv
  $sign = if ($delta -ge 0) { "+" } else { "" }
  Write-Host ("Turn {0,2} (msgs={1,3}): {2,6} -> {3,6} MiB  (delta {4}{5})" -f $i, $messages.Count, $before, $after, $sign, $delta)
}

Stop-Process -Id $smi.Id -Force -ErrorAction SilentlyContinue
Write-Host "Per-turn CSV: $turnsCsv"
Write-Host "1Hz VRAM CSV: $vramCsv"

[noodleprincss-ai]

I went back to my original launcher script and changed the KV cache from q8_0 on both to f16 on both (and reduced my context window from 131072 to 50000 to fit in VRAM) and I can see that the issue does not entirely go away, but is greatly reduced.

Here's the based-on-my-original launcher script with the KV cache and context window changed:

. "C:\llama.cpp\gpu-config.ps1"
$env:PATH = "C:\llama.cpp\b9275\build\bin\Release;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1\bin;" + $env:PATH
$env:CUDA_PATH = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.1"

$logFile = "C:\llama.cpp\logs\qwen3.6-27b-mtp-$(Get-Date -Format 'yyyyMMdd-HHmm').log"
New-Item -ItemType Directory -Force -Path "C:\llama.cpp\logs" | Out-Null
Write-Host "Logging to: $logFile"

& "C:\llama.cpp\b9275\build\bin\Release\llama-server.exe" `
  --model "C:\llama.cpp\models\Qwen3.6-27B-Q6_K.gguf" `
  --mmproj "C:\llama.cpp\models\Qwen-3.6-27B-mmproj-BF16.gguf" `
  --image-min-tokens 1024 `
  --no-mmproj-offload `
  --n-gpu-layers 999 `
  --ctx-size 50000 `
  --fit off `
  --parallel 1 `
  --threads 16 `
  --temp 0.7 `
  --top-p 0.95 `
  --min-p 0.00 `
  --top-k 20 `
  --presence_penalty 0.0 `
  --chat-template-kwargs '{\"preserve_thinking\": true}' `
  --flash-attn on `
  --spec-type draft-mtp `
  --spec-draft-n-max 3 `
  --cache-type-k-draft q4_0 `
  --cache-type-v-draft q4_0 `
  --batch-size 4096 `
  --ubatch-size 1024 `
  --cache-type-k f16 `
  --cache-type-v f16 `
  --checkpoint-every-n-tokens 16384 `
  --cache-ram 0 `
  --ctx-checkpoints 2 `
  --no-mmap `
  --host 0.0.0.0 `
  --port 8083 `
  --log-file $logFile

And the output of my previous comment's script to test multiple message turns for VRAM usage deltas:

PS C:\llama.cpp> pwsh -File C:\llama.cpp\vram-leak-harness-appending.ps1 -Tag full-minus-kvq -Turns 10 -Tokens 4096
Idle VRAM: 27280 MiB
Turn 1 (msgs= 3): 27280 -> 28392 MiB (delta +1112)
Turn 2 (msgs= 5): 28392 -> 28392 MiB (delta +0)
Turn 3 (msgs= 7): 28392 -> 28392 MiB (delta +0)
Turn 4 (msgs= 9): 28392 -> 28392 MiB (delta +0)
Turn 5 (msgs= 11): 28392 -> 28400 MiB (delta +8)
Turn 6 (msgs= 13): 28400 -> 28400 MiB (delta +0)
Turn 7 (msgs= 15): 28400 -> 28400 MiB (delta +0)
Turn 8 (msgs= 17): 28400 -> 28400 MiB (delta +0)
Turn 9 (msgs= 19): 28400 -> 28400 MiB (delta +0)
Turn 10 (msgs= 21): 28400 -> 28400 MiB (delta +0)

It seems like >90% of the issue is related to KV cache quantization causing the VRAM leak (?)... I did still see one turn where there was a +8MiB delta though... mostly isolating to KV cache quantization is great and terrible because that greatly reduces the available context window, but it's good that it seems to be mostly confined to one issue (though perhaps there is a second one lurking out there, just not nearly as severe). I have a mild hypothesis that the draft model's quantization may be the contributor for that single +8MiB I saw in this most recent run, but unconfirmed.

TL;DR seems to be --cache-type-k q8_0 & --cache-type-v q8_0 cause >90% of this issue based on my observations / setup.

[noodleprincss-ai]

I don't know the llama.cpp code at all, but using Claude, I think I was able to pinpoint this issue (note: AI generated code here, so not well reviewed yet) and I'm now testing on my personal setup, but the first few rounds of testing seem legitimately better with the caveat that it eats more VRAM up front on first message turn, but I think that's more truthful than simply growing VRAM forever and eventually OOMing:

• Repro: quantized KV cache (--cache-type-k/v q8_0 or any non-fp16) + flash attention (default auto enables on modern hardware) + long-output decoding. VRAM grows monotonically by Δ_context × n_kv_heads × head_dim × sizeof(half) × 2 (K+V) per turn.

• Mechanism: launch_fattn in ggml/src/ggml-cuda/fattn-common.cuh (lines 975, 1008 on master c0c7e14) allocates K_f16/V_f16 dequant workspaces sized by current ggml_nelements(K/V), which grows with context. The legacy CUDA pool allocator (ggml_cuda_pool_leg::alloc in ggml-cuda.cu) requires b.size >= request_size for reuse, so older smaller buffers strand in buffer_pool[] as context grows.

Two lines changed in substance:

Line 975: K_f16.alloc(ggml_nelements(K)); → K_f16.alloc(ggml_nelements(K->view_src ? K->view_src : K));
Line 1008: V_f16.alloc(ggml_nelements(V)); → V_f16.alloc(ggml_nelements(V->view_src ? V->view_src : V));

• Validating on my local build: Change eliminates per-turn growth; costs ~1 GiB additional upfront committed for a Qwen3.6-27B/131k-ctx with MTP config but no longer grows forever unbounded to OOM. Performance also seems unaffected, getting ~95-125tok/s TG but have not rigorously tested PP.

Early tests using my original launch parameters:

Idle VRAM: 28734 MiB
Turn 1 (msgs= 3): 28734 -> 30772 MiB (delta +2038)
Turn 2 (msgs= 5): 30772 -> 30772 MiB (delta +0)
Turn 3 (msgs= 7): 30772 -> 30772 MiB (delta +0)
Turn 4 (msgs= 9): 30772 -> 30772 MiB (delta +0)
Turn 5 (msgs= 11): 30772 -> 30772 MiB (delta +0)
Turn 6 (msgs= 13): 30772 -> 30772 MiB (delta +0)
Turn 7 (msgs= 15): 30772 -> 30772 MiB (delta +0)
Turn 8 (msgs= 17): 30772 -> 30772 MiB (delta +0)
Turn 9 (msgs= 19): 30772 -> 30772 MiB (delta +0)
Turn 10 (msgs= 21): 30772 -> 30772 MiB (delta +0)

11.39.780.578 I slot print_timing: id 0 | task 3948 | n_decoded = 100, tg = 100.89 t/s
11.42.802.896 I slot print_timing: id 0 | task 3948 | n_decoded = 399, tg = 99.40 t/s
11.45.826.802 I slot print_timing: id 0 | task 3948 | n_decoded = 725, tg = 103.02 t/s
11.48.848.485 I slot print_timing: id 0 | task 3948 | n_decoded = 1116, tg = 110.94 t/s
11.51.870.240 I slot print_timing: id 0 | task 3948 | n_decoded = 1521, tg = 116.27 t/s
11.54.896.055 I slot print_timing: id 0 | task 3948 | n_decoded = 1896, tg = 117.71 t/s
11.57.908.274 I slot print_timing: id 0 | task 3948 | n_decoded = 2313, tg = 120.98 t/s
12.00.914.412 I slot print_timing: id 0 | task 3948 | n_decoded = 2661, tg = 120.27 t/s
12.03.925.418 I slot print_timing: id 0 | task 3948 | n_decoded = 3007, tg = 119.63 t/s
12.06.936.874 I slot print_timing: id 0 | task 3948 | n_decoded = 3328, tg = 118.23 t/s
12.09.973.894 I slot print_timing: id 0 | task 3948 | n_decoded = 3696, tg = 118.52 t/s
12.12.996.514 I slot print_timing: id 0 | task 3948 | n_decoded = 4032, tg = 117.87 t/s
12.16.004.529 I slot print_timing: id 0 | task 3948 | n_decoded = 4369, tg = 117.40 t/s
12.19.028.156 I slot print_timing: id 0 | task 3948 | n_decoded = 4721, tg = 117.33 t/s
12.22.049.990 I slot print_timing: id 0 | task 3948 | n_decoded = 5099, tg = 117.87 t/s
12.25.067.039 I slot print_timing: id 0 | task 3948 | n_decoded = 5472, tg = 118.24 t/s
12.28.082.592 I slot print_timing: id 0 | task 3948 | n_decoded = 5817, tg = 118.01 t/s
12.31.085.322 I slot print_timing: id 0 | task 3948 | n_decoded = 6175, tg = 118.08 t/s
12.34.112.720 I slot print_timing: id 0 | task 3948 | n_decoded = 6541, tg = 118.23 t/s
12.37.115.602 I slot print_timing: id 0 | task 3948 | n_decoded = 6871, tg = 117.80 t/s
12.40.142.380 I slot print_timing: id 0 | task 3948 | n_decoded = 7264, tg = 118.40 t/s
12.43.159.595 I slot print_timing: id 0 | task 3948 | n_decoded = 7639, tg = 118.67 t/s
12.46.169.148 I slot print_timing: id 0 | task 3948 | n_decoded = 8039, tg = 119.31 t/s
12.49.170.028 I slot print_timing: id 0 | task 3948 | n_decoded = 8418, tg = 119.61 t/s
12.52.182.545 I slot print_timing: id 0 | task 3948 | n_decoded = 8791, tg = 119.78 t/s
12.55.203.220 I slot print_timing: id 0 | task 3948 | n_decoded = 9145, tg = 119.68 t/s
12.58.221.349 I slot print_timing: id 0 | task 3948 | n_decoded = 9510, tg = 119.73 t/s
13.01.229.171 I slot print_timing: id 0 | task 3948 | n_decoded = 9898, tg = 120.07 t/s
13.04.251.455 I slot print_timing: id 0 | task 3948 | n_decoded = 10261, tg = 120.07 t/s
13.07.267.187 I slot print_timing: id 0 | task 3948 | n_decoded = 10596, tg = 119.76 t/s
13.10.281.389 I slot print_timing: id 0 | task 3948 | n_decoded = 10907, tg = 119.21 t/s

[rafalhao]

@noodleprincss-ai good findings.

Just another anecdote,

When the MTP PR was first merged, I remember I could fill 128k context window without OOM errors on my RTX 3090 with q8 KV cache. (with maybe 100mb vram left, when the context was full).

Now in the past 4 or 5 days, not sure about the exact time frame, I pulled all the changes from main/master and I noticed that running llama-server with the exact same flags, changed two things:

1. First prompt consumed 600mb vram, maybe this is some kind of early allocation, not sure, I'll need to verify the PR's that were merged in the meantime.
2. Most importantly, I was always getting OOM errors, before filling my context window.

I also noticed that the VRAM usage increases as the context window fills up, but I think that's the expected behaviour, because this was also happening in the first few days MTP was merged. Although I think that now the increase of VRAM is bigger than before, but again not sure.

Whenever I get time, I'll mess around backtracking several commits in order to find the one, that caused this change of behaviour regarding the VRAM.

[noodleprincss-ai]

Can you identify which parameters cause this? Perhaps remove all -spec-type and check, and then add draft-mtp and then later add ngram-mod etc. I tried reproducing this and I can't

@am17an - Based on quite a bit of testing now (see previous comments), I'm very sure that this is now related to KV cache quantization. I am able to reproduce it on demand now and after a couple of AI generated code changes, I was able to validate that this behavior can be stopped, with the side effect of a larger Turn 1 upfront VRAM price (so, initial model load for me loads ~28.7GiB + turn 1 allocates ~2GiB but no additional allocation after.) See previous comments for what I did to test / potentially fix (maybe not optimally, but at least lets us hunt in the right area I think).

I've poked at this for an hour using that 2 line change from my previous comment ( #23446 (comment) ) and I'm pretty confident in it now. The real problem with it is the fact that when you load the server and it gets to the standard "all slots are idle" state, that is still not the actual VRAM price - you have to do a message turn to get the real VRAM price and I think that probably breaks --fit and requires a lot of hand fine tuning of the context window size.