server: do not recover prompt below cache-ram-similarity and other cleanup#2105
Merged
Conversation
ikawrakow
reviewed
Jul 10, 2026
| copy_data_to_cached_prompt(tokens, *ret); | ||
|
|
||
| ret->prompt_load(*prompt_cache, task.tokens); | ||
| ret->prompt_load(*prompt_cache, task.tokens, cache_ram_similarity); |
Owner
There was a problem hiding this comment.
Maybe rename this to min_reusable_fraction to match the actual meaning?
ikawrakow
approved these changes
Jul 10, 2026
joelfarthing
added a commit
to joelfarthing/ik_llama.cpp
that referenced
this pull request
Jul 10, 2026
Bring the openPangu-2.0-Flash port up to date with current main (70ea152). Two files conflicted; resolutions: - docs/parameters.md: keep the MTP heads key and default/experimental contract; add main's dflash stage, cross_ctx key, and draft-model example. - examples/server/server-context.cpp: keep the partial-KV-reuse and MTP fresh-warmup reset guards; adopt ikawrakow#2105's prompt-cache cleanup and reusable-fraction behavior around them. Merged tree is byte-identical to the validated integration tree c3e9b73.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #1877 as it is now stale
What it does now is that it does not restore the prompt if common prefix/cached prompt size less than cache-ram-similarity, which is the threshold that triggers the prompt caching. It makes the restoring process consistent with the saving process.
Other minor changes:
size_up_to_pos. With mtmd and max_pos=0, it should return 0 instead of 1. Add the check before the while loop.get_common_prefixstepsn_pastis subtracted by 1 to reflect "we have to evaluate at least 1 token to generate logits".apply_checkpoint()should happen after the finaln_past. Noticed this bug after Bug: KV cache prompt comparison runs before checkpoint restore, causing erroneous re-evaluation (with proposed fix) #2033