server: enable checkpoint reuse for recurrent/hybrid models (qwen3next, Mamba)#1888
Open
localweights wants to merge 1 commit into
Open
server: enable checkpoint reuse for recurrent/hybrid models (qwen3next, Mamba)#1888localweights wants to merge 1 commit into
localweights wants to merge 1 commit into
Conversation
…t, Mamba) For Gated DeltaNet, qwen3next, and Mamba architectures, llama_kv_cache_seq_pos_min returns the full sequence length, so the existing apply_checkpoint validity check (`cur.pos_min < pos_min_thold`) always evaluates false. The slot is then force-reset and every request re-processes the entire prompt from scratch. Also: create_checkpoint required the slot to hold at least 64 tokens, which blocks checkpoint creation on short follow-up prompts for recurrent models where each new turn often adds < 64 tokens. Drop the threshold to 4 tokens for recurrent/hybrid architectures only. Fixes ikawrakow#1762. Ported from ggml-org/llama.cpp #22384. Validated on Qwen3.6-27B (qwen3next): direct probes go from 0 to 1713/1718 = 99.7% cache reuse; real multi-pass synthesis sees 92% cache hits on shared-prefix calls.
Owner
|
As per this comment in the quoted |
|
I don't really think that that "preserve thinking" related to qwen3next... Though I tend to believe it's kind of jinja problem, still a problem... |
|
I've been working on the same issue independently. Your PR covers the checkpoint search predicate and threshold fixes - I found a third root cause (partial KV delete nukes checkpoints before apply_checkpoint() runs). Opened #1976 with the additional fix + a prompt cache load guard. Happy to coordinate if you want to merge efforts. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1762. Ports the fix from ggml-org/llama.cpp PR for the same root cause.
For Gated DeltaNet, qwen3next, and Mamba architectures,
llama_kv_cache_seq_pos_minreturns the full sequence length, so the existingapply_checkpointvalidity check (cur.pos_min < pos_min_thold) always evaluates false. Consequence: no saved checkpoint is ever considered usable, the slot is force-reset, and every follow-up request re-processes the entire prompt from scratch.A second small bug compounds it:
create_checkpointrequires the slot to hold at least 64 tokens before saving. For recurrent models a short follow-up prompt commonly adds far fewer than 64 new tokens, so the checkpoint that would have rescued the next turn never gets created.Changes
apply_checkpoint: extend the lambda's validity check to also acceptcur.pos_max <= pos_next. Dense transformer paths still trip the original conditions first, so no behavior change for them.create_checkpoint: split the minimum-tokens gate by architecture. Recurrent/hybrid models now create a checkpoint at >= 4 tokens; everything else still requires >= 64.Validation
Tested on Qwen3.6-27B (qwen3next arch) with
--multi-token-prediction --draft-max 3+--reasoning on:Before this patch:
cached_tokens: 0every callAfter this patch:
cached_tokens: 1713 / 1718(99.7% reuse)Reference