Skip to content

server: enable checkpoint reuse for recurrent/hybrid models (qwen3next, Mamba)#1888

Open
localweights wants to merge 1 commit into
ikawrakow:mainfrom
localweights:fix-qwen3next-checkpoint-reuse
Open

server: enable checkpoint reuse for recurrent/hybrid models (qwen3next, Mamba)#1888
localweights wants to merge 1 commit into
ikawrakow:mainfrom
localweights:fix-qwen3next-checkpoint-reuse

Conversation

@localweights

Copy link
Copy Markdown

Summary

Fixes #1762. Ports the fix from ggml-org/llama.cpp PR for the same root cause.

For Gated DeltaNet, qwen3next, and Mamba architectures, llama_kv_cache_seq_pos_min returns the full sequence length, so the existing apply_checkpoint validity check (cur.pos_min < pos_min_thold) always evaluates false. Consequence: no saved checkpoint is ever considered usable, the slot is force-reset, and every follow-up request re-processes the entire prompt from scratch.

A second small bug compounds it: create_checkpoint requires the slot to hold at least 64 tokens before saving. For recurrent models a short follow-up prompt commonly adds far fewer than 64 new tokens, so the checkpoint that would have rescued the next turn never gets created.

Changes

  • apply_checkpoint: extend the lambda's validity check to also accept cur.pos_max <= pos_next. Dense transformer paths still trip the original conditions first, so no behavior change for them.
  • create_checkpoint: split the minimum-tokens gate by architecture. Recurrent/hybrid models now create a checkpoint at >= 4 tokens; everything else still requires >= 64.

Validation

Tested on Qwen3.6-27B (qwen3next arch) with --multi-token-prediction --draft-max 3 + --reasoning on:

Before this patch:

  • Direct identical-prompt probe: cached_tokens: 0 every call
  • Multi-pass synthesis (12-section research report): every section paid full prefill, ~50s per section

After this patch:

  • Direct identical-prompt probe: cached_tokens: 1713 / 1718 (99.7% reuse)
  • Multi-pass synthesis: 92% prefix-cache reuse on shared-prefix calls

Reference

…t, Mamba)

For Gated DeltaNet, qwen3next, and Mamba architectures,
llama_kv_cache_seq_pos_min returns the full sequence length, so the
existing apply_checkpoint validity check (`cur.pos_min < pos_min_thold`)
always evaluates false. The slot is then force-reset and every request
re-processes the entire prompt from scratch.

Also: create_checkpoint required the slot to hold at least 64 tokens,
which blocks checkpoint creation on short follow-up prompts for
recurrent models where each new turn often adds < 64 tokens. Drop the
threshold to 4 tokens for recurrent/hybrid architectures only.

Fixes ikawrakow#1762. Ported from ggml-org/llama.cpp #22384.

Validated on Qwen3.6-27B (qwen3next): direct probes go from 0 to
1713/1718 = 99.7% cache reuse; real multi-pass synthesis sees 92%
cache hits on shared-prefix calls.
@ikawrakow

Copy link
Copy Markdown
Owner

As per this comment in the quoted llama.cpp issue, the "fix" is incorrect.

@FNsi

FNsi commented May 27, 2026

Copy link
Copy Markdown

I don't really think that that "preserve thinking" related to qwen3next...

Though I tend to believe it's kind of jinja problem, still a problem...

@razlani

razlani commented Jun 16, 2026

Copy link
Copy Markdown

I've been working on the same issue independently. Your PR covers the checkpoint search predicate and threshold fixes - I found a third root cause (partial KV delete nukes checkpoints before apply_checkpoint() runs).

Opened #1976 with the additional fix + a prompt cache load guard. Happy to coordinate if you want to merge efforts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] qwen3next (Qwen3.6): context checkpoints never restored — full prompt re-processing on every turn

4 participants