Skip to content

server : fix false speculative warning when not enabled#217

Closed
ken00H wants to merge 1 commit into
TheTom:feature/turboquant-kv-cachefrom
ken00H:feature/turboquant-kv-cache
Closed

server : fix false speculative warning when not enabled#217
ken00H wants to merge 1 commit into
TheTom:feature/turboquant-kv-cachefrom
ken00H:feature/turboquant-kv-cache

Conversation

@ken00H

@ken00H ken00H commented Jul 14, 2026

Copy link
Copy Markdown

Overview of the Change

  1. The Issue:
    Whenever you started llama-server , it checked if the context could remove tokens (sequence removal) and tried to set up speculative decoding. Because speculative decoding is disabled
    by default, it would print these warnings during startup:
    • "speculative decoding will use checkpoints"
    • "no implementations specified for speculative decoding"
  2. The Fix:
    We added a check is_speculative_enabled in tools/server/server-context.cpp . It checks if the speculative types configured actually contain a speculative method (i.e. anything other
    than NONE ).
  3. The Result:
    If speculative decoding is not enabled, the server skips the check and initialization, keeping the startup logs clean of these false-positive warnings.

@TheTom

TheTom commented Jul 14, 2026

Copy link
Copy Markdown
Owner

Thanks @ken00H — landed a corrected version of this on feature/turboquant-kv-cache as `7e2fe72f3`'s follow-up (commit message: server: gate speculative init/warnings without dropping seq_rm probe).

What was right in the PR

Gating speculative warnings + `common_speculative_init` on an actual non-`NONE` speculative type is correct. Default server startup was spamming false positives.

Bug in the PR as written

The PR also moved `ctx_tgt_seq_rm_type = common_context_can_seq_rm(ctx_tgt)` inside the speculative-only branch. That field is not speculative-only — completion checkpointing uses it later:

// server-context.cpp ~3297
do_checkpoint = do_checkpoint && (
        ctx_tgt_seq_rm_type == COMMON_CONTEXT_SEQ_RM_TYPE_FULL ||
        ctx_tgt_seq_rm_type == COMMON_CONTEXT_SEQ_RM_TYPE_RS ||
        n_swa > 0);

If speculative is off, the member would stay at the default `COMMON_CONTEXT_SEQ_RM_TYPE_NO` and FULL/RS models would skip checkpoints unless SWA forced them. Silent correctness regression for non-spec runs.

What we landed instead

  1. Always probe `common_context_can_seq_rm(ctx_tgt)`.
  2. Gate only the speculative warnings + `common_speculative_init` on `is_speculative_enabled`.

Metal `llama-server` build verified locally. Closing this PR in favor of the tip commit.

TheTom added a commit that referenced this pull request Jul 14, 2026
Based on #217 (ken00H): skip speculative init and its startup warnings
when no non-NONE speculative type is configured.

Do NOT skip common_context_can_seq_rm(ctx_tgt) — ctx_tgt_seq_rm_type also
drives completion checkpointing (FULL / RS models), independent of
speculative decoding. Leaving it at the default NO would disable those
checkpoints for non-spec runs.

TurboQuant: no kernel impact.
@TheTom TheTom closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants