Skip to content

Context checkpoint restore can use expanded recurrent cache topology after speculative decode #49

Description

@Nowayz

Problem

With hybrid/recurrent models, prompt caching, context checkpoints, and speculative decoding enabled, a later request can restore a prompt checkpoint while recurrent backup cells are still expanded for speculative decoding.

Recurrent backup cells are expanded for speculative decoding, but the server does not shrink them back before later prompt-cache/checkpoint restore. That means checkpoint restore is happening against a different recurrent-cache topology than the one used when the checkpoint was made.

Impact

This can leave recurrent cell metadata inconsistent and cause failures while loading recurrent state for the next graph. In CUDA builds this can surface as a GET_ROWS failure while reading recurrent convolution state.

Expected Behavior

Before prompt-cache save/load or checkpoint restore, the server should return recurrent memory to the prefill topology when speculative backup cells are no longer in use, and resize should sanitize recurrent cell metadata.

Repro Shape

  • hybrid/recurrent model
  • server prompt cache enabled
  • context checkpoints enabled
  • speculative decoding using recurrent backup cells
  • repeated prompts with a large shared prefix and changed tail

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions