Problem
With hybrid/recurrent models, prompt caching, context checkpoints, and speculative decoding enabled, a later request can restore a prompt checkpoint while recurrent backup cells are still expanded for speculative decoding.
Recurrent backup cells are expanded for speculative decoding, but the server does not shrink them back before later prompt-cache/checkpoint restore. That means checkpoint restore is happening against a different recurrent-cache topology than the one used when the checkpoint was made.
Impact
This can leave recurrent cell metadata inconsistent and cause failures while loading recurrent state for the next graph. In CUDA builds this can surface as a GET_ROWS failure while reading recurrent convolution state.
Expected Behavior
Before prompt-cache save/load or checkpoint restore, the server should return recurrent memory to the prefill topology when speculative backup cells are no longer in use, and resize should sanitize recurrent cell metadata.
Repro Shape
- hybrid/recurrent model
- server prompt cache enabled
- context checkpoints enabled
- speculative decoding using recurrent backup cells
- repeated prompts with a large shared prefix and changed tail
Problem
With hybrid/recurrent models, prompt caching, context checkpoints, and speculative decoding enabled, a later request can restore a prompt checkpoint while recurrent backup cells are still expanded for speculative decoding.
Recurrent backup cells are expanded for speculative decoding, but the server does not shrink them back before later prompt-cache/checkpoint restore. That means checkpoint restore is happening against a different recurrent-cache topology than the one used when the checkpoint was made.
Impact
This can leave recurrent cell metadata inconsistent and cause failures while loading recurrent state for the next graph. In CUDA builds this can surface as a
GET_ROWSfailure while reading recurrent convolution state.Expected Behavior
Before prompt-cache save/load or checkpoint restore, the server should return recurrent memory to the prefill topology when speculative backup cells are no longer in use, and resize should sanitize recurrent cell metadata.
Repro Shape