Repository navigation
Continuous batching does not support hybrid linear-attention models (all Qwen3.5/3.6/3.8) #48305
Description
Follow-up to #48298: with that fixed, the next wall for recent Qwen models is architectural.
Reproduction
(Requires the composite-config fix from #48299 first; without it the same call dies earlier on num_key_value_heads ... not found in the config.)
import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer, GenerationConfig
model_id = "Qwen/Qwen3.5-2B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda:0")
prompt = tokenizer.apply_chat_template([{"role": "user", "content": "Hi!"}], add_generation_prompt=True)["input_ids"]
outputs = model.generate_batch([prompt], generation_config=GenerationConfig(max_new_tokens=16))Outcome
The background generation thread dies in PagedAttentionCache.__init__:
ValueError: Invalid group type: linear_attention
Qwen3.5-style models interleave linear-attention layers with full attention (full_attention_interval: 4, i.e. 3 out of 4 layers are linear attention). The paged cache only implements full_attention and sliding_attention allocators; linear-attention layers need a per-request recurrent state (fixed size, no paging) instead of a KV block pool.
Every recent Qwen release (Qwen3.5, Qwen3.6, Qwen3.8) is such a hybrid, so continuous batching (and therefore transformers serve) supports none of them. Regular generate() works.
For scale: standard training forwards on these models work fine (verified: SFT on Qwen3.5-2B converges normally with sdpa), only the CB path is blocked.
Expected behavior
Either support linear-attention state in the CB cache (a LinearAttentionCacheAllocator holding a fixed-size recurrent state per request, analogous to what vLLM does for hybrid Qwen3.5), or fail at init_continuous_batching time with a clear "model architecture not supported by continuous batching" error instead of a background-thread crash that surfaces as a silent timeout (see also the fail-fast problem, #48298, fixed by #48301).
This one is a feature request rather than a bug fix; not attempted yet.
System info
transformers main (a353632), torch 2.13.0+cu130, Python 3.12.12, NVIDIA H100 80GB, flash attention via kernels-community/vllm-flash-attn3.
Activity
JagjotB commentedon Sep 2, 2026
I’d like to take a first pass at this.
My current understanding is that the main gap is that continuous batching assumes attention state can be managed through the paged KV-cache path, while the linear-attention layers in Qwen3.5/3.6/3.8 need fixed-size recurrent state per request instead.
My plan is to:
PagedAttentionCacheI’m going to start digging into the cache and continuous-batching code now and will share a draft once I have a clean implementation direction.
vasqu commentedon Sep 3, 2026
I think this is for internal tracking 👀 anyways cc @remi-or
remi-or commentedon Sep 4, 2026
Indeed, thanks @vasqu ! This will be solved internally.
github-actions commentedon Sep 29, 2026
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.