Repository navigation
Open
Description
Tracking which model families continuous batching (generate_batch, ContinuousBatchingManager) runs. The engine takes its attention types from layer_types (group_layers_by_attn_type) and sizes its paged cache from num_key_value_heads and head_dim; anything outside that is rejected at init or not representable.
- decoder-only models with
full_attentionand/orsliding_attentionlayers (dense and MoE), withpaged|eager,sdpa, flash attention 2/3; tensor parallelism - gated delta net hybrids: Qwen3-Next, Qwen3.5, Qwen3.5-MoE Continuous batching for linear attention layers #50350
- OLMo-Hybrid: own copy of the gated delta net forward (declare its states, add the packed-row branch: the smallest one)
- Kimi Linear, GLM5-next: KDA (
KimiLinearAttention,Glm5NextTextLinearAttention) - MiniMax:
MiniMaxLightningAttention - Bailing-Hybrid:
BailingHybridDeltaAttention - Mamba2 mixers: Bamba, Falcon-H1, NemotronH, Granite-MoE-Hybrid, Jamba, Zamba, Zamba2, Inkling, Nemotron-3.5-Transcribe (conv state + SSM state,
causal_conv1dandmamba_ssmwithcu_seqlens) -
mambalayers (Mamba, Jamba) andconvlayers (LFM2): same kind of per-request state - Qwen4-Exp:
indexed_attention(QSA indexer) and the per-layer n-gram embeddings Run Qwen4-Exp under continuous batching #50359 -
indexed_attentionwith MLA: DeepSeek-V3.2, GLM-5, Mistral Large 4, HY-V4, AXK2 Run multi-head latent attention models under continuous batching #50365 -
chunked_attention(Llama 4) - MLA: DeepSeek V2/V3, Kimi K2, GLM-4.7-Flash, MiniCPM3, Mistral 4, LongCat-Flash, Youtu, AXK1 Run multi-head latent attention models under continuous batching #50365
- DeepSeek V4: compressed sparse and heavily compressed attention (one compressed entry per window of tokens, the compressor's open window as a per-request state, sliding keys)
- multimodal prompts:
generate_batchtakes token ids only, no pixel inputs orinputs_embeds, so VLMs run text-only - encoder-decoder models
Groundwork that makes the per-family items mechanical: a varlen scan with per-sequence states for Mamba2, like causal_conv1d_varlen for the convolution. The cache takes the rows and states each layer declares, by name (cache_rows, cache_states, #50356), so a new family is that declaration in its modules plus the packed-row branch of its forward.
Metadata
Metadata
Assignees
Labels
No labels
Activity
[-]Continuous batching for the remaining linear-attention and state-space hybrid models[/-][+]Continuous batching: model support tracker[/+]