Skip to content

Continuous batching: model support tracker #50352

Open

Description

@qgallouedec

Tracking which model families continuous batching (generate_batch, ContinuousBatchingManager) runs. The engine takes its attention types from layer_types (group_layers_by_attn_type) and sizes its paged cache from num_key_value_heads and head_dim; anything outside that is rejected at init or not representable.

  • decoder-only models with full_attention and/or sliding_attention layers (dense and MoE), with paged|eager, sdpa, flash attention 2/3; tensor parallelism
  • gated delta net hybrids: Qwen3-Next, Qwen3.5, Qwen3.5-MoE Continuous batching for linear attention layers #50350
  • OLMo-Hybrid: own copy of the gated delta net forward (declare its states, add the packed-row branch: the smallest one)
  • Kimi Linear, GLM5-next: KDA (KimiLinearAttention, Glm5NextTextLinearAttention)
  • MiniMax: MiniMaxLightningAttention
  • Bailing-Hybrid: BailingHybridDeltaAttention
  • Mamba2 mixers: Bamba, Falcon-H1, NemotronH, Granite-MoE-Hybrid, Jamba, Zamba, Zamba2, Inkling, Nemotron-3.5-Transcribe (conv state + SSM state, causal_conv1d and mamba_ssm with cu_seqlens)
  • mamba layers (Mamba, Jamba) and conv layers (LFM2): same kind of per-request state
  • Qwen4-Exp: indexed_attention (QSA indexer) and the per-layer n-gram embeddings Run Qwen4-Exp under continuous batching #50359
  • indexed_attention with MLA: DeepSeek-V3.2, GLM-5, Mistral Large 4, HY-V4, AXK2 Run multi-head latent attention models under continuous batching #50365
  • chunked_attention (Llama 4)
  • MLA: DeepSeek V2/V3, Kimi K2, GLM-4.7-Flash, MiniCPM3, Mistral 4, LongCat-Flash, Youtu, AXK1 Run multi-head latent attention models under continuous batching #50365
  • DeepSeek V4: compressed sparse and heavily compressed attention (one compressed entry per window of tokens, the compressor's open window as a per-request state, sliding keys)
  • multimodal prompts: generate_batch takes token ids only, no pixel inputs or inputs_embeds, so VLMs run text-only
  • encoder-decoder models

Groundwork that makes the per-family items mechanical: a varlen scan with per-sequence states for Mamba2, like causal_conv1d_varlen for the convolution. The cache takes the rows and states each layer declares, by name (cache_rows, cache_states, #50356), so a new family is that declaration in its modules plus the packed-row branch of its forward.

Activity

  1. changed the title [-]Continuous batching for the remaining linear-attention and state-space hybrid models[/-] [+]Continuous batching: model support tracker[/+] on Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions