Skip to content

[Perf] [XPU] graph: Support full prefill graph for GDN/Mamba and Gemma models - #48847

Draft
CaoE wants to merge 7 commits into
sgl-project:mainfrom
CaoE:ecao/fix_for_prefill_graph_rebase
Draft

CaoE wants to merge 7 commits into
sgl-project:mainfrom
CaoE:ecao/fix_for_prefill_graph_rebase

Conversation

@CaoE

@CaoE CaoE commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Enable full prefill CUDA/XPU graph execution for hybrid GDN/Mamba architectures (e.g. Qwen3.5-35B-A3B) and Gemma-3/4 models. Previously, these models encountered runtime failures during prefill graph capture and replay due to:

  1. Strict (input_ids is None) ^ (inputs_embeds is not None) validation in Gemma models when graph capture requires passing input_embeds alongside input_ids.
  2. Lack of ForwardMode.EXTEND metadata replay support and missing extend_start_loc / extend_seq_lens handling in Mamba/Hybrid attention backends.
  3. Buffer size mismatch and re-entrancy issues across prefill and decode graph runners in ShortConvAttnBackend and HybridLinearAttnBackend.
  4. Missing padded track buffer bindings in PrefillCudaGraphRunner replay metadata.
  5. Incompatibility with older transformers releases lacking is_heterogeneous on Gemma4TextConfig.

Modifications

  • python/sglang/srt/models/gemma3n_causal.py & gemma3n_mm.py:

    • Removed strict mutual exclusion check between input_ids and input_embeds, allowing input_ids to be supplied alongside input_embeds to extract per_layer_inputs.
  • python/sglang/srt/models/gemma4_causal.py & gemma4_mm.py:

    • Relaxed input_ids and input_embeds check so input_ids can be used to generate per_layer_inputs while embedding tokens via input_embeds.
  • python/sglang/srt/layers/attention/hybrid_linear_attn_backend.py:

    • Added extend_start_loc and extend_seq_lens parameters to init_forward_metadata_out_graph and _replay_metadata.
    • Added forward_mode.is_extend_without_speculative() branch in _replay_metadata to copy extend_start_loc into static query_start_loc_list.
    • Updated init_cuda_graph_state to be incrementally growable across multiple initializations.
    • Implemented init_prefill_cuda_graph_state for MambaAttnBackendBase and HybridLinearAttnBackend to properly size state indices and query start locations for prefill max_req_slots.
  • python/sglang/srt/layers/attention/linear/short_conv_backend.py:

    • Changed _alloc_cache_indices_buf from single-initialization assertion to grow-only reallocation (buf.shape[0] >= max_bs) to support shared backend initialization by both prefill and decode graph runners.
  • python/sglang/srt/model_executor/runner/prefill_cuda_graph_runner.py:

    • Set extend_start_loc and num_padding on padded_view during replay forward metadata initialization.
    • Sliced and attached padded mamba_track_indices and mamba_track_mask buffers ([:r]) to padded_view to prevent length mismatch assertions during replay.
  • python/sglang/srt/hf_transformers/config.py:

    • Safely checked text_config.is_heterogeneous via getattr(..., False) and provided fallback to global_head_dim / num_global_key_value_heads for backward compatibility with older transformers versions.

Accuracy Tests

1. Unit & Metadata Tests

  • pytest test/registered/xpu/test_xpu_graph_metadata.py: 4 passed
  • pytest test/registered/xpu/test_intel_xpu_linear_attn_dispatch.py: 4 passed
  • pytest test/registered/xpu/test_xpu_basic.py: 1 passed

2. Full Graph Replay & Cosine Similarity Correctness

  • pytest test/registered/xpu/test_xpu_graph.py -s -v: Passed (3 subtests passed)
    • Bucket exact match (32 tokens -> bucket 64): Replayed prefill graph bucket 64, decode batch size 2 (cos_sim > 0.999, greedy tokens exactly matched eager).
    • Bucket padding (48 tokens -> bucket 128): Replayed prefill graph bucket 128, decode batch size 2 (cos_sim > 0.999, greedy tokens exactly matched eager).
    • Prefix-cached extend: Both prefix and suffix extends replayed prefill graph bucket 64 (cos_sim > 0.999, greedy tokens exactly matched eager).

Speed Tests and Profiling

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Missing run-ci label -- add it to run CI tests.
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant