Repository navigation
Conversation
…and Hybrid backends
…eplay and mm inputs
…rward_metadata_out_graph
4 of 5 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Enable full prefill CUDA/XPU graph execution for hybrid GDN/Mamba architectures (e.g. Qwen3.5-35B-A3B) and Gemma-3/4 models. Previously, these models encountered runtime failures during prefill graph capture and replay due to:
(input_ids is None) ^ (inputs_embeds is not None)validation in Gemma models when graph capture requires passinginput_embedsalongsideinput_ids.ForwardMode.EXTENDmetadata replay support and missingextend_start_loc/extend_seq_lenshandling in Mamba/Hybrid attention backends.ShortConvAttnBackendandHybridLinearAttnBackend.PrefillCudaGraphRunnerreplay metadata.transformersreleases lackingis_heterogeneousonGemma4TextConfig.Modifications
python/sglang/srt/models/gemma3n_causal.py&gemma3n_mm.py:input_idsandinput_embeds, allowinginput_idsto be supplied alongsideinput_embedsto extractper_layer_inputs.python/sglang/srt/models/gemma4_causal.py&gemma4_mm.py:input_idsandinput_embedscheck soinput_idscan be used to generateper_layer_inputswhile embedding tokens viainput_embeds.python/sglang/srt/layers/attention/hybrid_linear_attn_backend.py:extend_start_locandextend_seq_lensparameters toinit_forward_metadata_out_graphand_replay_metadata.forward_mode.is_extend_without_speculative()branch in_replay_metadatato copyextend_start_locinto staticquery_start_loc_list.init_cuda_graph_stateto be incrementally growable across multiple initializations.init_prefill_cuda_graph_stateforMambaAttnBackendBaseandHybridLinearAttnBackendto properly size state indices and query start locations for prefillmax_req_slots.python/sglang/srt/layers/attention/linear/short_conv_backend.py:_alloc_cache_indices_buffrom single-initialization assertion to grow-only reallocation (buf.shape[0] >= max_bs) to support shared backend initialization by both prefill and decode graph runners.python/sglang/srt/model_executor/runner/prefill_cuda_graph_runner.py:extend_start_locandnum_paddingonpadded_viewduring replay forward metadata initialization.mamba_track_indicesandmamba_track_maskbuffers ([:r]) topadded_viewto prevent length mismatch assertions during replay.python/sglang/srt/hf_transformers/config.py:text_config.is_heterogeneousviagetattr(..., False)and provided fallback toglobal_head_dim/num_global_key_value_headsfor backward compatibility with oldertransformersversions.Accuracy Tests
1. Unit & Metadata Tests
pytest test/registered/xpu/test_xpu_graph_metadata.py: 4 passedpytest test/registered/xpu/test_intel_xpu_linear_attn_dispatch.py: 4 passedpytest test/registered/xpu/test_xpu_basic.py: 1 passed2. Full Graph Replay & Cosine Similarity Correctness
pytest test/registered/xpu/test_xpu_graph.py -s -v: Passed (3 subtests passed)cos_sim > 0.999, greedy tokens exactly matched eager).cos_sim > 0.999, greedy tokens exactly matched eager).cos_sim > 0.999, greedy tokens exactly matched eager).Speed Tests and Profiling
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Missing
run-cilabel -- add it to run CI tests.Latest PR Test (Extra): ❌ Blocked --
run-ciis required first.Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: