Skip to content

[Fix] [XPU] graph: Retain capture inputs and reset graphs in FullXPUGraphBackend - #48609

Open
CaoE wants to merge 3 commits into
sgl-project:mainfrom
CaoE:ecao/xpu_graph_improve_rebase
Open

CaoE wants to merge 3 commits into
sgl-project:mainfrom
CaoE:ecao/xpu_graph_improve_rebase

Conversation

@CaoE

@CaoE CaoE commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

In SGLang, device graphs capture and record raw GPU virtual addresses rather than retaining Python object references. During graph capture (e.g., under data parallelism where dummy batches instantiate capture-only metadata tensors such as global_num_tokens_gpu), these temporary tensors must be preserved across the graph's lifetime to prevent Python GC from collecting them. If collected, the PyTorch caching allocator may reassign their underlying memory while the graph actively replays, resulting in dangling pointers, silent memory corruption, or illegal memory access errors.

While BreakableCudaGraphBackend tracks and retains capture_inputs, FullXPUGraphBackend previously accepted the capture_inputs parameter in capture_one() but omitted storing it. Additionally, captured graphs were not explicitly reset upon backend cleanup().

Modifications

  1. Retain Capture Inputs in FullXPUGraphBackend (python/sglang/srt/hardware_backend/xpu/graph_runner/xpu_full_graph_backend.py):

    • Initialized self._capture_inputs: Dict[Any, Any] = {} in __init__.
    • Stored self._capture_inputs[shape_key] = capture_inputs in capture_one() so that capture-time tensor lifetimes are preserved across the backend lifecycle.
    • Cleared self._capture_inputs.clear() in cleanup().
  2. Explicitly Reset Graphs upon Cleanup (xpu_full_graph_backend.py):

    • Called graph.reset() on all captured XPUGraph instances during cleanup() to ensure allocated memory pool segments are cleanly released back to the driver.

Accuracy Tests

  • XPU Graph Metadata & Capture Inputs Retention Tests (test/registered/xpu/test_xpu_graph_metadata.py):

    • Verified test_full_xpu_graph_backend_retains_capture_inputs_and_cleans_up along with SWA page table and chunk-size rejection tests.
    • Result: 5 passed (3 subtests passed)
  • XPU Full Graph End-to-End Integration Tests (test/registered/xpu/test_xpu_graph.py):

    • Ran full graph prefill + decode capture & replay integration tests on 8x Intel Arc Pro B60 (Qwen2.5-1.5B-Instruct).
    • Validated greedy token matching across exact-bucket and padded workloads against eager execution.
    • Result: 1 passed (3 subtests passed)

Speed Tests and Profiling

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ✅ Run #42778802437
Latest PR Test (Extra): ❌ Run #42778802070
Latest PR Test (AMD ROCm 10): ✅ Run #42778802192

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@mintlify

mintlify Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 058e06a.

Approve preview

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 5, 2026
@CaoE CaoE changed the title [Fix] [XPU] xpu-graph: support XPU graph read-done events [Fix] [XPU] xpu-graph: Support read-done events and retain capture inputs in graph backend Nov 6, 2026
@CaoE
CaoE marked this pull request as ready for review November 6, 2026 05:59

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@CaoE

CaoE commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

/tag-run-ci-label

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Nov 6, 2026
@CaoE
CaoE force-pushed the ecao/xpu_graph_improve_rebase branch from f00d7ce to 434c983 Compare November 9, 2026 06:24
@CaoE CaoE changed the title [Fix] [XPU] xpu-graph: Support read-done events and retain capture inputs in graph backend [Fix] [XPU] graph: Retain capture inputs and reset graphs in FullXPUGraphBackend Nov 9, 2026
…raphBackend

XPU device graphs record raw device addresses rather than holding Python
reference counts. Retain capture_inputs in FullXPUGraphBackend to ensure
capture-time tensors (e.g. dummy batch metadata under data parallelism)
are preserved across the graph's lifetime rather than being collected by GC.
Also ensure captured graphs are cleanly reset upon backend cleanup.

Update XPU graph limitations in xpu.mdx.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@CaoE
CaoE force-pushed the ecao/xpu_graph_improve_rebase branch from fd6c435 to f91dd9b Compare November 9, 2026 07:39
@github-actions

This comment has been minimized.

airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 10, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 11, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 12, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 13, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 15, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 16, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 17, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant