Skip to content

[Fix] [Intel] diffusion: Skip XPU component prefetch that would not fit - #48436

Open
dayanandav wants to merge 1 commit into
sgl-project:mainfrom
dayanandav:xpu-skip-prefetch-when-not-fit
Open

dayanandav wants to merge 1 commit into
sgl-project:mainfrom
dayanandav:xpu-skip-prefetch-when-not-fit

Conversation

@dayanandav

@dayanandav dayanandav commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

On XPU, component-offload prefetch is a synchronous load. When a component is kept for the next stage, the next component is loaded beside it. LongCat-Image keeps its 16 GiB Qwen2.5-VL encoder for text encoding and prefetches the 12 GiB DiT, which runs out of memory on a 24 GiB card even with every component offloaded.

Modifications

  • ComponentOffloadStrategy.prefetch_for_use: on XPU, skip the prefetch (warn once) when the component won't fit in free device memory; it then loads on first use.
  • Free memory comes from current_platform.get_available_gpu_memory; torch.xpu.mem_get_info reports free == total on torch 2.14.0+xpu.
  • Other backends unchanged.
  • Unit test for both branches (fits / does not fit).

Accuracy Tests

N/A: only changes when weights are moved to the device. The LongCat-Image output (1024x1024, 50 steps, seed 42) looks correct after the change.

Speed Tests and Profiling

LongCat-Image, --component-residency dit=component-offload text_encoder=component-offload vae=component-offload, 1024x1024, 50 steps, prompt rewrite on, no warmup, Intel XPU 24 GiB:

Setup Before After
1 card OOM 92.2 s
2 cards, --enable-cfg-parallel not run 56.6 s

Other backends take the same path as before, so their speed is unchanged.

Tests run on Intel XPU: test_component_residency.py, test_weight_snapshot.py, test_qwen_image21.py, test_text_encoding_cache.py, test_nvtx_pytorch_hooks.py, test_layerwise_offload.py (385 passed, 133 skipped). Not run on CUDA.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #41888677432
Latest PR Test (Extra): ❌ Run #41888677265
Latest PR Test (AMD ROCm 10): ❌ Run #41888677484

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

On XPU, component-offload prefetch is a synchronous load. When the previous
component is kept for the next stage, it loaded beside it and OOMed
LongCat-Image on a 24 GiB card. Skip the prefetch when the component would
not fit; it then loads on first use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dayanandav

Copy link
Copy Markdown
Contributor Author

@siju-samuel @ANSHUMAN87 help to review.

@dayanandav

Copy link
Copy Markdown
Contributor Author

@mickqian help to review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

diffusion SGLang Diffusion

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant