Skip to content

[Feat] [Intel] video_decoder: Decode video on the XPU media engine via torchcodec-xpu - #48074

Open
rahulvijayaraghavan wants to merge 4 commits into
sgl-project:mainfrom
rahulvijayaraghavan:xpu-video-decode
Open

rahulvijayaraghavan wants to merge 4 commits into
sgl-project:mainfrom
rahulvijayaraghavan:xpu-video-decode

Conversation

@rahulvijayaraghavan

@rahulvijayaraghavan rahulvijayaraghavan commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

On Intel XPU, video inputs are decoded on the host CPU; this decodes them on the GPU media engine through the torchcodec-xpu plugin, as CUDA already does with NVDEC.

Usage

SGLANG_ENABLE_XPU_VIDEO_HW_DECODE=1 (default 0, opt-in): on XPU, with torchcodec-xpu installed, video inputs decode on the media engine and the frames stay on the XPU. Decoded frames need memory headroom on the card, so pick a --mem-fraction-static that leaves room. Without it, or on other platforms, decode stays on the CPU as before.

Accuracy Tests

Decoded frames match CPU decode within colour-conversion rounding (max 2 levels, XPU hardware test). The Qwen3-VL video e2e test passes on XPU.

Speed Tests and Profiling

Intel Arc Pro B60, torch 2.14.0+xpu, torchcodec-xpu 0.17.0.

Workload Hardware decode off On
Decode 60 frames of 720p, standalone 482 ms, 9.8 CPU-s 471 ms, 0.1 CPU-s
Qwen3-VL-2B, 32 video requests, conc 8: server CPU per request 22.8 s 7.8 s
Qwen3-VL-2B, conc 8: requests/s 0.27 0.24 (no gain; run-to-run noise 14%)
Gemma-4 E2B, conc 8: requests/s 1.25 1.63
Gemma-4 E2B, conc 8: TTFT p50 4.20 s 2.33 s
Qwen3-Omni-30B-A3B, TP 4, conc 1: tokenizer CPU per request 19.0 s 4.0 s
Qwen3-Omni-30B-A3B, TP 4, conc 1: requests/s 0.07 0.07

Prefill-bound models (Qwen3-VL, Qwen3-Omni) gain host CPU, not throughput; models with a short prefill (Gemma-4) gain end to end. Decoded frames use device memory outside --mem-fraction-static, so the docs advise headroom on the decoding card.

Related: #42627 (open) also adds torchcodec-xpu, pinned to 0.15.0 for torch 2.13; main needs 0.17.0. #44966 (open) moves device calls onto current_platform, which _parallel_decode follows here.

Checklist

  • Format your code according to the Format code with pre-commit.
  • Add unit tests according to the Run and add unit tests.
  • Update documentation according to Write documentations.
  • Provide accuracy and speed benchmark results.
  • Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): ❌ Run #42036003411
Latest PR Test (Extra): ❌ Run #42036003094
Latest PR Test (AMD ROCm 10): ❌ Run #42036003077

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mintlify

mintlify Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 8e1eb62.

Approve preview

@github-actions github-actions Bot added documentation Improvements or additions to documentation dependencies Pull requests that update a dependency file labels Nov 3, 2026
@github-actions

This comment has been minimized.

@rahulvijayaraghavan
rahulvijayaraghavan marked this pull request as draft November 3, 2026 07:20
@rahulvijayaraghavan rahulvijayaraghavan changed the title [Feat] [Intel] Decode video on the XPU media engine via via torchcodec-xpu [Feat] [Intel] video_decoder: Decode video on the XPU media engine via torchcodec-xpu Nov 3, 2026
@polisettyvarma

Copy link
Copy Markdown
Contributor

/tag-run-ci-label

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Nov 3, 2026
@rahulvijayaraghavan
rahulvijayaraghavan force-pushed the xpu-video-decode branch 2 times, most recently from 0c4e2f0 to 897bdd9 Compare November 3, 2026 08:32
@rahulvijayaraghavan
rahulvijayaraghavan marked this pull request as ready for review November 4, 2026 03:49
…a torchcodec-xpu

Add an opt-in xpu branch to VideoDecoderWrapper and load_video (SGLANG_ENABLE_XPU_VIDEO_HW_DECODE=1) that decodes on the Intel GPU media engine through the torchcodec-xpu plugin.
@mingfeima mingfeima added intel xpu intel gpu with device `torch.xpu` labels Nov 5, 2026
…ts VA-API media driver

The PPA's iHD media driver needs gmmlib >= 22.10.1, so force-installing 22.10.0 removed it; take 22.10.2 from the compute-runtime 26.40 release, which also satisfies the pinned compute runtime.
…decode fallback warning

The warning only said decoding failed; it now carries the torchcodec exception, so a missing media driver is visible in the log.
… keeps its VA-API media driver"

The XPU image fix moves to sgl-project#48661 (intel-omix channel); keep this PR out of docker/xpu.Dockerfile.
@mingfeima

Copy link
Copy Markdown
Collaborator

Please resolve the merge conflicts and rerun CI on the updated head once #48661 is included in the rebuilt XPU CI image. The latest runs were blocked by the actor cooldown, so the required tests did not execute. Please confirm that all five hardware-decode tests and the Qwen3-VL video e2e actually run and pass, alongside Base and Xeon CI.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation intel run-ci CI: run the baseline test suite on this PR xpu intel gpu with device `torch.xpu`

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants