Repository navigation
[Feat] [Intel] video_decoder: Decode video on the XPU media engine via torchcodec-xpu - #48074
Open
rahulvijayaraghavan wants to merge 4 commits into
Open
rahulvijayaraghavan wants to merge 4 commits into
rahulvijayaraghavan wants to merge 4 commits into
Conversation
rahulvijayaraghavan
requested review from
kpham-sgl,
pyc96 and
wisclmy0611
as code owners
November 3, 2026 05:04
Contributor
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
rahulvijayaraghavan
requested review from
JustinTong0323,
sogalin and
zijiexia
as code owners
November 3, 2026 05:04
This comment has been minimized.
This comment has been minimized.
rahulvijayaraghavan
marked this pull request as draft
November 3, 2026 07:20
Contributor
|
/tag-run-ci-label |
rahulvijayaraghavan
force-pushed
the
xpu-video-decode
branch
2 times, most recently
from
November 3, 2026 08:32
0c4e2f0 to
897bdd9
Compare
rahulvijayaraghavan
marked this pull request as ready for review
November 4, 2026 03:49
…a torchcodec-xpu Add an opt-in xpu branch to VideoDecoderWrapper and load_video (SGLANG_ENABLE_XPU_VIDEO_HW_DECODE=1) that decodes on the Intel GPU media engine through the torchcodec-xpu plugin.
rahulvijayaraghavan
force-pushed
the
xpu-video-decode
branch
from
November 4, 2026 03:49
897bdd9 to
3ca0775
Compare
mingfeima
reviewed
Nov 5, 2026
…ts VA-API media driver The PPA's iHD media driver needs gmmlib >= 22.10.1, so force-installing 22.10.0 removed it; take 22.10.2 from the compute-runtime 26.40 release, which also satisfies the pinned compute runtime.
rahulvijayaraghavan
requested review from
Fridge003,
HaiShaw,
ishandhanani,
ispobock and
yctseng0211
as code owners
November 5, 2026 10:20
…decode fallback warning The warning only said decoding failed; it now carries the torchcodec exception, so a missing media driver is visible in the log.
Open
3 of 5 tasks
… keeps its VA-API media driver" The XPU image fix moves to sgl-project#48661 (intel-omix channel); keep this PR out of docker/xpu.Dockerfile.
Collaborator
|
Please resolve the merge conflicts and rerun CI on the updated head once #48661 is included in the rebuilt XPU CI image. The latest runs were blocked by the actor cooldown, so the required tests did not execute. Please confirm that all five hardware-decode tests and the Qwen3-VL video e2e actually run and pass, alongside Base and Xeon CI. |
This was referenced Nov 6, 2026
3 of 5 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On Intel XPU, video inputs are decoded on the host CPU; this decodes them on the GPU media engine through the torchcodec-xpu plugin, as CUDA already does with NVDEC.
Usage
SGLANG_ENABLE_XPU_VIDEO_HW_DECODE=1(default0, opt-in): on XPU, with torchcodec-xpu installed, video inputs decode on the media engine and the frames stay on the XPU. Decoded frames need memory headroom on the card, so pick a--mem-fraction-staticthat leaves room. Without it, or on other platforms, decode stays on the CPU as before.Accuracy Tests
Decoded frames match CPU decode within colour-conversion rounding (max 2 levels, XPU hardware test). The Qwen3-VL video e2e test passes on XPU.
Speed Tests and Profiling
Intel Arc Pro B60, torch 2.14.0+xpu, torchcodec-xpu 0.17.0.
Prefill-bound models (Qwen3-VL, Qwen3-Omni) gain host CPU, not throughput; models with a short prefill (Gemma-4) gain end to end. Decoded frames use device memory outside
--mem-fraction-static, so the docs advise headroom on the decoding card.Related: #42627 (open) also adds torchcodec-xpu, pinned to 0.15.0 for torch 2.13; main needs 0.17.0. #44966 (open) moves device calls onto
current_platform, which_parallel_decodefollows here.Checklist
CI States
Latest PR Test (Base): ❌ Run #42036003411
Latest PR Test (Extra): ❌ Run #42036003094
Latest PR Test (AMD ROCm 10): ❌ Run #42036003077
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: