VideoFrame support (WebCodecs) #1385
Description
Feature request
I noticed that the video examples use canvas.context.getImageData to get raw pixel values and construct a transformers.js RawImage. It might be possible to use VideoFrame instead.
VideoFrame can be constructed from a canvas, raw pixel data, or decoded from a video file via WebCodecs. It can then be copied into a GPUTexture via copyExternalImageToTexture that is then accessible by WebGPU, avoiding the CPU entirely. Or the raw pixel data can be copied from GPU -> CPU when using WASM.
Motivation
Accessing pixel data for video frames involves a copy from the GPU to the CPU, and when using the WebGPU backend, another copy from the CPU back to the GPU. This is a classic video performance issue and will hurt the frame rate, although I don't know by how much.
More info: https://developer.chrome.com/blog/from-webgl-to-webgpu#video_frame_processing
Your contribution
I don't know enough about the underlying ONNX runtime to know if this would work. If it sounds reasonable I could try hacking something together.