API for conversion between pixel formats

[chcunningham]

Currently the spec lists one format: I420. We'll want more, including some format to indicate "opaque", which will be used for GPU backed frames to avoid having to always copy to CPU memory.

Particularly for "opaque", we want an API to convert to another format where the planes can be read (performing the copy to CPU along the way). This API could also be used to convert between various non-opaque formats. Something like...

let frame = await gpu_frame.convertTo(I420);

[chcunningham]

The urgency of this API is reduced. Previously we expected this would be primary means to read back pixel data from GPU backed frames into the CPU space. Since then we've managed to add support (in chromium, not yet in the spec) for exposing pixel data from GPU backed video frames coming off of the users camera in the NV12 pixel format.

A conversion API is still interesting to pursue, but there are no urgent use cases that we are aware of (feel free to share some).

[padenot]

(in chromium, not yet in the spec)

#208, for reference, but this would still be good to have.

[chcunningham]

triage note: marking extension, as this would be a new API.

[alvestrand]

this means, of course, that any application that handles only a limited set of formats will be broken every time a new format is encountered. Seems a rough deal for app developers.

[sandersdan]

any application that handles only a limited set of formats will be broken every time a new format is encountered

There is a fallback path; you can createImageBitmap(videoFrame), then read back that ImageBitmap (presumably as RGBA).

This isn't great if the frame was mappable in the first place, but for GPU-backed frames it actually has acceptable performance.

[alvestrand]

So this means that we have a fallback (effectively "read as RGBA"). Apps can choose between using a fast path if they support the current format and a slow path if they don't. For a lot of cases, this will be sufficient.

[chcunningham]

Options for potential chromium impl in https://bugs.chromium.org/p/chromium/issues/detail?id=1139119