WebCodec as pass through to application metadata

[youennf]

This issues stems from w3c/mediacapture-transform#3.
It might be good to have some application specific metadata be passed through without any change through encoder or decoder.
As an example, WebRTC encoded transforms could be used to retrieve some metadata, set them in encoded frame, (like head position or some other info). On the decoded frame, having the same metadata could be useful for JS-based post processing.

[alvestrand]

The metadata would want to be preserved unchanged through transforms. we would need some mechanism to ensure multiple users of the mechanism don't step on each other, but I think that's solvable.

[chcunningham]

I closed #182 as duplicate of this one, but pasting a bit of @aboba's opening comment here to unify the discussion:

The additional Metadata in question would be things like CSRCs and the SSRC which are RTP specific and therefore do not make sense in WebCodecs (but would be useful in WebRTC Insertable Streams). But it does raise issues relating to metadata mutability (See Issue #161).

[chcunningham]

Similar to above, closed #95 as duplicate of this one. Pasting some comments from that bug here to unify discussion:

@sandersdan wrote:

I think at this point I'm resigned to mapping input packets to output frames 1:1, with the caveat that some might get dropped or duplicated in corner cases.

Bikeshed: if we add a metadata object, we should recommend that specs use Symbol rather than String keys so that they can't conflict with app code.

[chcunningham]

For my part, I hear the use case. I think there are a number of ways to do this that aren't breaking changes. For instance, we could add a frozen Map to chunks/frames that is passed through the codec. Note Dan's points above on using Symbol and the implications for 1:1 mapping of inputs/outputs (intentionally not currently guaranteed by the spec).

My vote is to wait for specific metadata to be demanded and then we can decide where/how to add it on a case by case basis.

[alvestrand]

A frozen map of Symbol -> Object sounds ideal for the purposes I've been hearing about.

I think people can adapt to frames being dropped relatively easily (just send info relevant to that frame; state-changing data needs to be kept on a separate object and just referenced by the per-frame map). I also think that having the map be transported from "frame that triggered this frame emission" to "first (or last) frame that resulted from the same trigger" is fine.

It gets complex if you have a codec where you can't really tell which frames are triggered by which input frame, but I would think that those would be scarce.

[chcunningham]

Triage note: marking 'extension' as proposed direction simply adds attributes to existing interfaces.

[alvestrand]

Ping - this is relevant to, among others, Intel's "face detection" proposal. Do we have thoughts on bringing it forward?

[aboba]

It's quite appealing for applications to be able to add metadata to VideoFrames and encodedChunks. Of course, the encodedChunk metadata needs to be serialized on the wire or it would not be available to the decoder on the receiver side.

[youennf]

Some thoughts:

• Existing native objects that are close to VideoFrames (CVPixelBuffer for instance) do have a way to add metadata. We could learn from these existing APIs.
• Given the WebRTC use cases, metadata should be preserved when cloning/postMessaging.
• Structure cloning of metadata when setting/getting metadata would be a way to make sure provided metadata can be easily cloned/postMessaged. It would isolate native metadata processors from concurrent JS modifications.
• WebCodecs encoders/decoders might not need to pass through metadata, this can probably be handled by JS itself. But this might be useful in WebRTC contexts (MediaStreamTrack VideoFrames -> WebRTCEncodedTransform chunks).

[alvestrand]

For my erudition: Are you referring to https://developer.apple.com/documentation/corevideo/cvbuffer/3242646-attachments when you talk about the CVPixelBuffer metadata?

It's not obvious to me (I don't read that language easily) what type of attachment can be carried, but it seems to behave like a dictionary of objects (indexed by string). Having a generic "nonpropagated/propagated" property of the attachments seems like an interesting idea.

[aboba]

@Djuffin @padenot This issue has been open for more than 3 years; this makes it prime candidate for discussion at TPAC 2024.