Announcement: Human face metadata entry to WebCodecs VideoFrame Metadata Registry

[ttoivone]

As per requirement of the VideoFrame Metadata Registry, this issue has been filed to allow discussion about the face metadata on being added into VideoFrameMetadata dictionary. The proposal is in the Media capture extensions PR #78 and the face detection explainer shall be updated.

To prevent discussions in multiple places, I do suggest that the discussion about the metadata is actually done in Media Capture Extensions Media capture extensions PR #78 where the explainer and spec will be updated, instead of this issue.

[aboba]

I'd prefer a more general approach, providing segmentation metadata rather than just metadata specific to face detection. The reason is that encoders can potentially use segmentation metadata to focus their efforts. So if we created face detection metadata, we might also need background blur metadata, whereas if we specify segmentation metadata, we could potentially use that for both face detection and background blur, as well as generic segmentation.

Related: w3c/mediacapture-extensions#79

[aboba]

FYI, the metadata proposal has been modified to make it more generic so as to make it more useful for segmentation.

This in turn raises the question as to how a WebCodecs Encoder could take advantage of the segmentation information (e.g. per-macroblock QP).

[chrisn]

One concern I have is that this adds face detection specific attributes under a top level segments entry, which is a rather generic name. It's still specific because of the SegmentType enum.

So I'd prefer to either rename segments to something indicating face detection as the source of the metadata, or relax the requirements to make SegmentType optional.

Aside from that, I'd be happy to review a PR that adds to the registry.

[youennf]

Discussed at TPAC:

• segments naming or SegmentType extension point should be discussed in WebRTC WG.
• above issue is not a blocker for the registry PR.

[Djuffin]

I've just noticed that this is already a part of Media Capture and Streams Extensions editor draft

w3c/mediacapture-extensions#78

[aboba]

@Djuffin If a registry entry would not affect the behavior of WebCodecs or any other MEDIA WG specification, I am wondering why defining the metadata entry in Media Capture and Streams Extensions isn't sufficient. Are we concerned about naming conflicts? That could be addressed without a registry. Or is the concern about poorly designed extensions? If the entries don't affect the behavior of WebCodecs or other MEDIA WG specs, why burden the MEDIA WG with the need to review other WG documents? The lack of convincing motivation could explain why no registry entries have been approved since the registry was created.

[padenot]

Monkey patching of other specs is bad practice and frowned upon, they should open a PR here and we should merge their stuff. Let's revert it from their spec.

[chrisn]

@padenot Do you think we should continue with the registry approach here or merge into WebCodecs spec?

[padenot]

The registry has all the right characteristics I think.

[chrisn]

AIUI the plan we agreed between WebRTC and Media WGs was to add the face detection metadata as a registry entry, so I'm not clear why that didn't happen and it was added directly to Media Capture and Streams Extensions.

[alvestrand]

The experience with use of registries is that when review is required, the effective use of registries requires speedy review to acceptance or (justified) rejection.

If registries don't have effective reviewers attached, people will search for other solutions to get their extensions published while they wait for the review to complete.