Capabilities

[pthatcherg]

We need an API for capabilities, like what MediaCapabilities has. Is that enough for us? What input do we need to provide to MediaCapabilities?

[steveanton]

We likely will need a surface to feature detect new encoder/decoder features as well. That doesn't seem like it would be covered by MediaCapabilities.

[chcunningham]

new encoder/decoder features

Steve, can you give an example?

[chcunningham]

@mounirlamouri fyi

[steveanton]

I don't think we've discussed how specific we want encoder/decoder settings yet, but here's some examples I could think of (some of this may be suitable for MediaCapabilities):

• How much latency an encoder requires (measured in frames)
• What kind of SVC/simulcast a video encoder/decoder supports
• Available encoder keyframe modes (e.g., does it support forcing keyframes)
• Support for explicitly tagged golden frames (I've heard some hardware decoders don't support this?)
• Supported encoding modes: VBR vs CBR vs ??
• Post processing features

Disclaimer: Not a codec expert

[markafoltz]

At first glance I don't think we need a different API surface from MediaCapabilities. MediaCapabilities already has feedback on both decode and encode capabilities.

How much latency an encoder requires (measured in frames)

Is this a capability or something that can be measured by observing the behavior of the encoder?

What kind of SVC/simulcast a video encoder/decoder supports

Is this contentType: "video/H264-SVC" or similar values?

Available encoder keyframe modes (e.g., does it support forcing keyframes)
Support for explicitly tagged golden frames (I've heard some hardware decoders don't support this?)
Supported encoding modes: VBR vs CBR vs ??

MediaCapabilities may need a VideoEncodingConfiguration dictionary to be able to query for specific encoder capabilities.

Post processing features

How is this related to encoding? Can you give specific examples?

[pthatcherg]

At first glance I don't think we need a different API surface from MediaCapabilities. MediaCapabilities already has feedback on both decode and encode capabilities.

Right. We want to use MediaCapabilities, but we may need things added to it.

How much latency an encoder requires (measured in frames)

Is this a capability or something that can be measured by observing the behavior of the encoder?

I think it's something that can be statically known and which you would like to know before creating the encoder, because there might be a range of delay options you can choose from and you might want to choose within that range.

What kind of SVC/simulcast a video encoder/decoder supports

Is this contentType: "video/H264-SVC" or similar values?

Available encoder keyframe modes (e.g., does it support forcing keyframes)
Support for explicitly tagged golden frames (I've heard some hardware decoders don't support this?)
Supported encoding modes: VBR vs CBR vs ??

MediaCapabilities may need a VideoEncodingConfiguration dictionary to be able to query for specific encoder capabilities.

Yes I think that would make sense.

Post processing features

How is this related to encoding? Can you give specific examples?

No, it only applies to decode, not encode.

[markafoltz]

Would the next step then to be to craft a PR on what changes we'd like to see to Media Capabilities (as part of the explainer in this repository)? I can take a stab at that.

[steveanton]

Yes, I think that would be a good next step. Thanks for helping.

[chcunningham]

Would the next step then to be to craft a PR on what changes we'd like to see to Media Capabilities (as part of the explainer in this repository)? I can take a stab at that.

Happy to review! You may also/instead want to file a bug with a rough outline of your proposed change. Guess it depends on how fast you are at spec-work (I'm slow) and how large the PR ends up being. Particularly for the big changes, discussion in an GH issue can be a bit easier and save you time re-writing spec-speak.

[chcunningham]

We've now added an IsConfigSupported() API for all encoders and decoders. Later integration with MC is not off the table, but not a priority at this time.

[aboba]

@steveanton WebCodecs doesn't explicitly support simulcast (e.g. multiple encoders need to be created). Question is whether explicit support is required or whether the implementation can optimize after noticing that there are multiple encodings utilizing the same track as input. SVC is supported using scalabilityMode values and metadata.

[chcunningham]

Triage note: marking 'extension', as proposal here would be to add new dictionary members to the adjacent MediaCapabilities API. This doesn't change my stance mentioned earlier: I think WebCodecs has now solved most of this problem with it's various IsConfigSupported() methods.

[aboba]

@dalecurtis Can we close this issue?

[Djuffin]

I think it is covered by isConfigSupported(). Closing it