Skip to content

Add SVC modes and metadata #187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 21, 2021
Merged

Add SVC modes and metadata #187

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5f87bcf
Interface changes for SVC and metadata
chcunningham Apr 21, 2021
9c321ad
Merge branch 'main' of https://github.com/w3c/webcodecs into svc_config
chcunningham Apr 30, 2021
d9eba76
make optional metadata default to empty
chcunningham Apr 30, 2021
18c3eea
Algorithm steps to populate Encoded*ChunkMetadata
chcunningham May 21, 2021
636611f
Merge remote-tracking branch 'origin/main' into svc_config
chcunningham May 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
193 changes: 143 additions & 50 deletions index.src.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,9 @@
type: dfn; text: natural dimensions; url: #natural-dimensions
type: dfn; text: natural width; url: #natural-width
type: dfn; text: natural height; url: #natural-height

spec: webrtc-svc; urlPrefix: https://w3c.github.io/webrtc-svc/
type: dfn; text: scalability mode identifier; url:#scalabilitymodes*
</pre>

<style>
Expand Down Expand Up @@ -123,6 +126,10 @@
objects. Such resources may be quickly exhuasted and should be released
immediately when no longer in use.

: <dfn>Temporal Layer</dfn>
:: A grouping of {{EncodedVideoChunk}}s who's timestamp cadence produces a
particular framerate. See {{VideoEncoderConfig/scalabilityMode}}.

: <dfn>Progressive Image</dfn>
:: An image that supports decoding to multiple levels of detail, with lower
levels becoming available while the encoded data is not yet fully buffered.
Expand Down Expand Up @@ -732,7 +739,9 @@
required WebCodecsErrorCallback error;
};

callback EncodedAudioChunkOutputCallback = undefined(EncodedAudioChunk output);
callback EncodedAudioChunkOutputCallback =
undefined (EncodedAudioChunk output,
optional EncodedAudioChunkMetadata metadata = {});
</xmp>

Internal Slots {#audioencoder-internal-slots}
Expand All @@ -744,6 +753,13 @@
<dd>Callback given at construction for encoded outputs.</dd>
<dt><dfn attribute for=AudioEncoder>[[error callback]]</dfn></dt>
<dd>Callback given at construction for encode errors.</dd>
<dt><dfn attribute for=AudioEncoder>[[active encoder config]]</dfn></dt>
<dd>The {{AudioEncoderConfig}} that is actively applied.</dd>
<dt><dfn attribute for=AudioEncoder>[[active output config]]</dfn></dt>
<dd>
The {{AudioDecoderConfig}} that describes how to decode the most recently
emitted {{EncodedAudioChunk}}.
</dd>
</dl>

Constructors {#audioencoder-constructors}
Expand All @@ -755,7 +771,9 @@
2. Assign `init.output` to the {{AudioEncoder/[[output callback]]}} internal slot.
3. Assign `init.error` to the {{AudioEncoder/[[error callback]]}} internal slot.
4. Assign "unconfigured" to `e.state`.
5. Return e.
5. Assign `null` to {{AudioEncoder/[[active encoder config]]}}.
6. Assign `null` to {{AudioEncoder/[[active output config]]}}.
7. Return e.

Attributes {#audioencoder-attributes}
-------------------------------------
Expand Down Expand Up @@ -801,7 +819,8 @@
{{AudioEncoder/[[codec implementation]]}} with an implementation
supporting |config|.
3. Otherwise, run the <a>Close AudioEncoder</a> algorithm with
{{NotSupportedError}}.
{{NotSupportedError}} and abort these steps.
4. Assign |config| to {{AudioEncoder/[[active encoder config]]}}
</dd>

<dt><dfn method for=AudioEncoder>encode(data)</dfn></dt>
Expand Down Expand Up @@ -920,17 +939,48 @@
|output|.
2. Let |chunk| be a new {{EncodedAudioChunk}} constructed with
|chunkInit|.
3. Invoke {{AudioEncoder/[[output callback]]}} with |chunk|.
3. Let |chunkMetadata| be a new {{EncodedAudioChunkMetadata}}.
4. Let |encoderConfig| be the
{{AudioEncoder/[[active encoder config]]}}.
5. Let |outputConfig| be a new {{AudioDecoderConfig}} that describes
|output|. Intialize |outputConfig| as follows:
1. Assign |encoderConfig|.{{AudioEncoderConfig/codec}} to
|outputConfig|.{{AudioDecoderConfig/codec}}.
2. Assign |encoderConfig|.{{AudioEncoderConfig/sampleRate}} to
|outputConfig|.{{AudioDecoderConfig/sampleRate}}.
3. Assign to
|encoderConfig|.{{AudioEncoderConfig/numberOfChannels}} to
|outputConfig|.{{AudioDecoderConfig/numberOfChannels}}.
4. Assign |outputConfig|.{{AudioDecoderConfig/description}} with a
sequence of codec specific bytes as determined by the
{{AudioEncoder/[[codec implementation]]}}. The user agent must
ensure that the provided description could be used to
correctly decode output.

NOTE: The codec specific requirements for populating the
{{AudioDecoderConfig/description}} are described in the
[[WEBCODECS-CODEC-REGISTRY]].

6. If |outputConfig| and {{AudioEncoder/[[active output config]]}} are
not [=equal dictionaries=]:
1. Assign |outputConfig| to
|chunkMetadata|.{{EncodedAudioChunkMetadata/decoderConfig}}.
2. Assign |outputConfig| to
{{AudioEncoder/[[active output config]]}}.
7. Invoke {{AudioEncoder/[[output callback]]}} with |chunk| and
|chunkMetadata|.
</dd>
<dt><dfn>Reset AudioEncoder</dfn></dt>
<dd>
Run these steps:
1. If {{AudioEncoder/state}} is `"closed"`, throw an {{InvalidStateError}}.
2. Set {{AudioEncoder/state}} to `"unconfigured"`.
3. Signal {{AudioEncoder/[[codec implementation]]}} to cease producing
3. Set {{AudioEncoder/[[active encoder config]]}} to `null`.
4. Set {{AudioEncoder/[[active output config]]}} to `null`.
5. Signal {{AudioEncoder/[[codec implementation]]}} to cease producing
output for the previous configuration.
4. [=Reset the control message queue=].
5. Set {{AudioEncoder/encodeQueueSize}} to zero.
6. [=Reset the control message queue=].
7. Set {{AudioEncoder/encodeQueueSize}} to zero.
</dd>
<dt><dfn>Close AudioEncoder</dfn> (with |error|)</dt>
<dd>
Expand All @@ -944,6 +994,23 @@
</dd>
</dl>

EncodedAudioChunkMetadata {#encoded-audio-chunk-metadata}
---------------------------------------------------------
The following metadata dictionary is emitted by the
{{EncodedVideoChunkOutputCallback}} alongside an associated
{{EncodedVideoChunk}}.

<xmp class='idl'>
dictionary EncodedAudioChunkMetadata {
AudioDecoderConfig decoderConfig;
};
</xmp>

: <dfn dict-member for=EncodedAudioChunkMetadata>decoderConfig</dfn>
:: A {{AudioDecoderConfig}} that authors may use to decode the associated
{{EncodedAudioChunk}}.


VideoEncoder Interface {#videoencoder-interface}
================================================

Expand All @@ -969,7 +1036,9 @@
required WebCodecsErrorCallback error;
};

callback EncodedVideoChunkOutputCallback = undefined(EncodedVideoChunk output, VideoDecoderConfig? output_config);
callback EncodedVideoChunkOutputCallback =
undefined (EncodedVideoChunk chunk,
optional EncodedVideoChunkMetadata metadata = {});
</xmp>

Internal Slots {#videoencoder-internal-slots}
Expand Down Expand Up @@ -1046,7 +1115,7 @@
supporting |config|.
3. Otherwise, run the <a>Close VideoEncoder</a> algorithm with
{{NotSupportedError}} and abort these steps.
2. Set {{VideoEncoder/[[active encoder config]]}} to `config`.
4. Assign |config| to {{VideoEncoder/[[active encoder config]]}}.
</dd>

<dt><dfn method for=VideoEncoder>encode(|frame|, |options|)</dfn></dt>
Expand Down Expand Up @@ -1155,45 +1224,7 @@
<dd>
Run these steps:
1. For each |output| in |outputs|:
1. Let |encoder_config| be the
{{VideoEncoder/[[active encoder config]]}}.

ISSUE: The intent is for |encoder_config| to be the
{{VideoEncoder/[[active encoder config]]}} that was used to
encode |output|. But, as written, it may occur that |output| was
encoded using a previous {{VideoEncoderConfig}} that has since
been replaced by a later call to {{VideoEncoder/configure()}}.
See [#138](https://github.com/w3c/webcodecs/issues/138).

2. Let |output_config| be a {{VideoDecoderConfig}} that describes
|output|. Initialize |output_config| as follows:
1. Assign `encoder_config.codec` to `output_config.codec`.
2. Assign `encoder_config.width` to `output_config.cropWidth`.
3. Assign `encoder_config.height` to `output_config.cropHeight`.
4. Assign `encoder_config.displayWidth` to
`output_config.displayWidth`.
5. Assign `encoder_config.displayHeight` to
`output_config.displayHeight`.
6. Assign the remaining keys of `output_config` as determined by
{{VideoEncoder/[[codec implementation]]}}. The user agent
must ensure that the configuration is completely described
such that |output_config| could be used to correctly decode
|output|.

NOTE: This includes supplying the
{{VideoDecoderConfig/description}} to describe codec
specific "extradata", the use of which may be further
described in codec registrations listed in the
[[WEBCODECS-CODEC-REGISTRY]].
3. If |output_config| and {{VideoEncoder/[[active output config]]}} are
<a>equal dictionaries</a>, set |output_config| to null. Otherwise,
set {{VideoEncoder/[[active output config]]}} to |output_config|.

NOTE: The {{VideoDecoderConfig}} |output_config| will be `null`
if the configuration hasn't changed from previous outputs. The
first output will always include a non-null |output_config|.

4. Let |chunkInit| be an {{EncodedVideoChunkInit}} with the following
1. Let |chunkInit| be an {{EncodedVideoChunkInit}} with the following
keys:
1. Let {{EncodedVideoChunkInit/data}} contain the encoded video data
from |output|.
Expand All @@ -1205,9 +1236,44 @@
4. Let {{EncodedVideoChunkInit/duration}} be the
{{VideoFrame/[[duration]]}} from the {{VideoFrame}} associated
with |output|.
5. Let |chunk| be a new {{EncodedVideoChunk}} constructed with
2. Let |chunk| be a new {{EncodedVideoChunk}} constructed with
|chunkInit|.
6. Invoke {{VideoEncoder/[[output callback]]}} with |chunk|.
3. Let |chunkMetadata| be a new {{EncodedVideoChunkMetadata}}.
4. Let |encoderConfig| be the
{{VideoEncoder/[[active encoder config]]}}.
5. Let |outputConfig| be a {{VideoDecoderConfig}} that describes
|output|. Initialize |outputConfig| as follows:
1. Assign `encoderConfig.codec` to `outputConfig.codec`.
2. Assign `encoderConfig.width` to `outputConfig.cropWidth`.
3. Assign `encoderConfig.height` to `outputConfig.cropHeight`.
4. Assign `encoderConfig.displayWidth` to
`outputConfig.displayWidth`.
5. Assign `encoderConfig.displayHeight` to
`outputConfig.displayHeight`.
6. Assign the remaining keys of `outputConfig` as determined by
{{VideoEncoder/[[codec implementation]]}}. The user agent
must ensure that the configuration is completely described
such that |outputConfig| could be used to correctly decode
|output|.

NOTE: The codec specific requirements for populating the
{{VideoDecoderConfig/description}} are described in the
[[WEBCODECS-CODEC-REGISTRY]].

6. If |outputConfig| and {{VideoEncoder/[[active output config]]}} are
not <a>equal dictionaries</a>:
1. Assign |outputConfig| to
|chunkMetadata|.{{EncodedVideoChunkMetadata/decoderConfig}}.
2. Assign |outputConfig| to
{{VideoEncoder/[[active output config]]}}.
7. If |encoderConfig|.{{VideoEncoderConfig/scalabilityMode}}
describes multiple [=temporal layers=]:
1. Let |temporal_layer_id| be the zero-based index describing the
temporal layer for |output|.
2. Assign |temporal_layer_id| to
|chunkMetadata|.{{EncodedVideoChunkMetadata/temporalLayerId}}.
8. Invoke {{VideoEncoder/[[output callback]]}} with |chunk| and
|chunkMetadata|.
</dd>
<dt><dfn>Reset VideoEncoder</dfn></dt>
<dd>
Expand All @@ -1233,6 +1299,27 @@
</dd>
</dl>

EncodedVideoChunkMetadata {#encoded-video-chunk-metadata}
---------------------------------------------------------
The following metadata dictionary is emitted by the
{{EncodedVideoChunkOutputCallback}} alongside an associated
{{EncodedVideoChunk}}.

<xmp class='idl'>
dictionary EncodedVideoChunkMetadata {
VideoDecoderConfig decoderConfig;
unsigned long temporalLayerId;
};
</xmp>

: <dfn dict-member for=EncodedVideoChunkMetadata>decoderConfig</dfn>
:: A {{VideoDecoderConfig}} that authors may use to decode the associated
{{EncodedVideoChunk}}.

: <dfn dict-member for=EncodedVideoChunkMetadata>temporalLayerId</dfn>
:: A number that identifying the [=temporal layer=] for the associated
{{EncodedVideoChunk}}.


Configurations{#configurations}
===============================
Expand Down Expand Up @@ -1569,6 +1656,7 @@
unsigned long displayWidth;
unsigned long displayHeight;
HardwareAcceleration hardwareAcceleration = "allow";
DOMString scalabilityMode;
};
</xmp>

Expand Down Expand Up @@ -1642,6 +1730,11 @@
Configures hardware acceleration for this codec. See
{{HardwareAcceleration}}.
</dd>

<dt><dfn dict-member for=VideoEncoderConfig>scalabilityMode</dfn></dt>
<dd>
An encoding [=scalability mode identifier=] as defined by [[WebRTC-SVC]].
</dd>
</dl>

Hardware Acceleration{#hardware-acceleration}
Expand Down