diff --git a/index.src.html b/index.src.html index 1fa9093a..c32b1e90 100644 --- a/index.src.html +++ b/index.src.html @@ -25,28 +25,26 @@ for: MediaSource; text: isTypeSupported(); url: #dom-mediasource-istypesupported spec: html; urlPrefix: https://html.spec.whatwg.org/multipage/; - type: method - for: HTMLMediaElement; text: canPlayType(); url: #dom-navigator-canplaytype - type: attribute - for: PlatformObject; text: [[Detached]]; url: structured-data.html#detached - type: attribute - for: ImageBitmap; text: resizeWidth; url:#dom-imagebitmapoptions-resizewidth - type: attribute - for: ImageBitmap; text: resizeHeight; url:#dom-imagebitmapoptions-resizeheight + for: HTMLMediaElement; + type: method; text: canPlayType(); url: #dom-navigator-canplaytype + for: PlatformObject; + type: attribute; text: [[Detached]]; url: structured-data.html#detached + for: ImageBitmap; + type: attribute; text: resizeWidth; url:#dom-imagebitmapoptions-resizewidth + type: attribute; text: resizeHeight; url:#dom-imagebitmapoptions-resizeheight + type: dfn; text: global object; url: webappapis.html#global-object spec: mediacapture-streams; urlPrefix: https://www.w3.org/TR/mediacapture-streams/ - type: method - for: mediaDevices; text: getUserMedia(); url: #dom-mediadevices-getusermedia + for: mediaDevices; + type: method; text: getUserMedia(); url: #dom-mediadevices-getusermedia spec: mediacapture-screen-share; urlPrefix: https://w3c.github.io/mediacapture-screen-share/ - type: method - for: mediaDevices; text: getDisplayMedia(); url: #dom-mediadevices-getdisplaymedia + for: mediaDevices; type: method; text: getDisplayMedia(); url: #dom-mediadevices-getdisplaymedia spec: mediacapture-main; urlPrefix: https://w3c.github.io/mediacapture-main/ - type: enum-value - for:MediaStreamTrackState; text: live; url: #idl-def-MediaStreamTrackState.live - type: enum-value - for:MediaStreamTrackState; text: ended; url: #idl-def-MediaStreamTrackState.ended + for:MediaStreamTrackState; + type: enum-value; text: live; url: #idl-def-MediaStreamTrackState.live + type: enum-value; text: ended; url: #idl-def-MediaStreamTrackState.ended spec: mimesniff; urlPrefix: https://mimesniff.spec.whatwg.org/# type: dfn; text: MIME type; url: mime-type @@ -56,6 +54,7 @@ type: dfn; text: enqueing; url: queue-enqueue; type: dfn; text: dequeued; url: queue-dequeue; type: dfn; text: empty; url: list-is-empty; + type: dfn; text: list; url: lists; spec: mediastream-recording; urlPrefix: https://www.w3.org/TR/mediastream-recording/# type: interface; text: MediaRecorder; url: mediarecorder @@ -69,51 +68,95 @@ Definitions {#definitions} ========================== -: Codec +: Codec :: Refers generically to an instance of AudioDecoder, AudioEncoder, VideoDecoder, or VideoEncoder. : Key Frame :: An encoded frame that does not depend on any other frames for decoding. +: Internal Pending Output +:: Codec outputs such as {{VideoFrame}}s that currently reside in the internal + pipeline of the underlying codec implementation. The underlying codec + implementation may emit new outputs only when a new inputs are provided. The + underlying codec implementation must emit all outputs in response to a + flush. + +: Codec System Resources +:: Resources including CPU memory, GPU memory, and exclusive handles to specific + decoding/encoding hardware that may be allocated by the User Agent as part + of codec configuration or generation of {{AudioFrame}} and {{VideoFrame}} + objects. Such resources may be quickly exhuasted and should be released + immediately when no longer in use. Codec Processing Model {#codec-processing-model} ================================================ -New codec tasks may be scheduled while previous tasks are still pending. For -example, web authors may call `decode()` without waiting for the previous -`decode()` to generate an output. This is facilitated by the following -mechanisms. +Background {#processing-model-background} +----------------------------------------- -Each codec has a single control message queue that is a queue -of control messages. +This section is non-normative. -Queuing a control - message means enqueing the message to the codec’s control - message queue. Invoking codec methods will often queue a control message - to schedule work. +The codec interfaces defined by the specification designed such that new codec +tasks may be scheduled while previous tasks are still pending. For example, web +authors may call `decode()` without waiting for a previous `decode()` to +complete. This is achieved by offloading underlying codec tasks to a separate +thread for parallel execution. -Running a control - message means executing a sequence of steps specified by the method - that enqueued the message. +This section describes threading behaviors as they are visible from the +perspective of web authors. Implementers may choose to use more or less threads +as long the exernally visible behaviors of blocking and sequencing are +maintained as follows. + +Control Thread and Codec Thread {#control-thread-and-codec-thread} +------------------------------------------------------------------ + +All steps in this specificaiton will run on either a [=control thread=] or +a [=codec thread=]. -Running the control message - processing loop means executing these steps. -1. While the control message queue is not empty - 1. Let |front message| be the next dequeued control message - 2. If |front message| cannot be executed now, return. +The control thread is the thread from which authors will construct +a [=codec=] and invoke its methods. Invoking a codec's methods will typically +result in the creation of [=control messages=] which are later executed on the +[=codec thread=]. Each [=global object=] has a separate control thread. - The User Agent must decide when further processing is blocked because of - ongoing work as an implementation detail (e.g. the underlying - decoder cannot accept more requests yet). The UA must restart the - processing loop when the blockage is resolved. +The codec thread is the thread from which a [=codec=] will +[=dequeue=] [=control messages=] and execute their steps. Each [=codec=] +instance has a separate codec thread. The lifetime of a codec thread matches +that of its associated [=codec=] instance. - NOTE: a blocked processing loop is visible to authors via the - `decodeQueueSize` and `encodeQueueSize` attributes. +The [=control thread=] uses a traditional event loop, as described in [[!HTML]]. - 3. Dequeue |front message| from the control message queue. - 4. Run the |front message| control message steps. +The [=codec thread=] uses a specialized [=codec processing loop=]. +Communication from the [=control thread=] to the [=codec thread=] is done using +[=control message=] passing. Communication in the other direction is done using +regular event loop tasks. + +Each [=codec=] instance has a single control message queue that is +a [=queue=] of control messages. + +Queuing a control +message means [=enqueing=] the message to a [=codec=]’s [=control +message queue=]. Invoking codec methods will often queue a control message +to schedule work. + +Running a control +message means performing a sequence of steps specified by the method +that enqueued the message. The steps of a control message may depend on +injected state, supplied by the method that enqueued the message. + +Resetting the control message + queue means performing these steps: +1. For each [=control message=] in the [=control message queue=]: + 1. If a control message's [=injected state=] includes a promise, reject + that promise. + 2. Remove the message from the queue. + +The codec processing loop must run these steps: +1. While true: + 1. If the [=control message queue=] is emtpy, [=continue=]. + 2. Dequeue |front message| from the [=control message queue=]. + 3. Run [=control message steps=] described by |front message|. AudioDecoder Interface {#audiodecoder-interface} ================================================ @@ -176,8 +219,8 @@ decodeQueueSize
- The number of pending decode requests. This does not include requests that - have been sent to the underlying codec. + The number of pending decode requests. This number will decrease as the + underlying codec is ready to accept new input.
@@ -186,50 +229,123 @@
configure(config)
- Enqueues a control message to configure the audio decoder for - decoding chunks as described by |config|. + [=Enqueues a control message=] to configure the audio decoder for decoding + chunks as described by |config|. When invoked, run these steps: - 1. If |config| is not a valid AudioDecoderConfig, throw a + 1. If |config| is not a [=valid AudioDecoderConfig=], throw a {{TypeError}}. - 2. Run the Configure Decoder algorithm with |config|. + 2. If {{AudioDecoder/state}} is `"closed"`, throw an {{InvalidStateError}}. + 3. If the user agent cannot provide a codec implementation to support + |config|, throw a {{NotSupportedError}}. + 4. Set {{AudioDecoder/state}} to `"configured"`. + 5. [=Queue a control message=] to configure the decoder with |config|. + + [=Running a control message=] to configure the decoder means running + these steps: + 1. Assign {{AudioDecoder/[[codec implementation]]}} with an implementation + supporting |config|.
decode(chunk)
- Enqueues a control message to decode the given |chunk|. + [=Enqueues a control message=] to decode the given |chunk|. When invoked, run these steps: - 1. Let |output algorithm| be the AudioFrame Output algorithm. - 2. Run the Decode Chunk algorithm with |chunk| and - |output algorithm|. + 1. If {{VideoDecoder/state}} is not `"configured"`, throw an + {{InvalidStateError}}. + 2. Increment {{VideoDecoder/decodeQueueSize}}. + 3. [=Queue a control message=] to decode the |chunk|. + + Running a control message to decode the chunk means performing these steps: + 1. Attempt to use {{VideoDecoder/[[codec implementation]]}} to decode the + chunk. + 2. If decoding results in an error, queue a task on the [=control thread=] + event loop to run the [=Close VideoDecoder=] algorithm with + {{EncodingError}}. + 3. Queue a task on the [=control thread=] event loop to decrement + {{VideoDecoder/decodeQueueSize}} + 4. Let |decoded outputs| be a [=list=] of decoded video data outputs emitted + by {{VideoDecoder/[[codec implementation]]}}. + 5. If |decoded outputs| is not empty, queue a task on the [=control thread=] + event loop to run the [=Output VideoFrames=] algorithm with + |decoded outputs|.
flush()
- Completes all control messages in the control message queue + Completes all [=control messages=] in the [=control message queue=] and emits all outputs. When invoked, run these steps: - 1. Let |output algorithm| be the AudioFrame Output algorithm. - 2. Run the Flush algorithm with |output algorithm|. + 1. If {{AudioDecoder/state}} is not `"configured"`, return + [=a promise rejected with=] {{InvalidStateError}} {{DOMException}}. + 2. Let |promise| be a new Promise. + 3. [=Queue a control message=] to flush the codec with |promise|. + 4. Return |promise|. + + Running a control message to flush the codec means performing these steps + with |promise|. + 1. Signal {{AudioDecoder/[[codec implementation]]}} to emit all [=internal + pending outputs=]. + 2. Let |decoded outputs| be a [=list=] of decoded audio data outputs emitted + by {{AudioDecoder/[[codec implementation]]}}. + 3. If |decoded outputs| is not empty, queue a task on the [=control thread=] + event loop to run the [=Output AudioFrames=] algorithm with + |decoded outputs|. + 4. Queue a task on the [=control thread=] event loop to resolve |promise|.
reset()
Immediately resets all state including configuration, - control messages in the control message queue, and all pending + [=control messages=] in the [=control message queue=], and all pending callbacks. - When invoked, run the Reset algorithm. + When invoked, run the [=Reset AudioDecoder=] algorithm.
close()
- Immediately aborts all pending work and releases system resources. Close is - permanent. + Immediately aborts all pending work and releases [=system resources=]. + Close is final. - When invoked, run the Close algorithm. + When invoked, run the [=Close AudioDecoder=] algorithm. +
+
+ +Algorithms {#audiodecoder-algorithms} +------------------------------------- +
+
Output AudioFrames (with |outputs|)
+
+ Run these steps: + 1. For each |output| in |outputs|: + 1. Let |buffer| be an {{AudioBuffer}} containing the decoded audio data in + |output|. + 2. Let |frame| be an {{AudioFrame}} containing |buffer| and a timestamp for + the output. + 3. Invoke {{AudioDecoder/[[output callback]]}} with frame. +
+
Reset AudioDecoder
+
+ Run these steps: + 1. If {{AudioDecoder/state}} is `"closed"`, throw an {{InvalidStateError}}. + 2. Set {{AudioDecoder/state}} to `"unconfigured"`. + 3. Signal {{AudioDecoder/[[codec implementation]]}} to cease producing + output for the previous configuration. + 4. [=Reset the control message queue=]. + 5. Set {{AudioDecoder/decodeQueueSize}} to zero. +
+
Close AudioDecoder (with error)
+
+ Run these steps: + 1. Run the [=Reset AudioDecoder=] algorithm. + 2. Set {{AudioDecoder/state}} to `"closed"`. + 3. Clear {{AudioDecoder/[[codec implementation]]}} and release associated + [=system resources=]. + 4. If |error| is set, queue a task on the [=control thread=] event loop to + invoke the {{AudioDecoder/[[error callback]]}} with |error|.
@@ -294,8 +410,8 @@ decodeQueueSize
- The number of pending decode requests. This does not include requests that - have been sent to the underlying codec. + The number of pending decode requests. This number will decrease as the + underlying codec is ready to accept new input.
@@ -304,50 +420,140 @@
configure(config)
- Enqueues a control message to configure the video decoder for - decoding chunks as described by |config|. + [=Enqueues a control message=] to configure the video decoder for decoding + chunks as described by |config|. When invoked, run these steps: - 1. If |config| is not a valid VideoDecoderConfig, throw a + 1. If |config| is not a [=valid VideoDecoderConfig=], throw a {{TypeError}}. - 2. Run the Configure Decoder algorithm with |config|. + 2. If {{VideoDecoder/state}} is `"closed"`, throw an {{InvalidStateError}}. + 3. If the user agent cannot provide a codec implementation to support + |config|, throw a {{NotSupportedError}}. + 4. Set {{VideoDecoder/state}} to `"configured"`. + 5. [=Queue a control message=] to configure the decoder with |config|. + + [=Running a control message=] to configure the decoder means running + these steps: + 1. Assign {{VideoDecoder/[[codec implementation]]}} with an implementation + supporting |config|.
decode(chunk)
- Enqueues a control message to decode the given |chunk|. + [=Enqueues a control message=] to decode the given |chunk|. When invoked, run these steps: - 1. Let |output algorithm| be the VideoFrame Output algorithm. - 2. Run the Decode Chunk algorithm with |chunk| and - |output algorithm|. + 1. If {{VideoDecoder/state}} is not `"configured"`, throw an + {{InvalidStateError}}. + 2. Increment {{VideoDecoder/decodeQueueSize}}. + 3. [=Queue a control message=] to decode the |chunk|. + + Running a control message to decode the chunk means performing these steps: + 1. Attempt to use {{VideoDecoder/[[codec implementation]]}} to decode the + chunk. + 2. If decoding results in an error, queue a task on the [=control thread=] + event loop to run the [=Close VideoDecoder=] algorithm with + {{EncodingError}}. + 3. Queue a task on the [=control thread=] event loop to decrement + {{VideoDecoder/decodeQueueSize}} + 4. Let |decoded outputs| be a [=list=] of decoded video data outputs emitted + by {{VideoDecoder/[[codec implementation]]}}. + 5. If |decoded outputs| is not empty, queue a task on the [=control thread=] + event loop to run the [=Output VideoFrames=] algorithm with + |decoded outputs|.
flush()
- Completes all control messages in the control message queue + Completes all [=control messages=] in the [=control message queue=] and emits all outputs. When invoked, run these steps: - 1. Let |output algorithm| be the VideoFrame Output algorithm. - 2. Run the Flush algorithm with |output algorithm|. + 1. If {{VideoDecoder/state}} is not `"configured"`, return + [=a promise rejected with=] {{InvalidStateError}} {{DOMException}}. + 2. Let |promise| be a new Promise. + 3. [=Queue a control message=] to flush the codec with |promise|. + 4. Return |promise|. + + Running a control message to flush the codec means performing these steps + with |promise|. + 1. Signal {{VideoDecoder/[[codec implementation]]}} to emit all [=internal + pending outputs=]. + 2. Let |decoded outputs| be a [=list=] of decoded video data outputs emitted + by {{VideoDecoder/[[codec implementation]]}}. + 3. If |decoded outputs| is not empty, queue a task on the [=control thread=] + event loop to run the [=Output VideoFrames=] algorithm with + |decoded outputs|. + 4. Queue a task on the [=control thread=] event loop to resolve |promise|.
reset()
Immediately resets all state including configuration, - control messages in the control message queue, and all pending + [=control messages=] in the [=control message queue=], and all pending callbacks. - When invoked, run the Reset algorithm. + When invoked, run the [=Reset VideoDecoder=] algorithm.
close()
- Immediately aborts all pending work and releases system resources. Close is - permanent. + Immediately aborts all pending work and releases [=system resources=]. + Close is final. + + When invoked, run the [=Close VideoDecoder=] algorithm. +
+
- When invoked, run the Close algorithm. +Algorithms {#videodecoder-algorithms} +------------------------------------- +
+
Output VideoFrames (with |outputs|)
+
+ Run these steps: + 1. For each |output| in |outputs|: + 1. Let |planes| be a sequence of {{Plane}}s containing the decoded video + frame data from |output|. + 2. Let |pixelFormat| be the {{PixelFormat}} of |planes|. + 3. Let |frameInit| be a {{VideoFrameInit}} with the following keys: + 1. Let {{VideoFrameInit/timestamp}} and {{VideoFrameInit/duration}} be + the {{EncodedVideoChunk/timestamp}} and + {{EncodedVideoChunk/duration}} from the {{EncodedVideoChunk}} + associated with |output|. + 2. Let {{VideoFrameInit/codedWidth}} and + {{VideoFrameInit/codedHeight}} + be the width and height of the decoded video frame |output| in + pixels, prior to any cropping or aspect ratio adjustments. + 3. Let {{VideoFrameInit/cropLeft}}, {{VideoFrameInit/cropTop}}, + {{VideoFrameInit/cropWidth}}, and {{VideoFrameInit/cropHeight}} be + the crop region of the decoded video frame |output| in pixels, + prior to any aspect ratio adjustments. + 4. Let {{VideoFrameInit/displayWidth}} and + {{VideoFrameInit/displayHeight}} be the display size of the + decoded video frame in pixels. + 4. Let |frame| be a {{VideoFrame}}, constructed with |pixelFormat|, + |planes|, and |frameInit|. + 5. Invoke {{VideoDecoder/[[output callback]]}} with |frame|. +
+
Reset VideoDecoder
+
+ Run these steps: + 1. If {{VideoDecoder/state}} is `"closed"`, throw an {{InvalidStateError}}. + 2. Set {{VideoDecoder/state}} to `"unconfigured"`. + 3. Signal {{VideoDecoder/[[codec implementation]]}} to cease producing + output for the previous configuration. + 4. [=Reset the control message queue=]. + 5. Set {{VideoDecoder/decodeQueueSize}} to zero. +
+
Close VideoDecoder (with |error|)
+
+ Run these steps: + 1. Run the [=Reset VideoDecoder=] algorithm. + 2. Set {{VideoDecoder/state}} to `"closed"`. + 3. Clear {{VideoDecoder/[[codec implementation]]}} and release associated + [=system resources=]. + 4. If |error| is set, queue a task on the [=control thread=] event loop to + invoke the {{VideoDecoder/[[error callback]]}} with |error|.
@@ -360,8 +566,10 @@ [Exposed=(Window,Worker)] interface AudioEncoder { constructor(AudioEncoderInit init); + readonly attribute CodecState state; readonly attribute long encodeQueueSize; + undefined configure(AudioEncoderConfig config); undefined encode(AudioFrame frame); Promise flush(); @@ -411,8 +619,8 @@ encodeQueueSize
- The number of pending encode requests. This does not include requests that - have been sent to the underlying codec. + The number of pending encode requests. This number will decrease as the + underlying codec is ready to accept new input.
@@ -421,58 +629,138 @@
configure(config)
- Enqueues a control message to configure the audio encoder for + [=Enqueues a control message=] to configure the audio encoder for decoding chunks as described by |config|. When invoked, run these steps: - 1. If |config| is not a valid AudioEncoderConfig, throw a + 1. If |config| is not a [=valid AudioEncoderConfig=], throw a {{TypeError}}. - 2. Run the Configure Encoder algorithm with |config|. + 2. If {{AudioEncoder/state}} is `"closed"`, throw an {{InvalidStateError}}. + 3. If the user agent cannot provide a codec implementation to support + |config|, throw a {{NotSupportedError}}. + 4. Set {{AudioEncoder/state}} to `"configured"`. + 5. [=Queue a control message=] to configure the encoder using |config|. + + Running a control message to configure the encoder means performing these + steps: + 1. Assign {{AudioEncoder/[[codec implementation]]}} with an implementation + supporting |config|.
encode(frame)
- Enqueues a control message to encode the given |frame|. + [=Enqueues a control message=] to encode the given |frame|. - NOTE: This method will destroy the VideoFrame. Authors who wish to retain a + NOTE: This method will destroy the AudioFrame. Authors who wish to retain a copy, should call `frame.clone()` prior to calling encode(). When invoked, run these steps: 1. If the value of |frame|'s {{AudioFrame/[[detached]]}} internal slot is `true`, throw a {{TypeError}}. - 2. Let |output algorithm| be the EncodedAudioChunk Output algorithm. - 3. Run the Encode Frame algorithm with |frame| and - |output algorithm|. + 2. If {{AudioEncoder/state}} is not `"configured"`, throw an + {{InvalidStateError}}. + 3. Let |frameClone| hold the result of running the [=Clone Frame=] + algorithm with |frame|. + 4. Destroy the original |frame| by invoking `frame.destroy()`. + 5. Increment {{AudioEncoder/encodeQueueSize}}. + 6. [=Queue a control message=] to encode |frameClone|. + + Running a control message to encode the frame means performing these steps. + 1. Attempt to use {{AudioEncoder/[[codec implementation]]}} to encode + |frameClone|. + 2. If encoding results in an error, queue a task on the [=control thread=] + event loop to run the [=Close AudioEncoder=] algorithm with + {{EncodingError}}. + 3. Queue a task on the [=control thread=] event loop to decrement + {{AudioEncoder/encodeQueueSize}}. + 4. Let |encoded outputs| be a [=list=] of encoded audio data outputs + emitted by {{AudioEncoder/[[codec implementation]]}}. + 5. If |encoded outputs| is not empty, queue a task on the [=control thread=] + event loop to run the [=Output EncodedAudioChunks=] algorithm with + |encoded outputs|.
flush()
- Completes all control messages in the control message queue + Completes all [=control messages=] in the [=control message queue=] and emits all outputs. When invoked, run these steps: - 1. Let |output algorithm| be the EncodedAudioChunk Output algorithm. - 2. Run the Flush algorithm with |output algorithm|. + 1. If {{AudioEncoder/state}} is not `"configured"`, return + [=a promise rejected with=] {{InvalidStateError}} {{DOMException}}. + 2. Let |promise| be a new Promise. + 3. [=Queue a control message=] to flush the codec with |promise|. + 4. Return |promise|. + + Running a control message to flush the codec means performing these steps + with |promise|. + 1. Signal {{AudioEncoder/[[codec implementation]]}} to emit all [=internal + pending outputs=]. + 2. Let |encoded outputs| be a [=list=] of encoded audio data outputs + emitted by {{AudioEncoder/[[codec implementation]]}}. + 5. If |encoded outputs| is not empty, queue a task on the [=control thread=] + event loop to run the [=Output EncodedAudioChunks=] algorithm with + |encoded outputs|. + 3. Queue a task on the [=control thread=] event loop to resolve |promise|.
reset()
Immediately resets all state including configuration, - control messages in the control message queue, and all pending + [=control messages=] in the [=control message queue=], and all pending callbacks. - When invoked, run the Reset algorithm. + When invoked, run the [=Reset AudioEncoder=] algorithm.
close()
- Immediately aborts all pending work and releases system resources. Close is - permanent. + Immediately aborts all pending work and releases [=system resources=]. + Close is final. - When invoked, run the Close algorithm. + When invoked, run the [=Close AudioEncoder=] algorithm.
+Algorithms {#audioencoder-algorithms} +------------------------------------- +
+
Output EncodedAudioChunks (with |outputs|)
+
+ Run these steps: + 1. For each |output| in |outputs|: + 1. Let |chunkInit| be an {{EncodedAudioChunkInit}} with the following keys: + 1. Let {{EncodedAudioChunkInit/data}} contain the encoded audio data + from |output|. + 2. Let {{EncodedAudioChunkInit/type}} be the {{EncodedAudioChunkType}} + of |output|. + 3. Let {{EncodedAudioChunkInit/timestamp}} be the + {{AudioFrame/timestamp}} from the AudioFrame associated with + |output|. + 2. Let |chunk| be a new {{EncodedAudioChunk}} constructed with |chunkInit|. + 3. Invoke {{AudioEncoder/[[output callback]]}} with |chunk|. +
+
Reset AudioEncoder
+
+ Run these steps: + 1. If {{AudioEncoder/state}} is `"closed"`, throw an {{InvalidStateError}}. + 2. Set {{AudioEncoder/state}} to `"unconfigured"`. + 3. Signal {{AudioEncoder/[[codec implementation]]}} to cease producing + output for the previous configuration. + 4. [=Reset the control message queue=]. + 5. Set {{AudioEncoder/encodeQueueSize}} to zero. +
+
Close AudioEncoder (with |error|)
+
+ Run these steps: + 1. Run the [=Reset AudioEncoder=] algorithm. + 2. Set {{AudioEncoder/state}} to `"closed"`. + 3. Clear {{AudioEncoder/[[codec implementation]]}} and release associated + [=system resources=]. + 4. If |error| is set, queue a task on the [=control thread=] event loop + invoke the {{AudioEncoder/[[error callback]]}} with |error|. +
+
VideoEncoder Interface {#videoencoder-interface} ================================================ @@ -482,8 +770,10 @@ [Exposed=(Window,Worker)] interface VideoEncoder { constructor(VideoEncoderInit init); + readonly attribute CodecState state; readonly attribute long encodeQueueSize; + undefined configure(VideoEncoderConfig config); undefined encode(VideoFrame frame, optional VideoEncoderEncodeOptions options = {}); Promise flush(); @@ -533,8 +823,8 @@ encodeQueueSize
- The number of pending encode requests. This does not include requests that - have been sent to the underlying codec. + The number of pending encode requests. This number will decrease as the + underlying codec is ready to accept new input.
@@ -543,18 +833,27 @@
configure(config)
- Enqueues a control message to configure the video encoder for + [=Enqueues a control message=] to configure the video encoder for decoding chunks as described by |config|. When invoked, run these steps: - 1. If |config| is not a valid VideoEncoderConfig, throw a + 1. If |config| is not a [=valid VideoEncoderConfig=], throw a {{TypeError}}. - 2. Run the Configure Encoder algorithm with |config|. + 2. If {{VideoEncoder/state}} is `"closed"`, throw an {{InvalidStateError}}. + 3. If the user agent cannot provide a codec implementation to support + |config|, throw a {{NotSupportedError}}. + 4. Set {{VideoEncoder/state}} to `"configured"`. + 5. [=Queue a control message=] to configure the encoder using |config|. + + Running a control message to configure the encoder means performing these + steps: + 1. Assign {{VideoEncoder/[[codec implementation]]}} with an implementation + supporting |config|.
encode(frame, options)
- Enqueues a control message to encode the given |frame|. + [=Enqueues a control message=] to encode the given |frame|. NOTE: This method will destroy the VideoFrame. Authors who wish to retain a copy, should call `frame.clone()` prior to calling encode(). @@ -562,230 +861,113 @@ When invoked, run these steps: 1. If the value of |frame|'s {{VideoFrame/[[detached]]}} internal slot is `true`, throw a {{TypeError}}. - 2. Let |output algorithm| be the EncodedVideoChunk Output algorithm. - 3. Run the Encode Frame algorithm with |frame| and - |output algorithm|. + 2. If {{VideoEncoder/state}} is not `"configured"`, throw an + {{InvalidStateError}}. + 3. Let |frameClone| hold the result of running the [=Clone Frame=] + algorithm with |frame|. + 4. Destroy the original |frame| by invoking `frame.destroy()`. + 5. Increment {{VideoEncoder/encodeQueueSize}}. + 6. [=Queue a control message=] to encode |frameClone|. + + Running a control message to encode the frame means performing these steps. + 1. Attempt to use {{VideoEncoder/[[codec implementation]]}} to encode + |frameClone| according to |options|. + 2. If encoding results in an error, queue a task on the [=control thread=] + event loop to run the [=Close VideoEncoder=] algorithm with + {{EncodingError}}. + 3. Queue a task on the [=control thread=] event loop to decrement + {{VideoEncoder/encodeQueueSize}}. + 4. Let |encoded outputs| be a [=list=] of encoded video data outputs + emitted by {{VideoEncoder/[[codec implementation]]}}. + 5. If |encoded outputs| is not empty, queue a task on the [=control thread=] + event loop to run the [=Output EncodedVideoChunks=] algorithm with + |encoded outputs|.
flush()
- Completes all control messages in the control message queue + Completes all [=control messages=] in the [=control message queue=] and emits all outputs. When invoked, run these steps: - 1. Let |output algorithm| be the EncodedVideoChunk Output algorithm. - 2. Run the Flush algorithm with |output algorithm|. + 1. If {{VideoEncoder/state}} is not `"configured"`, return + [=a promise rejected with=] {{InvalidStateError}} {{DOMException}}. + 2. Let |promise| be a new Promise. + 3. [=Queue a control message=] to flush the codec with |promise|. + 4. Return |promise|. + + Running a control message to flush the codec means performing these steps + with |promise|. + 1. Signal {{VideoEncoder/[[codec implementation]]}} to emit all [=internal + pending outputs=]. + 2. Let |encoded outputs| be a [=list=] of encoded video data outputs + emitted by {{VideoEncoder/[[codec implementation]]}}. + 5. If |encoded outputs| is not empty, queue a task on the [=control thread=] + event loop to run the [=Output EncodedVideoChunks=] algorithm with + |encoded outputs|. + 3. Queue a task on the [=control thread=] event loop to resolve |promise|.
reset()
Immediately resets all state including configuration, - control messages in the control message queue, and all pending + [=control messages=] in the [=control message queue=], and all pending callbacks. - When invoked, run the Reset algorithm. + When invoked, run the [=Reset VideoEncoder=] algorithm.
close()
- Immediately aborts all pending work and releases system resources. Close is - permanent. + Immediately aborts all pending work and releases [=system resources=]. + Close is final. - When invoked, run the Close algorithm. + When invoked, run the [=Close VideoEncoder=] algorithm.
+Algorithms {#videoencoder-algorithms} +------------------------------------- +
+
Output EncodedVideoChunks (with |outputs|)
+
+ Run these steps: + 1. For each |output| in |outputs|: + 1. Let |chunkInit| be an {{EncodedVideoChunkInit}} with the following keys: + 1. Let {{EncodedVideoChunkInit/data}} contain the encoded video data + from |output|. + 2. Let {{EncodedVideoChunkInit/type}} be the {{EncodedVideoChunkType}} + of |output|. + 3. Let {{EncodedVideoChunkInit/timestamp}} be the + {{VideoFrame/timestamp}} from the {{VideoFrame}} associated with + |output|. + 4. Let {{EncodedVideoChunkInit/duration}} be the {{VideoFrame/duration}} + from the {{VideoFrame}} associated with |output|. + 2. Let |chunk| be a new {{EncodedVideoChunk}} constructed with |chunkInit|. + 3. Invoke {{VideoEncoder/[[output callback]]}} with chunk. +
+
Reset VideoEncoder
+
+ Run these steps: + 1. If {{VideoEncoder/state}} is `"closed"`, throw an {{InvalidStateError}}. + 2. Set {{VideoEncoder/state}} to `"unconfigured"`. + 3. Signal {{VideoEncoder/[[codec implementation]]}} to cease producing + output for the previous configuration. + 4. [=Reset the control message queue=]. + 5. Set {{VideoEncoder/encodeQueueSize}} to zero. +
+
Close VideoEncoder (with |error|)
+
+ Run these steps: + 1. Run the [=Reset VideoEncoder=] algorithm. + 2. Set {{VideoEncoder/state}} to `"closed"`. + 3. Clear {{VideoEncoder/[[codec implementation]]}} and release associated + [=system resources=]. + 4. If |error| is set, queue a task on the [=control thread=] event loop + invoke the {{VideoEncoder/[[error callback]]}} with |error|. +
+
-Decoder and Encoder Algorithms {#decoder-and-encoder-algorithms} -================================================================ - -The following algorithms run in the scope of the methods that invoke them. -Mentions of attributes and internal slots refer to members of the interface that -owns the invoking method. - -Configure Decoder{#configure-decoder-algorithm} ----------------------------------------------------------- -Given either an {{AudioDecoderConfig}} or {{VideoDecoderConfig}} |config|, this -algorithm attempts to select a codec implementation that supports |config|. - -Run the following steps: -1. If `state` is `“closed”`, throw an {{InvalidStateError}}. -2. If the user agent cannot provide a codec implementation to support config, - throw a {{NotSupportedError}}. -3. Set `state` to `"configured"`. -4. Queue a control message to configure the decoder with |config|. -5. Run the control message processing loop. - -Running a control message to configure the decoder means running these - steps: -1. Assign **[[codec implementation]]** with an implementation - supporting |config|. - - -Decode Chunk (with |chunk| and |output algorithm|) {#decode-chunk-algorithm} ---------------------------------------------------------------------------------------- -Run these steps: -1. If `state` is not `"configured"`, throw an {{InvalidStateError}}. -2. Increment `decodeQueueSize`. -3. Queue a control message to decode the |chunk|. -4. Run the control message processing loop. - -Running a control message to decode the chunk means running these steps: -1. Decrement `decodeQueueSize` -2. Let |codec implementation queue| be the result of starting a new parallel - queue. -3. Enqueue the following steps to |codec implementation queue|: - 1. Attempt to use **[[codec implementation]]** to decode the chunk. - 2. If decoding results in an error, queue a task on the media element task - source to run the Codec Error algorithm. -4. Otherwise, for each output, queue a task on the media element task source to - run the provided output algorithm. - - -Flush{#flush-algorithm} ----------------------------------- -Given an |output algorithm|, this algorithm flushes all pending outputs to the - output callback. - -Run these steps: -1. If `state` is not `"configured"`, return a Promise rejected with a newly - created {{InvalidStateError}}. -2. Let |promise| be a new Promise. -3. Queue a control message to flush the codec with |promise| and - |output algorithm| -4. Return |promise|. - -Running a control message to flush the codec means running these steps - with |promise| and |output algorithm|. -1. Signal **[[codec implementation]]** to emit all pending outputs. -2. For each output, run |output algorithm|. -3. Resolve |promise|. - -Codec Error{#codec-error-algorithm} ----------------------------------------------- -This algorithm fires the error callback and permanently closes the codec. - -Run these steps: -1. Cease processing of control message queue. -2. Run the Close algorithm with {{EncodingError}}. - -AudioFrame Output{#audio-frame-output-algorithm} ------------------------------------------------------------ -Run these steps: -1. If `state` is not `“configured”`, abort the following steps. -2. Let |buffer| be an {{AudioBuffer}} containing the decoded audio data. -3. Let |frame| be an {{AudioFrame}} containing |buffer| and a timestamp for the - output. -4. Invoke **[[output callback]]** with frame. - -VideoFrame Output{#video-frame-output-algorithm} ------------------------------------------------------------ -Run these steps: -1. If state is not “configured”, abort the following steps. -2. Let |planes| be a sequence of {{Plane}}s containing the decoded video frame - data. -3. Let |pixelFormat| be the {{PixelFormat}} of |planes|. -4. Let |frameInit| be a {{VideoFrameInit}} with the following keys: - 1. Let timestamp and duration be the presentation timestamp and duration - from the EncodedVideoChunk associated with this output. - 2. Let codedWidth and codedHeight be the width and height of the decoded - video frame in pixels, prior to any cropping or aspect ratio - adjustments. - 3. Let cropLeft, cropTop, cropWidth, and cropHeight be the crop region of - the decoded video frame in pixels, prior to any aspect ratio - adjustments. - 4. Let displayWidth and displayHeight be the display size of the decoded - video frame in pixels. -9. Let |frame| be a {{VideoFrame}}, constructed with |pixelFormat|, |planes|, - and |frameInit|. -10. Invoke **[[output callback]]** with |frame|. - -Reset{#reset-algorithm} ----------------------------------- -Run these steps: -1. If `state` is `“closed”`, throw an {{InvalidStateError}}. -2. Set `state` to `“unconfigured”`. -3. Signal **[[codec implementation]]** to cease producing output - for the previous configuration. - -NOTE: Some tasks to emit outputs may already be queued in the event loop. These - outputs will be dropped by the output algorithms, which abort if `state` is - not `“configured”`. - -4. For each control message in the control message queue: - 1. If a control message has an associated promise, reject the promise. - 2. Remove the message from the queue. - -Close (with error){#close-algorithm} ------------------------------------------------ -Run these steps: -1. Run the Reset algorithm. -2. Set `state` to `“closed”`. -3. Clear **[[codec implementation]]** and release associated system - resources. -4. If |error| is set, invoke **[[error callback]]** with |error|. - -Configure Encoder (with config){#configure-encoder-algorithm} ------------------------------------------------------------------------- -Run the following steps: -1. If `state` is `"closed"`, throw an {{InvalidStateError}}. -2. If the user agent cannot provide a codec implementation to support |config|, - throw a {{NotSupportedError}}. -3. Set `state` to `"configured"`. -4. Queue a control message to configure the encoder using |config|. -5. Run the control message processing loop. - -Running a control message to configure the encoder means running these steps: -1. Assign **[[codec implementation]]** with an implementation - supporting |config|. - -Encode Frame (with frame, options, and output algorithm){#encode-frame-algorithm} --------------------------------------------------------------------------------------------- -Run these steps: -1. If `state` is not `"configured"`, throw an {{InvalidStateError}}. -2. Let |frameClone| hold the result of running the Clone Frame algorithm - with |frame|. -3. Destroy the original |frame| by invoking `frame.destroy()`. -4. Increment `encodeQueueSize`. -5. Queue a control message to encode |frameClone| with |options| and - |output algorithm|. -6. Run the control message processing loop. - -Running a control message to encode the frame means running these steps. -1. Decrement `encodeQueueSize`. -2. Let |codec implementation queue| be the result of starting a new - parallel queue. -3. Enqueue the following steps to |codec implementation queue|: - 1. Attempt to use **[[codec implementation]]** and options to encode - |frameClone|. - 2. If encoding results in an error, queue a task on the media element task - source to run the codec error algorithm. - 3. Otherwise, for each output, queue a task on the media element task source - to run the provided output algorithm. - -EncodedAudioChunk Output{#encodedaudiochunk-output-algorithm} ------------------------------------------------------------------------- -Run these steps: -1. If `state` is not `“configured”`, abort the following steps. -2. Let |chunkInit| be an {{EncodedAudioChunkInit}} with the following keys: - 1. Let data contain the encoded audio data. - 2. Let type be the EnocdedAudioChunkType of the encoded audio data. - 3. Let timestamp be the timestamp from the associated input AudioFrame. - 4. Let duration be the duration from the associated input AudioFrame. -3. Let |chunk| be a new {{EncodedAudioChunk}} constructed with |chunkInit|. -4. Invoke **[[output callback]]** with |chunk|. - -EncodedVideoChunk Output{#encodedvideochunk-output-algorithm} ------------------------------------------------------------------------- -Run these steps: -1. If `state` is not `“configured”`, abort the following steps. -2. Let |chunkInit| be an {{EncodedVideoChunkInit}} with the following keys: - 1. Let data contain the encoded video data. - 2. Let type be the {{EncodedVideoChunkType}} of the encoded video data. - 3. Let timestamp be the timestamp from the associated input {{VideoFrame}}. - 4. Let duration be the duration from the associated input {{VideoFrame}}. -3. Let |chunk| be a new {{EncodedVideoChunk}} constructed with |chunkInit|. -4. Invoke **[[output callback]]** with chunk. Configurations{#configurations} =============================== @@ -793,7 +975,7 @@ Codec String{#config-codec-string} ---------------------------------------------------- In other media specifications, codec strings historically accompanied a - MIME type as the “codecs=” parameter + [=MIME type=] as the "codecs=" parameter ({{MediaSource/isTypeSupported()}}, {{HTMLMediaElement/canPlayType()}}) [[RFC6381]]. In this specification, encoded media is not containerized; hence, only the value of the codecs parameter is accepted. @@ -882,7 +1064,7 @@
A sequence of codec specific bytes, commonly known as extradata. - NOTE: For example, the vorbis “code book”. + NOTE: For example, the vorbis "code book".
@@ -1086,8 +1268,8 @@
closed
- The codec is no longer usable and underlying system resources have been - released. + The codec is no longer usable and underlying [=system resources=] have + been released.
@@ -1265,8 +1447,8 @@
close()
- Immediately frees system resources. When invoked, run these steps: - 1. Release system resources for buffer and set its value to null. + Immediately frees [=system resources=]. When invoked, run these steps: + 1. Release [=system resources=] for buffer and set its value to null. 2. Assign `true` to the {{AudioFrame/[[detached]]}} internal slot. NOTE: This section needs work. We should use the name and semantics of @@ -1338,7 +1520,7 @@ underlying UA resources may be GPU backed or formatted in such a way that conversion to an allowed pixel format requires expensive copies and translation. When this occurs, we should allow planes to be null and format - to be “opaque” to avoid early optimization. We should make conversion + to be "opaque" to avoid early optimization. We should make conversion explicit and user controlled by offering a `videoFrame.convertTo(format)` that returns a Promise containing a new VideoFrame for which the copies/translations are performed. @@ -1346,7 +1528,7 @@ VideoFrame(imageBitmap, frameInit) -1. If |frameInit| is not a valid VideoFrameInit, throw a {{TypeError}}. +1. If |frameInit| is not a [=valid VideoFrameInit=], throw a {{TypeError}}. 2. If the value of |imageBitmap|'s' {{PlatformObject/[[Detached]]}} internal slot is set to `true`, then throw an {{InvalidStateError}} DOMException. 3. Let |frame| be a new {{VideoFrame}}. @@ -1389,7 +1571,7 @@ 1. If either {{VideoFrameInit/codedWidth}} or {{VideoFrameInit/codedHeight}} is not present in |frameInit|, throw a {{TypeError}}. -2. If |frameInit| is not a valid VideoFrameInit, throw a {{TypeError}}. +2. If |frameInit| is not a [=valid VideoFrameInit=], throw a {{TypeError}}. 3. If the length of |planes| is incompatible with the given pixelFormat, throw a TypeError. 4. Let |frame| be a new {{VideoFrame}} object. @@ -1483,8 +1665,9 @@ ### Methods ###{#videoframe-methods} destroy() -Immediately frees system resources. Destruction applies to all references, - including references that are serialized and passed across Realms. +Immediately frees [=system resources=]. Destruction applies to all + references, including references that are serialized and passed across + Realms. NOTE: Authors should take care to manage frame lifetimes by calling {{VideoFrame/destroy()}} immediately when frames are no longer needed. @@ -1506,8 +1689,8 @@ {{VideoFrame/clone()}} sparingly. When invoked, run the following steps: -1. If the value of the {{VideoFrame/[[detached]]}} slot is `true`, return a - Promise rejected with a newly created {{InvalidStateError}}. +1. If the value of the {{VideoFrame/[[detached]]}} slot is `true`, return + [=a promise rejected with=] {{InvalidStateError}} {{DOMException}}. 2. Let |p| be a new Promise. 3. In parallel, resolve |p| with the result of running the Clone Frame algorithm with this. @@ -1800,7 +1983,7 @@

Clone Frame (with |frame|)

arrival of new picture data as well as changes to `[[track]].readyState`. While `[[track]].readyState` is {{MediaStreamTrackState/live}}, for each new - picture that arrives in {{VideoTrackReader/[[track]]}}, execute the + picture that arrives in {{VideoTrackReader/[[track]]}}, perform the following steps: NOTE: Pictures that arrived prior to the start of this loop are not