Adding new element gvastreammux

docs/user-guide/elements/elements.md

@@ -31,6 +31,8 @@ gst-inspect-1.0 utility.
 | [gvaattachroi](./gvaattachroi.md)     | Adds user-defined regions of interest to perform inference on (instead of full frame). Example: monitoring road traffic in a city camera feed; splitting large image into smaller pieces, and running inference on each piece (healthcare cell analytics).<br>Example:<br> gst-launch-1.0 … ! decodebin3 ! gvaattachroi roi=xtl,ytl,xbr,ybr gvadetect inference-region=1 ! … OUT<br> |
 | [gvafpscounter](./gvafpscounter.md)    | Measures frames per second across multiple video streams in a single GStreamer process.<br>Example:<br> gst-launch-1.0 … ! decodebin3 ! gvadetect … ! gvafpscounter ! … OUT<br> |
 | [gvafpsthrottle](./gvafpsthrottle.md)   | Throttles the framerate of video streams by enforcing a maximum frames-per-second (FPS) rate. Useful for rate limiting in pipelines or for testing at specific processing framerates.<br>Example:<br> gst-launch-1.0 … ! decodebin3 ! gvafpsthrottle target-fps=10 ! … OUT<br> |
+| [gvastreammux](./gvastreammux.md)       | Muxes multiple video streams into a single pipeline using round-robin scheduling, attaching source-tracking metadata to each buffer. No frame dropping — batch size always equals input stream count.<br>Example:<br> gst-launch-1.0 gvastreammux name=mux ! queue ! gvadetect model=$mDetect device=GPU ! gvafpscounter ! fakesink rtspsrc location=rtsp://host:8554/stream ! … ! mux.sink_0 rtspsrc location=rtsp://host:8555/stream ! … ! mux.sink_1<br> |
+| [gvastreamdemux](./gvastreamdemux.md)   | Demuxes a single interleaved stream back into per-source output pads based on GstGvaStreammuxMeta. Must be used with gvastreammux.<br>Example:<br> gst-launch-1.0 gvastreammux name=mux ! queue ! gvadetect model=$mDetect device=GPU ! gvastreamdemux name=demux demux.src_0 ! queue ! fakesink demux.src_1 ! queue ! fakesink rtspsrc location=rtsp://host:8554/stream ! … ! mux.sink_0 rtspsrc location=rtsp://host:8555/stream ! … ! mux.sink_1<br> |
 | [gvametaaggregate](./gvametaaggregate.md) | Aggregates inference results from multiple pipeline branches.<br>Example:<br> gst-launch-1.0 … ! decodebin3 ! tee name=t t. ! queue ! gvametaaggregate name=a ! gvaclassify … ! gvaclassify … ! gvametaconvert … ! gvametapublish … ! fakesink t. ! queue ! gvadetect … ! a.<br>                                                                                               |
 | [gvametaconvert](./gvametaconvert.md)   | Converts the metadata structure to JSON or raw text formats. Can write output to a file.|
 | [gvametapublish](./gvametapublish.md)   | Publishes the JSON metadata to MQTT or Kafka message brokers or files.<br>Example:<br> gst-launch-1.0 … ! decodebin3 ! gvadetect model=$mDetect device=GPU … ! gvametaconvert format=json … ! gvametapublish … ! … OUT<br> |
@@ -56,6 +58,8 @@ g3dlidarparse
 gvaattachroi
 gvafpscounter
 gvafpsthrottle
+gvastreammux
+gvastreamdemux
 gvametaaggregate
 gvametaconvert
 gvametapublish

docs/user-guide/elements/gvastreamdemux.md

@@ -0,0 +1,174 @@
+# gvastreamdemux
+
+Demuxes a single interleaved video stream back into multiple per-source output pads based on
+`GstGvaStreammuxMeta` metadata. This element is the companion to `gvastreammux` and **must** be
+used together with it — it cannot function standalone.
+
+## Overview
+
+The `gvastreamdemux` element reads `GstGvaStreammuxMeta` from each incoming buffer and routes it
+to the corresponding source pad (`src_0`, `src_1`, ...) based on the `source_id` field. This enables
+a common multi-stream pattern:
+
+```
+N sources → gvastreammux → shared inference → gvastreamdemux → N independent outputs
+```
+
+Key design principles:
+
+- **Metadata-Driven Routing**: Each buffer is routed to `src_{source_id}` using the metadata attached
+  by `gvastreammux`. Buffers without `GstGvaStreammuxMeta` cause a pipeline error.
+- **No Frame Dropping**: Every buffer received on the sink pad is forwarded to exactly one source pad.
+- **Batch Ordering Validation**: The element tracks `batch_id` per source pad to detect out-of-order
+  delivery. If a buffer arrives with a `batch_id` lower than the previous one for the same source,
+  a warning is logged.
+- **Source Count Validation**: On the first buffer, the element checks that the number of requested
+  source pads matches `num_sources` in the metadata. A mismatch causes a pipeline error.
+
+> **Important**: Because `gvastreammux` interleaves frames in round-robin order, the downstream
+> inference element (e.g., `gvadetect`) **must** have `inference-interval=1`. Setting
+> `inference-interval` to N > 1 causes certain sources to be consistently skipped from inference.
+> A future enhancement will add frame-dropping logic inside `gvastreammux` to apply the interval
+> uniformly across all input streams.
+
+## How It Works
+
+1. The pipeline requests source pads (`src_0`, `src_1`, ...) — one per original input source.
+   The number of `src` pads **must** match the number of `sink` pads on the upstream `gvastreammux`.
+2. When a buffer arrives on the sink pad, the element reads `GstGvaStreammuxMeta`.
+3. The buffer is pushed to `src_{source_id}`.
+4. On the first buffer, the element validates `num_sources == num_src_pads`.
+5. EOS on the sink pad is forwarded to all source pads.
+
+## Properties
+
+| Property  | Type   | Description                                                                                   | Default |
+|-----------|--------|-----------------------------------------------------------------------------------------------|---------|
+| max-fps   | Double | Maximum output frame rate per source (0 = unlimited). Only set for local file sources. Do not set for RTSP or live sources as it may cause pipeline stalls. | 0       |
+
+The `max-fps` throttle is applied globally (shared across all source pads), not per individual pad.
+
+## Pipeline Examples
+
+### Local Files with Per-Source Output
+
+Mux two local files for shared inference, then demux for independent downstream processing.
+Set `max-fps` on both mux and demux to control throughput:
+
+```bash
+gst-launch-1.0 \
+  gvastreammux name=mux max-fps=30 \
+  ! queue \
+  ! gvadetect model=model.xml device=GPU \
+    pre-process-backend=va-surface-sharing \
+  ! gvastreamdemux name=demux max-fps=30 \
+  demux.src_0 ! queue ! gvafpscounter ! fakesink \
+  demux.src_1 ! queue ! gvafpscounter ! fakesink \
+  filesrc location=video0.h265 ! h265parse ! vah265dec ! mux.sink_0 \
+  filesrc location=video1.h265 ! h265parse ! vah265dec ! mux.sink_1
+```
+
+### RTSP Sources with Per-Source Output
+
+Two RTSP streams muxed for shared inference, then demuxed. No `max-fps` needed for live sources:
+
+```bash
+gst-launch-1.0 \
+  gvastreammux name=mux \
+  ! queue \
+  ! gvadetect model=model.xml device=GPU \
+    pre-process-backend=va-surface-sharing \
+  ! gvastreamdemux name=demux \
+  demux.src_0 ! queue ! gvafpscounter ! fakesink \
+  demux.src_1 ! queue ! gvafpscounter ! fakesink \
+  rtspsrc location=rtsp://host:8554/stream latency=200 \
+  ! rtph265depay ! h265parse ! vah265dec ! mux.sink_0 \
+  rtspsrc location=rtsp://host:8555/stream latency=200 \
+  ! rtph265depay ! h265parse ! vah265dec ! mux.sink_1
+```
+
+### Per-Source Watermark and Display
+
+Demux after inference, then apply per-source watermark and render to screen:
+
+```bash
+gst-launch-1.0 \
+  gvastreammux name=mux \
+  ! queue \
+  ! gvadetect model=model.xml device=GPU \
+    pre-process-backend=va-surface-sharing \
+  ! gvastreamdemux name=demux \
+  demux.src_0 ! queue ! gvawatermark ! videoconvert ! autovideosink \
+  demux.src_1 ! queue ! gvawatermark ! videoconvert ! autovideosink \
+  rtspsrc location=rtsp://host:8554/stream latency=200 \
+  ! rtph265depay ! h265parse ! vah265dec ! mux.sink_0 \
+  rtspsrc location=rtsp://host:8555/stream latency=200 \
+  ! rtph265depay ! h265parse ! vah265dec ! mux.sink_1
+```
+
+## Required Metadata
+
+This element requires `GstGvaStreammuxMeta` on every incoming buffer. The metadata is attached
+by `gvastreammux` and contains:
+
+| Field        | Type     | Description                                          |
+|--------------|----------|------------------------------------------------------|
+| source_id    | guint    | Pad index the buffer originated from (0, 1, 2, ...) |
+| batch_id     | guint64  | Monotonically increasing batch cycle counter         |
+| num_sources  | guint    | Total number of active input sources at batch time   |
+
+## Error Conditions
+
+| Condition                             | Behavior                                |
+|---------------------------------------|-----------------------------------------|
+| Buffer missing GstGvaStreammuxMeta    | Returns `GST_FLOW_ERROR`                |
+| `num_src_pads != num_sources`         | Returns `GST_FLOW_ERROR` (first buffer) |
+| `source_id` out of range              | Returns `GST_FLOW_ERROR`                |
+| `batch_id` out of order for a pad     | `GST_WARNING` (continues processing)    |
+
+## Element Details (gst-inspect-1.0)
+
+```
+Factory Details:
+  Rank                     none (0)
+  Long-name                GVA Stream Demuxer
+  Klass                    Video/Demuxer
+  Description              Demuxes a single stream into multiple output pads
+                           based on GstGvaStreammuxMeta source_id.
+                           Must be used with gvastreammux.
+  Author                   Intel Corporation
+
+Pad Templates:
+  SINK template: 'sink'
+    Availability: Always
+    Capabilities:
+      video/x-raw
+               format: { BGRx, BGRA, BGR, NV12, I420, RGB, RGBA, RGBx }
+      video/x-raw(memory:VAMemory)
+               format: { NV12 }
+      video/x-raw(memory:DMABuf)
+               format: { DMA_DRM }
+
+  SRC template: 'src_%u'
+    Availability: On request
+    Capabilities:
+      video/x-raw
+               format: { BGRx, BGRA, BGR, NV12, I420, RGB, RGBA, RGBx }
+      video/x-raw(memory:VAMemory)
+               format: { NV12 }
+      video/x-raw(memory:DMABuf)
+               format: { DMA_DRM }
+
+Element Properties:
+  max-fps             : Maximum output frame rate per source (0 = unlimited).
+                        Only set this when the video source is a local file.
+                        Do not set for RTSP or live sources as it may cause pipeline stalls.
+                        flags: readable, writable
+                        Double. Range: 0 - 1.797693e+308  Default: 0
+  name                : The name of the object
+                        flags: readable, writable
+                        String. Default: "gvastreamdemux0"
+  parent              : The parent of the object
+                        flags: readable, writable
+                        Object of type "GstObject"
+```

docs/user-guide/elements/gvastreammux.md

@@ -0,0 +1,171 @@
+# gvastreammux
+
+Muxes multiple video streams into a single pipeline, attaching source-tracking metadata to each buffer.
+The element enables multi-stream inference by interleaving frames from N input sources through a single
+downstream processing chain (e.g., `gvadetect`), eliminating the need for N separate inference pipelines.
+
+## Overview
+
+The `gvastreammux` element collects video frames from multiple sink pads and outputs them through a single
+source pad in **round-robin** order. Key design principles:
+
+- **No Frame Dropping**: Every input frame is forwarded downstream. The element uses GStreamer's `GstCollectPads`
+  mechanism which blocks until all active pads have at least one buffer available, then pops exactly one buffer
+  from each pad per cycle.
+- **One Frame Per Source Per Cycle**: Each batch cycle outputs exactly one frame from each input stream.
+  The batch size always equals the number of input streams — it is not configurable separately.
+- **Source Metadata**: Each output buffer is tagged with `GstGvaStreammuxMeta` containing `source_id` (pad index),
+  `batch_id` (cycle counter), and `num_sources` (total input count). This metadata is required by
+  `gvastreamdemux` to route buffers back to per-source branches.
+
+> **Important**: Because frames from all sources are interleaved in round-robin order, the downstream
+> inference element (e.g., `gvadetect`) **must** have `inference-interval=1`. If `inference-interval` is
+> set to N > 1, the element only runs inference on every Nth buffer — which means certain input sources
+> will consistently be skipped. For example, with 2 sources and `inference-interval=2`, only source 0
+> will ever get inference results while source 1 is always skipped.
+>
+> A future enhancement will add frame-dropping logic inside `gvastreammux` itself, applying the interval
+> uniformly across all input streams so that every source gets the same inference coverage.
+
+## How It Works
+
+1. The pipeline requests sink pads (`sink_0`, `sink_1`, ...) — one per input source.
+2. `GstCollectPads` waits until every non-EOS sink pad has at least one buffer.
+3. The `collected` callback pops one buffer from each pad, attaches `GstGvaStreammuxMeta`, and pushes
+   it downstream through the single `src` pad.
+4. The `batch_id` counter increments after each complete cycle.
+5. When a source sends EOS, `GstCollectPads` stops waiting for that pad. The remaining sources continue
+   to produce output until all sources have sent EOS.
+
+## Properties
+
+| Property  | Type   | Description                                                                                   | Default |
+|-----------|--------|-----------------------------------------------------------------------------------------------|---------|
+| max-fps   | Double | Maximum output frame rate (0 = unlimited). Only set for local file sources. Do not set for RTSP or live sources as it may cause pipeline stalls. | 0       |
+
+## Pipeline Examples
+
+### Local Files with max-fps (gvastreammux only)
+
+Two local video files decoded and muxed for shared inference. Set `max-fps` to avoid processing
+at unbounded speed:
+
+```bash
+gst-launch-1.0 \
+  gvastreammux name=mux max-fps=30 \
+  ! queue \
+  ! gvadetect model=model.xml device=GPU \
+    pre-process-backend=va-surface-sharing \
+  ! queue ! gvafpscounter ! fakesink \
+  filesrc location=video0.h265 ! h265parse ! vah265dec ! mux.sink_0 \
+  filesrc location=video1.h265 ! h265parse ! vah265dec ! mux.sink_1
+```
+
+### RTSP Sources (gvastreammux only)
+
+Two RTSP streams muxed for shared inference. No `max-fps` needed — live sources are naturally
+rate-limited by the network:
+
+```bash
+gst-launch-1.0 \
+  gvastreammux name=mux \
+  ! queue \
+  ! gvadetect model=model.xml device=GPU \
+    pre-process-backend=va-surface-sharing \
+  ! queue ! gvafpscounter ! fakesink \
+  rtspsrc location=rtsp://host:8554/stream latency=200 \
+  ! rtph265depay ! h265parse ! vah265dec ! mux.sink_0 \
+  rtspsrc location=rtsp://host:8555/stream latency=200 \
+  ! rtph265depay ! h265parse ! vah265dec ! mux.sink_1
+```
+
+### Local Files with gvastreamdemux (per-source output)
+
+Mux two files for shared inference, then demux back to per-source branches:
+
+```bash
+gst-launch-1.0 \
+  gvastreammux name=mux max-fps=30 \
+  ! queue \
+  ! gvadetect model=model.xml device=GPU \
+    pre-process-backend=va-surface-sharing \
+  ! gvastreamdemux name=demux \
+  demux.src_0 ! queue ! gvafpscounter ! fakesink \
+  demux.src_1 ! queue ! gvafpscounter ! fakesink \
+  filesrc location=video0.h265 ! h265parse ! vah265dec ! mux.sink_0 \
+  filesrc location=video1.h265 ! h265parse ! vah265dec ! mux.sink_1
+```
+
+### RTSP Sources with gvastreamdemux (per-source output)
+
+Mux two RTSP streams, run shared inference, and demux for independent downstream processing:
+
+```bash
+gst-launch-1.0 \
+  gvastreammux name=mux \
+  ! queue \
+  ! gvadetect model=model.xml device=GPU \
+    pre-process-backend=va-surface-sharing \
+  ! gvastreamdemux name=demux \
+  demux.src_0 ! queue ! gvafpscounter ! fakesink \
+  demux.src_1 ! queue ! gvafpscounter ! fakesink \
+  rtspsrc location=rtsp://host:8554/stream latency=200 \
+  ! rtph265depay ! h265parse ! vah265dec ! mux.sink_0 \
+  rtspsrc location=rtsp://host:8555/stream latency=200 \
+  ! rtph265depay ! h265parse ! vah265dec ! mux.sink_1
+```
+
+## Metadata Structure
+
+Each output buffer carries `GstGvaStreammuxMeta`:
+
+| Field        | Type     | Description                                          |
+|--------------|----------|------------------------------------------------------|
+| source_id    | guint    | Pad index the buffer originated from (0, 1, 2, ...) |
+| batch_id     | guint64  | Monotonically increasing batch cycle counter         |
+| num_sources  | guint    | Total number of active input sources at batch time   |
+
+## Element Details (gst-inspect-1.0)
+
+```
+Factory Details:
+  Rank                     none (0)
+  Long-name                GVA Stream Muxer
+  Klass                    Video/Muxer
+  Description              Muxes multiple video streams into a single pipeline with source metadata
+  Author                   Intel Corporation
+
+Pad Templates:
+  SINK template: 'sink_%u'
+    Availability: On request
+    Capabilities:
+      video/x-raw
+               format: { BGRx, BGRA, BGR, NV12, I420, RGB, RGBA, RGBx }
+      video/x-raw(memory:VAMemory)
+               format: { NV12 }
+      video/x-raw(memory:DMABuf)
+               format: { DMA_DRM }
+
+  SRC template: 'src'
+    Availability: Always
+    Capabilities:
+      video/x-raw
+               format: { BGRx, BGRA, BGR, NV12, I420, RGB, RGBA, RGBx }
+      video/x-raw(memory:VAMemory)
+               format: { NV12 }
+      video/x-raw(memory:DMABuf)
+               format: { DMA_DRM }
+
+Element Properties:
+  max-fps             : Maximum output frame rate (0 = unlimited).
+                        Only set this when the video source is a local file.
+                        Do not set for RTSP or live sources as it may cause pipeline stalls.
+                        flags: readable, writable
+                        Double. Range: 0 - 1.797693e+308  Default: 0
+  name                : The name of the object
+                        flags: readable, writable
+                        String. Default: "gvastreammux0"
+  parent              : The parent of the object
+                        flags: readable, writable
+                        Object of type "GstObject"
+```