[Feat] Camera Stream API proposal

[edgarriba]

🚀 Feature

An API that interfaces directly with video camera streams.

Motivation

After including encoder/decoders for image and video would be to also great to have native support for accessing to video cameras e.g for robotics or real-time applications possibly making use of hardware acceleration.

Pitch

I would expect to have a simple API where you could pass a video file path, camera index, or set of cameras that return a batch with synchronized frames (this might need some logic).

import torch
import torchvision as tv

# create camera instance
stream = tv.VideoStream(cam_id=0)  # from specific camera found in your device
stream = tv.VideoStream("http://<username:password>@<ip_address>/video.cgi?.mjpg")  # e.g from ip camera
stream = tv.VideoStream([path1, path2, ...])  # multiple cameras

stream.open()

# get data (this could implemented as a callback too)
for frame, meta in stream:
    # frame: torch.Tensor (BxCxHxW)  # 1 for single camera/ B for multi-cam
    # meta: Dict[str, Any] (e.g. 'audio')
    frame = gaussian_blur(frame)  # BxCxHxW
    pl.imshow(frame) # visualizer is another topic :) 

stream.close()

Alternatives

Existing solutions is using opencv for now.

Additional context

This comes from the motivation to integrate pytorch vision stuff like kornia into ROS2.

cc @bjuncek

[bjuncek]

To follow up on this, it would technically be possible to deal with arbitrary memory stream (e.g. buffer), interfacing with a camera is something we'd have to research.

I'll try to research a bit on how does OpenCV do that and get back to you later next week if that's ok?

[ibadamimarz]

Any update on this feature please?

[bjuncek]

Sadly, we're not adding additional features at the moment.
We'll add this to list of feature requests, but have no ETA -- added to the larger tracking issue