Add automatic writing encoded video streams to disk when streaming from RTSP sources using cudacodec #3098

cudawarped · 2021-11-03T11:45:22Z

Update cudacoded VideoReader to;

automatically write the raw encoded RTSP video streams to video files if desired,
use less GPU memory by more closely mirroring the Nvidia samples, and
decode mpeg4 files.

This pull request relies on the changes in opencv/opencv#20978.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

force_builders=Custom
buildworker:Custom=linux-4,linux-6
build_image:Custom=ubuntu-cuda:16.04

…luding checks to ensure codec used in input video file is supported on the current device.

…o update_nvcuvid_codecs

# Conflicts: # modules/cudacodec/include/opencv2/cudacodec.hpp # modules/cudacodec/src/video_decoder.cpp

1) automatically write the raw encoded video stream to a video file and, 2) use less GPU memory by more closely mirroring the Nvidia samples. Specifically querying the decoder for the number of decode surfaces (h265 commonly uses 4) instead of always using 20 and not using adaptive deinterlacing when the video sequence is progressive. Additional updates to mirror the Nvidia sample include initializing the decoder so that HandleVideoSequence() gets called every time before the decoder is initialized, ensuring all the parameters for the decoder are provided by nvcudec. Added facility to decode AV1, not tested as VideoCapture doesn't return a valid fourcc for this. Add facility to decode MPEG4 video - requires modification to VideoCapture see pull request.

…les to that they play in vlc. Notes: VideoCapture - returns mpeg as the codec for mpeg4 files, so files written as .m4v from mpeg4 sources cannot currently be decoded. This is also true for AV1 sources where cap.get(CAP_PROP_FOURCC) returns 0. Added mpeg4 test file which can be decoded when VideoCapture adds the extra_data.

…stead of appended to the first packet.

alalek

Be careful with creation of GOD classes.
Reader should read, not write.

Please take a look on comments below before merge.

alalek · 2021-11-23T10:15:15Z

modules/cudacodec/include/opencv2/cudacodec.hpp

+    file, especially from container formats avi, mp4 etc.  If the filename provided is invalid, cannot be opened
+    or written to, the first call to nextFrame() after calling this function will return false.
+     */
+    CV_WRAP virtual void writeToFile(const std::string filename, const bool autoDetectExt = false) = 0;


const std::string filename

const reference

Here and below

alalek · 2021-11-28T23:54:11Z

modules/cudacodec/include/opencv2/cudacodec.hpp


 FFMPEG is used to read videos. User can implement own demultiplexing with cudacodec::RawVideoSource
 */
-CV_EXPORTS_W Ptr<VideoReader> createVideoReader(const String& filename);
+CV_EXPORTS_W Ptr<VideoReader> createVideoReader(const String& filename, const String filenameToWrite = "", const bool autoDetectExt = false);


String filenameToWrite

Main problem of this approach is missing unicode support.

I would create dedicated API for that:

keep existed API "as is"

add new API like proposed (with _W)

add extra new API with std::ostream&& or std::ofstream&& (note, move ctor is available only)

Main problem of this approach is missing unicode support.

Ahh I see. How can we support unicode in the new api (VideoReader_W) for const String& filename?

add extra new API with std::ostream&& or std::ofstream&& (note, move ctor is available only)

If possible I would like to be able to automatically detect the extension before creating the file so that the file can be played back without having to be renamed after writing has finished.

If this is not performed internally then this would involve creating a VideoCapture object before creating the new class just to detect the codec and then using that info to determine the file extension.

VideoCapture cap("rtsp://...") std::string ext = FourCCToExtension(cap.get(CAP_PROP_FOURCC)) std::ofstream fileOut = (fileName + ext); cap.release(); // release the stream so we can connect to it again below cv::Ptr<cv::cudacodec::VideoReader_W> reader_w = cv::cudacodec::createVideoReader_W(inputFile, fileOut);

I would like to avoid that can you suggest and alternative?

How can we support unicode in the new api (VideoReader_W) for const String& filename

String may handle UTF-8 strings only (it is std::string , BTW). There is no wchar_t or std::wstring in OpenCV.
Main question is about support on target platform (e.g., support of UTF-8 in std::ofstream) - Win 10 brings some support for UTF-8 but it should be pre-configured (also this configuration may break legacy apps).

alternative

Capture to temporary file and then rename() with correct extension on user side.
But it also looks weird, so I don't have strong suggestion or recommendation here.

cudawarped · 2021-11-29T19:22:25Z

Thank you for taking a look.

Be careful with creation of GOD classes.
Reader should read, not write.

I am not sure how to achieve my objective without creating a GOD class. I want to be able to stream and decode from an RTSP source at the same time as archiving the footage. This seems like a common use case for IP camera's, e.g. run DNN over live decoded footage and archive footage for training later or to simply see why the DNN failed.
With two classes I would need two streams which is what I am trying to avoid due to unecessary network overhead and lack of support from IP camera's (usually each stream is a different resolution if multiple streams are supported).
Can you suggest a way to achieve this without placing the read and write functionality in the same class?

alalek · 2021-11-29T21:39:56Z

I can suggest .retrieve() approach to fetch decoded frames + RAW encoded packets (0-N per frame) and write them by user code.
But there is no strong requirements for that in opencv_contrib.

open() with enabling of RAW stream
std::ofstream ostream = ...;
... retrieve and white "codec_extradata" ...
int rawIdxBase = (int)cudacodec.get(PROP_RAW_PACKAGES_BASE_INDEX);
...
if (cudacodec.grab())
{
    cudacodec.retrive(frame, 0);  // regular decoded frame data
    int N = (int)cudacodec.get(PROP_NUMBER_OF_RAW_PACKAGES_FOR_CURRENT_FRAME);  // [0-N] range
    for (int i = 0; i < N; i++)
    {
        cudacodec.retrieve(raw_buffer, rawIdxBase + i);
        ostream.write(raw_buffer);
    }

    ... process frame ...
}
else
{
    ... do we have RAW packages in case of end of stream? ...
}

…t the same time as decoded frames.

cudawarped · 2021-12-08T17:40:15Z

I can suggest .retrieve() approach to fetch decoded frames + RAW encoded packets (0-N per frame) and write them by user code.
But there is no strong requirements for that in opencv_contrib.

I have added retrieve, grab etc. to enable the raw packets to be retrieved from the VideoReader class. I understand that there may not be a strong requirement for this, however these additions should not interfere with the current functionality when rawMode is disabled.

The only "issue" I can see is that there is no straight forward way to get the packets corresponding to the current frame because the decoder is fed encoded data as fast as possible. As a result when it outputs a decoded frame (HandlePictureDisplay) there is no guarantee that that frame corresponds to all the packets that were fed in. The best we can do is get all the packets since the last call to grab() or the creation of the VideoReader object, which is fine for file writing purposes.

alalek

Well done! Thank you for contribution 👍

alalek · 2021-12-12T12:43:47Z

modules/cudacodec/src/ffmpeg_video_source.cpp


+int StartCodeLen(unsigned char* data, const int sz) {


static to keep code local

cudawarped · 2021-12-13T12:38:17Z

Thanks for all you help alalek, I will add static when cudacodec needs modifying again to account for depreciation of CUvideosource.

James Bowley and others added 10 commits June 11, 2019 09:57

Add missing codecs to cudacodec which uses Nvidia Video Codec SDK inc…

736fe2a

…luding checks to ensure codec used in input video file is supported on the current device.

Merge branch 'master' of https://github.com/opencv/opencv_contrib int…

4d4fc5d

…o update_nvcuvid_codecs

Merge branch 'master' of https://github.com/cudawarped/opencv_contrib

3b5acef

# Conflicts: # modules/cudacodec/include/opencv2/cudacodec.hpp # modules/cudacodec/src/video_decoder.cpp

Merge branch 'master' of https://github.com/opencv/opencv_contrib

511c8a5

Update to account for the extraData being passed from cap.retrieve in…

2ee3050

…stead of appended to the first packet.

Update to be compatible with changes to VideoCapture

f718d8c

Remove redundant test.

7ae7eb9

Add check to ensure retrieve is successful.

a179fe4

alalek reviewed Nov 29, 2021

View reviewed changes

cudawarped added 2 commits December 8, 2021 16:33

Remove writeToFile and allow VideoReader to return raw encoded data a…

65a09c3

…t the same time as decoded frames.

Fix missing documentation.

c644e14

cudawarped requested a review from alalek December 8, 2021 17:44

alalek approved these changes Dec 12, 2021

View reviewed changes

alalek merged commit 1cecd2c into opencv:4.x Dec 12, 2021

alalek mentioned this pull request Dec 30, 2021

(5.x) Merge 4.x #3142

Merged

alalek mentioned this pull request Feb 22, 2022

(5.x) Merge 4.x #3179

Merged

cudawarped mentioned this pull request Aug 26, 2022

Fix memory leak in cudacodec #3339

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add automatic writing encoded video streams to disk when streaming from RTSP sources using cudacodec #3098

Add automatic writing encoded video streams to disk when streaming from RTSP sources using cudacodec #3098

Uh oh!

cudawarped commented Nov 3, 2021 •

edited by alalek

Loading

Uh oh!

alalek left a comment

Uh oh!

alalek Nov 23, 2021

Uh oh!

alalek Nov 28, 2021

Uh oh!

cudawarped Nov 29, 2021 •

edited

Loading

Uh oh!

alalek Nov 29, 2021

Uh oh!

cudawarped commented Nov 29, 2021

Uh oh!

alalek commented Nov 29, 2021

Uh oh!

cudawarped commented Dec 8, 2021

Uh oh!

alalek left a comment

Uh oh!

alalek Dec 12, 2021

Uh oh!

cudawarped commented Dec 13, 2021

Uh oh!

Uh oh!

Add automatic writing encoded video streams to disk when streaming from RTSP sources using cudacodec #3098

Add automatic writing encoded video streams to disk when streaming from RTSP sources using cudacodec #3098

Uh oh!

Conversation

cudawarped commented Nov 3, 2021 • edited by alalek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

alalek Nov 23, 2021

Choose a reason for hiding this comment

Uh oh!

alalek Nov 28, 2021

Choose a reason for hiding this comment

Uh oh!

cudawarped Nov 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alalek Nov 29, 2021

Choose a reason for hiding this comment

Uh oh!

cudawarped commented Nov 29, 2021

Uh oh!

alalek commented Nov 29, 2021

Uh oh!

cudawarped commented Dec 8, 2021

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

alalek Dec 12, 2021

Choose a reason for hiding this comment

Uh oh!

cudawarped commented Dec 13, 2021

Uh oh!

Uh oh!

cudawarped commented Nov 3, 2021 •

edited by alalek

Loading

cudawarped Nov 29, 2021 •

edited

Loading