`cudacodec::VideoReader` FORCC update with main #23540 #3488

cudawarped · 2023-04-30T05:44:35Z

Update cudacodec::VideoReader to use codec fourcc.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

cudawarped · 2023-06-16T12:39:01Z

I think this is failing because the branch containing the dependant PR (opencv/opencv#23540) has a different name (add_CAP_PROP_CODEC_FOURCC).

asmorkalov

👍

asmorkalov · 2023-06-20T09:18:50Z

I tested both branches together locally. Ubuntu 18.04, CUDA 10.2 crashes tests:

[ RUN      ] CUDA_Codec/Video.Reader/8, where GetParam() = (NVIDIA GeForce GTX 1080, "highgui/video/sample_322x242_15frames.yuv420p.libaom-av1.mp4")
[ERROR:[email protected]] global cap_ffmpeg_impl.hpp:1237 open Could not find decoder for codec_id=32797
[ERROR:[email protected]] global cap_ffmpeg_impl.hpp:1286 open VIDEOIO/FFMPEG: Failed to initialize VideoCapture
[ WARN:[email protected]] global cap.cpp:206 open VIDEOIO(FFMPEG): backend is generally available but can't be used to capture by name
[ERROR:[email protected]] global video_decoder.cpp:127 create Video source is not supported by hardware video decoder.
[ERROR:[email protected]] global video_parser.cpp:169 HandleVideoSequence Attempt to configure Nvidia decoder failed!
[ERROR:[email protected]] global video_parser.cpp:83 parseVideoData Call to cuvidParseVideoData failed!
unknown file: Failure
C++ exception with description "OpenCV(4.7.0-dev) /home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_reader.cpp:212: error: (-2:Unspecified error) Parsing/Decoding video source failed, check GPU memory is available and GPU supports hardware decoding. in function 'internalGrab'
" thrown in the test body.
[  FAILED  ] CUDA_Codec/Video.Reader/8, where GetParam() = (NVIDIA GeForce GTX 1080, "highgui/video/sample_322x242_15frames.yuv420p.libaom-av1.mp4") (63 ms)
[ RUN      ] CUDA_Codec/Video.Reader/9, where GetParam() = (NVIDIA GeForce GTX 1080, "cv/tracking/faceocc2/data/faceocc2.webm")
[       OK ] CUDA_Codec/Video.Reader/9 (144 ms)
[ RUN      ] CUDA_Codec/Video.Reader/10, where GetParam() = (NVIDIA GeForce GTX 1080, "highgui/video/sample_322x242_15frames.yuv420p.mpeg2video.mp4")
segmentation fault (core dumped)

asmorkalov · 2023-06-20T09:20:56Z

looks like libaom-av1 is not supported with my FFmpeg version. I propose to skip the test for it in case if this particular video is not opened, but not raise test failure.

cudawarped · 2023-06-20T09:30:27Z

looks like libaom-av1 is not supported with my FFmpeg version. I propose to skip the test for it in case if this particular video is not opened, but not raise test failure.

Agreed. I'm wondering if FFmpeg needs the decoder to simply parse the file or not, if so we may be able to modify cap_ffmpeg_impl.hpp in the future to remove decoder initialization when CAP_PROP_FORMAT == -1 (rawMode) is passed to the VideoCapture constructor.

cudawarped · 2023-06-20T10:37:24Z

I propose to skip the test for it in case if this particular video is not opened, but not raise test failure.

Do you mean simply return from the test case (resulting in a PASS) instead of using throw SkipTestException()?

asmorkalov · 2023-06-20T12:20:24Z

I propose to use skip exception like it's done here: https://github.com/opencv/opencv/blob/726ba0210e1592606ccb8843fb092504ca787096/modules/videoio/test/test_ffmpeg.cpp#L37

asmorkalov · 2023-06-20T12:57:20Z

modules/cudacodec/test/test_video.cpp

+        reader = cv::cudacodec::createVideoReader(inputFile);
+    }
+    catch (const Exception& e) {
+        ASSERT_TRUE(relativeFilePath == "highgui/video/sample_322x242_15frames.yuv420p.libaom-av1.mp4");


I propose:

if (relativeFilePath == "highgui/video/sample_322x242_15frames.yuv420p.libaom-av1.mp4") throw SkipTestException("AV1 decoding not supported in this version of FFmpeg - SKIP"); else throw e;

It's more informative for debugging in case of real exception.

I should have paid more attention to the error

[ERROR:[email protected]] global video_decoder.cpp:127 create Video source is not supported by hardware video decoder.

FFmpeg is failing to parse the file but VideoReader is falling back to CuvidVideoSource which is able to parse the file. Then its failing because your GPU doesn't support AV1 decoding meaning the new error message in the test is wrong.

Because the exceptions from the decoder are caught and logged it is difficult without modification to assess what the exact cause of the failure is, hence the additional logging of the above message and the general exception which is thrown from the main thread

error: (-2:Unspecified error) Parsing/Decoding video source failed, check GPU memory is available and GPU supports hardware decoding.

Should I at the very least modify the skip test exception to output the above instead of

"AV1 decoding not supported in this version of FFmpeg - SKIP"

?

Going forward I could modify the decoder to internally log that the codec (or other parameters) are not supported by the Nvidia hardware decoder making this information available at test time, what do you think?

I should have added (seeing opencv/opencv#23540 (comment)) that the error is also thrown after construction. This will be rectified so that the exception is thrown in by the constructor once #3486 (itself a direct result of similar discussions like this we've had in the past) is merged.

asmorkalov · 2023-06-21T19:47:00Z

@cudawarped I merged your PR with awaiting of CUDA codec initialization and rebased the PR on top of it. The issue is still there:

[ RUN      ] CUDA_Codec/Video.Reader/8, where GetParam() = (NVIDIA GeForce GTX 1080, "highgui/video/sample_322x242_15frames.yuv420p.libaom-av1.mp4")
[ERROR:[email protected]] global cap_ffmpeg_impl.hpp:1237 open Could not find decoder for codec_id=32797
[ERROR:[email protected]] global cap_ffmpeg_impl.hpp:1286 open VIDEOIO/FFMPEG: Failed to initialize VideoCapture
[ WARN:[email protected]] global cap.cpp:206 open VIDEOIO(FFMPEG): backend is generally available but can't be used to capture by name
[ERROR:[email protected]] global video_decoder.cpp:127 create Video source is not supported by hardware video decoder.
[ERROR:[email protected]] global video_parser.cpp:169 HandleVideoSequence Attempt to configure Nvidia decoder failed!
[ERROR:[email protected]] global video_parser.cpp:83 parseVideoData Call to cuvidParseVideoData failed!
unknown file: Failure
C++ exception with description "AV1 decoding not supported in this version of FFmpeg - SKIP" thrown in the test body.
[  FAILED  ] CUDA_Codec/Video.Reader/8, where GetParam() = (NVIDIA GeForce GTX 1080, "highgui/video/sample_322x242_15frames.yuv420p.libaom-av1.mp4") (66 ms)
[ RUN      ] CUDA_Codec/Video.Reader/9, where GetParam() = (NVIDIA GeForce GTX 1080, "cv/tracking/faceocc2/data/faceocc2.webm")
[       OK ] CUDA_Codec/Video.Reader/9 (147 ms)
[ RUN      ] CUDA_Codec/Video.Reader/10, where GetParam() = (NVIDIA GeForce GTX 1080, "highgui/video/sample_322x242_15frames.yuv420p.mpeg2video.mp4")
unknown file: Failure
C++ exception with description "OpenCV(4.7.0-dev) /home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_reader.cpp:75: error: (-217:Gpu API call) NPP_CUDA_KERNEL_EXECUTION_ERROR [Code = -1000] in function 'cvtFromNv12'
" thrown in the test body.
[  FAILED  ] CUDA_Codec/Video.Reader/10, where GetParam() = (NVIDIA GeForce GTX 1080, "highgui/video/sample_322x242_15frames.yuv420p.mpeg2video.mp4") (73 ms)
[ RUN      ] CUDA_Codec/Video.Reader/11, where GetParam() = (NVIDIA GeForce GTX 1080, "highgui/video/sample_322x242_15frames.yuv420p.mjpeg.mp4")
Segmentation fault (core dumped)

System information: Ubuntu 18.04, CUDA 10.2. CMake details:

"  Video I/O:\n"
"    DC1394:                      YES (2.2.5)\n"
"    FFMPEG:                      YES\n"
"      avcodec:                   YES (57.107.100)\n"
"      avformat:                  YES (57.83.100)\n"
"      avutil:                    YES (55.78.100)\n"
"      swscale:                   YES (4.8.100)\n"
"      avresample:                YES (3.7.0)\n"
"    GStreamer:                   YES (1.14.5)\n"
"    v4l/v4l2:                    YES (linux/videodev2.h)\n"
...
"  NVIDIA CUDA:                   YES (ver 10.2, CUFFT CUBLAS NVCUVID NVCUVENC)\n"
"    NVIDIA GPU arch:             61\n"
"    NVIDIA PTX archs:\n"

cudawarped · 2023-06-22T04:54:20Z

@asmorkalov That's a different error. The first error is now emited from the constructor and caught in the test. I'm investigating the second one now I just have to find a 10 series GPU first.

asmorkalov · 2023-06-22T06:32:10Z

I' working on incomming 4.8.0 release now and need to rebuild ffmpeg wrapper for Windows. I propose to disable the test case for now and merge both patches in main and contrib and then propose bug fix to contrib only. Also I'll debug it on my side too.

cudawarped · 2023-06-22T07:58:35Z

I've disabled the following files for the CUDA_Codec/Video.Reader/ test

sample_322x242_15frames.yuv420p.libaom-av1.mp4 - av1 not supported until Ampere (2021)
faceocc2.webm - vp8 not fully supported until Turing (2018) does support your GTX 1080
sample_322x242_15frames.yuv420p.mpeg2video.mp4 - should be supported on your GTX 1080, not working on my GTX 1060
sample_322x242_15frames.yuv420p.mjpeg.mp4 - not working on my GTX 1060

Intrestingly on my GTX 1060 in windows sample_322x242_15frames.yuv420p.mpeg2video.mp4 fails gracefully (no exception). I'm wondering if the exception could be due to lack of support in the Nvidia driver? Can you post your driver version and the info from the top of the test please? e.g.

Device 0: "NVIDIA GeForce GTX1060"
CUDA Driver Version / Runtime Version 12.10 / 10.20

asmorkalov · 2023-06-22T09:16:21Z

All test cases besides libaom-av1 works on my 1080. Details on versions:

libnvidia-cfg1-470/now 470.161.03-0ubuntu0.18.04.1 amd64 
libnvidia-common-470/now 470.161.03-0ubuntu0.18.04.1 all 
libnvidia-compute-470/now 470.161.03-0ubuntu0.18.04.1 amd64 
libnvidia-container-tools/bionic,now 1.3.3-1 amd64 
libnvidia-container1/bionic,now 1.3.3-1 amd64 
libnvidia-decode-470/now 470.161.03-0ubuntu0.18.04.1 amd64 
libnvidia-encode-470/now 470.161.03-0ubuntu0.18.04.1 amd64 
libnvidia-extra-470/now 470.161.03-0ubuntu0.18.04.1 amd64 
libnvidia-fbc1-470/now 470.161.03-0ubuntu0.18.04.1 amd64 
libnvidia-gl-470/now 470.161.03-0ubuntu0.18.04.1 amd64 
libnvidia-ifr1-470/now 470.161.03-0ubuntu0.18.04.1 amd64 
nvidia-compute-utils-470/now 470.161.03-0ubuntu0.18.04.1 amd64 
nvidia-container-runtime/bionic,now 3.4.2-1 amd64
nvidia-container-toolkit/bionic,now 1.4.2-1 amd64 
nvidia-dkms-470/now 470.161.03-0ubuntu0.18.04.1 amd64 
nvidia-driver-470/now 470.161.03-0ubuntu0.18.04.1 amd64 
nvidia-kernel-common-470/now 470.161.03-0ubuntu0.18.04.1 amd64 
nvidia-kernel-source-470/now 470.161.03-0ubuntu0.18.04.1 amd64 
nvidia-modprobe/неизвестно,now 520.61.05-0ubuntu1 amd64
nvidia-prime/now 0.8.8.2 all 
nvidia-settings/неизвестно,now 520.61.05-0ubuntu1 amd64
nvidia-utils-470/now 470.161.03-0ubuntu0.18.04.1 amd64 
xserver-xorg-video-nvidia-470/now 470.161.03-0ubuntu0.18.04.1 amd64

asmorkalov · 2023-06-22T09:17:59Z

Cuda configuration reported by test:

./bin/opencv_test_cudacodec 

Available options besides google test option: 
Usage: opencv_test_cudacodec [params] 

        --cuda_device (value:-1)
                CUDA device on which tests will be executed (-1 means all devices)
        -h, --help (value:false)
                Print help info

Run tests on all supported CUDA devices 

*** CUDA Device Query (Runtime API) version (CUDART static linking) *** 

Device count: 1

Device 0: "NVIDIA GeForce GTX 1080"
  CUDA Driver Version / Runtime Version          11.40 / 10.20
  CUDA Capability Major/Minor version number:    6.1
  Total amount of global memory:                 8118 MBytes (8511881216 bytes)
  GPU Clock Speed:                               1.77 GHz
  Max Texture Dimension Size (x,y,z)             1D=(131072), 2D=(131072,65536), 3D=(16384,16384,16384)
  Max Layered Texture Size (dim) x layers        1D=(32768) x 2048, 2D=(32768,32768) x 2048
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per block:           1024
  Maximum sizes of each dimension of a block:    1024 x 1024 x 64
  Maximum sizes of each dimension of a grid:     2147483647 x 65535 x 65535
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and execution:                 Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Concurrent kernel execution:                   Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support enabled:                No
  Device is using TCC driver mode:               No
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Bus ID / PCI location ID:           1 / 0
  Compute Mode:
      Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) 

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version  = 11.40, CUDA Runtime Version = 10.20, NumDevs = 1

CTEST_FULL_OUTPUT
OpenCV version: 4.7.0-dev
OpenCV VCS version: 4.7.0-441-g024c836462
Build type: Release
Compiler: /usr/bin/c++  (ver 7.5.0)
Parallel framework: pthreads (nthreads=4)
CPU features: SSE? SSE2? SSE3? *SSE4.1? *SSE4.2? *FP16? *AVX? *AVX2? *AVX512-SKX?
Intel(R) IPP version: disabled
OpenCL is disabled
TEST: Skip tests with tags: 'mem_6gb', 'verylong'

cudawarped · 2023-06-22T09:22:31Z

All test cases besides libaom-av1 works on my 1080. Details on versions:

Are you sure from the test output above, repeated below

[ RUN ] CUDA_Codec/Video.Reader/10, where GetParam() = (NVIDIA GeForce GTX 1080, "highgui/video/sample_322x242_15frames.yuv420p.mpeg2video.mp4")
unknown file: Failure
C++ exception with description "OpenCV(4.7.0-dev) /home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_reader.cpp:75: error: (-217:Gpu API call) NPP_CUDA_KERNEL_EXECUTION_ERROR [Code = -1000] in function 'cvtFromNv12'
" thrown in the test body.
[ FAILED ] CUDA_Codec/Video.Reader/10, where GetParam() = (NVIDIA GeForce GTX 1080, "highgui/video/sample_322x242_15frames.yuv420p.mpeg2video.mp4") (73 ms)

sample_322x242_15frames.yuv420p.mpeg2video.mp4 is also throwing an exception from here (which is also an error but shouldn't be causing the observed issue see #3513 for details)?

On my GTX 1060 it returns with an empty frame instead of throwing an exception.

asmorkalov · 2023-06-22T09:23:13Z

@cudawarped I merged the minimal set of tests. Let's debug the remaining cases and resolve found issues one-by-one.

asmorkalov self-requested a review May 3, 2023 07:08

asmorkalov added the category: cuda label May 3, 2023

cudawarped mentioned this pull request Jun 6, 2023

Add Support for Fragmented MP4 (FMP4) #3504

Closed

6 tasks

cudawarped force-pushed the fix_cudacodec_codec_resolution branch from a42b8be to 461d672 Compare June 16, 2023 08:32

asmorkalov approved these changes Jun 20, 2023

View reviewed changes

asmorkalov changed the title ~~cudacodec::VideoReader - update inline with opencv/23540~~ cudacodec::VideoReader FORCC update with main #23540 Jun 20, 2023

asmorkalov mentioned this pull request Jun 20, 2023

VideoCapture: change CAP_PROP_FOURCC to fix #22876 opencv/opencv#23540

Merged

6 tasks

asmorkalov self-assigned this Jun 20, 2023

asmorkalov self-requested a review June 20, 2023 09:18

cudawarped force-pushed the fix_cudacodec_codec_resolution branch from 461d672 to b4c6c25 Compare June 20, 2023 12:06

cudawarped force-pushed the fix_cudacodec_codec_resolution branch from b4c6c25 to d6c74a1 Compare June 20, 2023 12:25

asmorkalov reviewed Jun 20, 2023

View reviewed changes

cudawarped force-pushed the fix_cudacodec_codec_resolution branch from d6c74a1 to b12dfa6 Compare June 20, 2023 13:06

cudacodec::VideoReader - amend FFmpeg codec resolution

72080b8

cudawarped force-pushed the fix_cudacodec_codec_resolution branch from b12dfa6 to 72080b8 Compare June 22, 2023 07:39

asmorkalov merged commit 9c10a6d into opencv:4.x Jun 22, 2023

cudawarped mentioned this pull request Jun 22, 2023

cudacodec::VideoReader - fix CUDART_VERSION tests #3513

Merged

6 tasks

cudawarped mentioned this pull request Jul 19, 2023

VideoCapture: remove decoder initialization when demuxing opencv/opencv#24012

Merged

6 tasks

asmorkalov mentioned this pull request Jul 27, 2023

(5.x) Merge 4.x #3538

Merged

cudawarped mentioned this pull request Dec 30, 2023

cudacodec: Enable features available with updated ffmpeg dll #3615

Merged

6 tasks

cudacodec::VideoReader FORCC update with main #23540 #3488

cudacodec::VideoReader FORCC update with main #23540 #3488

Uh oh!

Conversation

cudawarped commented Apr 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

cudawarped commented Jun 16, 2023

Uh oh!

asmorkalov left a comment

Choose a reason for hiding this comment

Uh oh!

asmorkalov commented Jun 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asmorkalov commented Jun 20, 2023

Uh oh!

cudawarped commented Jun 20, 2023

Uh oh!

cudawarped commented Jun 20, 2023

Uh oh!

asmorkalov commented Jun 20, 2023

Uh oh!

asmorkalov Jun 20, 2023

Choose a reason for hiding this comment

Uh oh!

cudawarped Jun 20, 2023

Choose a reason for hiding this comment

Uh oh!

cudawarped Jun 20, 2023

Choose a reason for hiding this comment

Uh oh!

asmorkalov commented Jun 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cudawarped commented Jun 22, 2023

Uh oh!

asmorkalov commented Jun 22, 2023

Uh oh!

cudawarped commented Jun 22, 2023

Uh oh!

asmorkalov commented Jun 22, 2023

Uh oh!

asmorkalov commented Jun 22, 2023

Uh oh!

cudawarped commented Jun 22, 2023

Uh oh!

asmorkalov commented Jun 22, 2023

Uh oh!

Uh oh!

`cudacodec::VideoReader` FORCC update with main #23540 #3488

`cudacodec::VideoReader` FORCC update with main #23540 #3488

cudawarped commented Apr 30, 2023 •

edited

Loading

asmorkalov commented Jun 20, 2023 •

edited

Loading

asmorkalov commented Jun 21, 2023 •

edited

Loading