Make video_reader backend be available in the binaries

[fmassa]

We currently provide two backends for video-reading: pyav (which is based on PyAV python library) and video_reader (which uses a custom C++ implementation based on ffmpeg).
Due to complexities in the packaging of ffmpeg in the build process at the time of the video_reader release, for now if the user wants the faster video_reader backend they need to compile torchvision from source.

It would be preferable if we could instead make it more seamless to the user, by providing the necessary bits so that they can use the video_reader backend just by installing torchvision via either pip or conda.

[jamt9000]

What implications will this and #2596 have for users of the conda-forge version of ffmpeg with libx264? Would they be ABI compatible or not?

For example, even if it isn't using video_reader, a research project might include a conda environment.yml with torchvision and conda-forge::ffmpeg intended to be used for some pre-processing scripts, which would break if the pytorch ffmpeg can't open the same files.

Will the plan be to autodownload the (possibly slower) openh264 binaries from cisco like Firefox for the patent workaround?

[andfoy]

Hi @jamt9000, thanks for posting these questions. Right now we require a LGPL-licensed version of FFmpeg, so that torchvision can link against it, since this project has a BSD3 license, different from the version found in conda-forge (GPL). Here are some answers to your questions

Would they be ABI compatible or not?

With respect to the versions, we had to build FFmpeg 4.2, due to some SSE/MMX errors (#2650) on libswscale that were fixed after 4.3.1 was published. So basically we are linking against a different version from the one on conda-forge, which corresponds to 4.3.

a research project might include a conda environment.yml with torchvision and conda-forge::ffmpeg

In order to use the conda-forge FFmpeg (which contains libx264), one should recompile torchvision in order to prevent any unexpected binary errors. Until we cannot use the same ffmpeg version for PyAV and conda-forge, due to #2650, we need to link against ffmpeg 4.2. Once a new version of FFmpeg is available, we can ensure ABI compatibility between binaries.

Edit: Please see the next comment

Will the plan be to autodownload the (possibly slower) openh264 binaries from cisco like Firefox for the patent workaround?

Right now, we are not building openh264 ourselves, thus this discussion is open in both FFmpeg and OpenH264 conda-forge feedstocks: conda-forge/openh264-feedstock#5 and conda-forge/ffmpeg-feedstock#66

I hope that this covers your questions

[andfoy]

@jamt9000, I was looking at the FFmpeg documentation, and it states:

FFmpeg guarantees backward API and ABI compatibility for each library as long as its major version number is unchanged. This means that no public symbols will be removed or renamed. Types and names of the public struct members and values of public macros and enums will remain the same (unless they were explicitly declared as not part of the public API). Documented behavior will not change.

So, basically my assessment is completely wrong, so there shouldn't be any problem if you install ffmpeg/av from conda-forge.

[jamt9000]

Ok, I think I understand.

So the potential versions of ffmpeg one is likely to have installed through conda are:

anaconda (4.2.2, SONAME=libavcodec.so.58, GPL)

ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers built with gcc 7.3.0 (crosstool-NG 1.23.0.449-a04d0) configuration: --prefix=/home/ubuntu/anaconda3/envs/ffmpegtest --cc=/tmp/build/80754af9/ffmpeg_1587154242452/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc --disable-doc --enable-avresample --enable-gmp --enable-hardcoded-tables --enable-libfreetype --enable-libvpx --enable-pthreads --enable-libopus --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-static --enable-version3 --enable-zlib --enable-libmp3lame --disable-nonfree --enable-gpl --enable-gnutls --disable-openssl --enable-libopenh264 --enable-libx264

conda-forge (4.3.1, SONAME=libavcodec.so.58, GPL)

ffmpeg version 4.3.1 Copyright (c) 2000-2020 the FFmpeg developers built with gcc 7.5.0 (crosstool-NG 1.24.0.131_87df0e6_dirty) configuration: --prefix=/home/ubuntu/anaconda3/envs/ffmpegforge --cc=/home/conda/feedstock_root/build_artifacts/ffmpeg_1596712246804/_build_env/bin/x86_64-conda-linux-gnu-cc --disable-doc --disable-openssl --enable-avresample --enable-gnutls --enable-gpl --enable-hardcoded-tables --enable-libfreetype --enable-libopenh264 --enable-libx264 --enable-pic --enable-pthreads --enable-shared --enable-static --enable-version3 --enable-zlib --enable-libmp3lame

pytorch (4.3, SONAME=libavcodec.so.58, LGPL [segfaults and cannot encode x264?])

ffmpeg version 4.3 Copyright (c) 2000-2020 the FFmpeg developers built with gcc 7.3.0 (crosstool-NG 1.23.0.449-a04d0) configuration: --prefix=/home/ubuntu/anaconda3/envs/ffmpegtorch --cc=/opt/conda/conda-bld/ffmpeg_1597178665428/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc --disable-doc --disable-openssl --enable-avresample --enable-gnutls --enable-hardcoded-tables --enable-libfreetype --enable-libopenh264 --enable-pic --enable-pthreads --enable-shared --disable-static --enable-version3 --enable-zlib --enable-libmp3lame

pytorch (4.2, SONAME=libavcodec.so.58, LGPL [cannot encode x264?])

ffmpeg version 4.2 Copyright (c) 2000-2019 the FFmpeg developers built with gcc 7.3.0 (crosstool-NG 1.23.0.449-a04d0) configuration: --prefix=/home/ubuntu/anaconda3/envs/ffmpegtorch42 --cc=/opt/conda/conda-bld/ffmpeg_1600723013725/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc --disable-doc --disable-openssl --enable-avresample --enable-gnutls --enable-hardcoded-tables --enable-libfreetype --enable-libopenh264 --enable-pic --enable-pthreads --enable-shared --disable-static --enable-version3 --enable-zlib --enable-libmp3lame

Is that right? Which should at least be ABI compatible (in the eyes of the dynamic linker due to the same SONAME, and hopefully also in practice)

[andfoy]

By judging by the FFmpeg documentation, they should be ABI compatible, since the major version is the same

conda-forge (4.3.1, SONAME=libavcodec.so.58, GPL)]

Just for the record, this one also segfaults