Skip to content

Make video_reader backend be available in the binaries #2365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
fmassa opened this issue Jun 29, 2020 · 5 comments · Fixed by #2596
Closed

Make video_reader backend be available in the binaries #2365

fmassa opened this issue Jun 29, 2020 · 5 comments · Fixed by #2596
Assignees

Comments

@fmassa
Copy link
Member

fmassa commented Jun 29, 2020

We currently provide two backends for video-reading: pyav (which is based on PyAV python library) and video_reader (which uses a custom C++ implementation based on ffmpeg).
Due to complexities in the packaging of ffmpeg in the build process at the time of the video_reader release, for now if the user wants the faster video_reader backend they need to compile torchvision from source.

It would be preferable if we could instead make it more seamless to the user, by providing the necessary bits so that they can use the video_reader backend just by installing torchvision via either pip or conda.

@jamt9000
Copy link
Contributor

jamt9000 commented Sep 29, 2020

What implications will this and #2596 have for users of the conda-forge version of ffmpeg with libx264? Would they be ABI compatible or not?

For example, even if it isn't using video_reader, a research project might include a conda environment.yml with torchvision and conda-forge::ffmpeg intended to be used for some pre-processing scripts, which would break if the pytorch ffmpeg can't open the same files.

Will the plan be to autodownload the (possibly slower) openh264 binaries from cisco like Firefox for the patent workaround?

@andfoy
Copy link
Contributor

andfoy commented Sep 29, 2020

Hi @jamt9000, thanks for posting these questions. Right now we require a LGPL-licensed version of FFmpeg, so that torchvision can link against it, since this project has a BSD3 license, different from the version found in conda-forge (GPL). Here are some answers to your questions

Would they be ABI compatible or not?

With respect to the versions, we had to build FFmpeg 4.2, due to some SSE/MMX errors (#2650) on libswscale that were fixed after 4.3.1 was published. So basically we are linking against a different version from the one on conda-forge, which corresponds to 4.3.

a research project might include a conda environment.yml with torchvision and conda-forge::ffmpeg

In order to use the conda-forge FFmpeg (which contains libx264), one should recompile torchvision in order to prevent any unexpected binary errors. Until we cannot use the same ffmpeg version for PyAV and conda-forge, due to #2650, we need to link against ffmpeg 4.2. Once a new version of FFmpeg is available, we can ensure ABI compatibility between binaries.

Edit: Please see the next comment

Will the plan be to autodownload the (possibly slower) openh264 binaries from cisco like Firefox for the patent workaround?

Right now, we are not building openh264 ourselves, thus this discussion is open in both FFmpeg and OpenH264 conda-forge feedstocks: conda-forge/openh264-feedstock#5 and conda-forge/ffmpeg-feedstock#66

I hope that this covers your questions

@andfoy
Copy link
Contributor

andfoy commented Sep 29, 2020

@jamt9000, I was looking at the FFmpeg documentation, and it states:

FFmpeg guarantees backward API and ABI compatibility for each library as long as its major version number is unchanged. This means that no public symbols will be removed or renamed. Types and names of the public struct members and values of public macros and enums will remain the same (unless they were explicitly declared as not part of the public API). Documented behavior will not change.

So, basically my assessment is completely wrong, so there shouldn't be any problem if you install ffmpeg/av from conda-forge.

@jamt9000
Copy link
Contributor

jamt9000 commented Sep 30, 2020

Ok, I think I understand.

So the potential versions of ffmpeg one is likely to have installed through conda are:

anaconda (4.2.2, SONAME=libavcodec.so.58, GPL) ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers built with gcc 7.3.0 (crosstool-NG 1.23.0.449-a04d0) configuration: --prefix=/home/ubuntu/anaconda3/envs/ffmpegtest --cc=/tmp/build/80754af9/ffmpeg_1587154242452/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc --disable-doc --enable-avresample --enable-gmp --enable-hardcoded-tables --enable-libfreetype --enable-libvpx --enable-pthreads --enable-libopus --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-static --enable-version3 --enable-zlib --enable-libmp3lame --disable-nonfree --enable-gpl --enable-gnutls --disable-openssl --enable-libopenh264 --enable-libx264
conda-forge (4.3.1, SONAME=libavcodec.so.58, GPL) ffmpeg version 4.3.1 Copyright (c) 2000-2020 the FFmpeg developers built with gcc 7.5.0 (crosstool-NG 1.24.0.131_87df0e6_dirty) configuration: --prefix=/home/ubuntu/anaconda3/envs/ffmpegforge --cc=/home/conda/feedstock_root/build_artifacts/ffmpeg_1596712246804/_build_env/bin/x86_64-conda-linux-gnu-cc --disable-doc --disable-openssl --enable-avresample --enable-gnutls --enable-gpl --enable-hardcoded-tables --enable-libfreetype --enable-libopenh264 --enable-libx264 --enable-pic --enable-pthreads --enable-shared --enable-static --enable-version3 --enable-zlib --enable-libmp3lame
pytorch (4.3, SONAME=libavcodec.so.58, LGPL [segfaults and cannot encode x264?]) ffmpeg version 4.3 Copyright (c) 2000-2020 the FFmpeg developers built with gcc 7.3.0 (crosstool-NG 1.23.0.449-a04d0) configuration: --prefix=/home/ubuntu/anaconda3/envs/ffmpegtorch --cc=/opt/conda/conda-bld/ffmpeg_1597178665428/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc --disable-doc --disable-openssl --enable-avresample --enable-gnutls --enable-hardcoded-tables --enable-libfreetype --enable-libopenh264 --enable-pic --enable-pthreads --enable-shared --disable-static --enable-version3 --enable-zlib --enable-libmp3lame
pytorch (4.2, SONAME=libavcodec.so.58, LGPL [cannot encode x264?]) ffmpeg version 4.2 Copyright (c) 2000-2019 the FFmpeg developers built with gcc 7.3.0 (crosstool-NG 1.23.0.449-a04d0) configuration: --prefix=/home/ubuntu/anaconda3/envs/ffmpegtorch42 --cc=/opt/conda/conda-bld/ffmpeg_1600723013725/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc --disable-doc --disable-openssl --enable-avresample --enable-gnutls --enable-hardcoded-tables --enable-libfreetype --enable-libopenh264 --enable-pic --enable-pthreads --enable-shared --disable-static --enable-version3 --enable-zlib --enable-libmp3lame

Is that right? Which should at least be ABI compatible (in the eyes of the dynamic linker due to the same SONAME, and hopefully also in practice)

@andfoy
Copy link
Contributor

andfoy commented Sep 30, 2020

By judging by the FFmpeg documentation, they should be ABI compatible, since the major version is the same

conda-forge (4.3.1, SONAME=libavcodec.so.58, GPL)]

Just for the record, this one also segfaults

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants