fix JPEG reference tests

[pmeier]

We are currently skipping JPEG reference tests on Windows, since the Windows wheels of Pillow are built against libjpeg-turbo while we build torchvision against libjpeg

vision/test/test_image.py

Line 481 in bbeb320

@pytest.mark.skipif(IS_WINDOWS, reason=("this test fails on windows because PIL uses libjpeg-turbo on windows"))

As of Pillow==9.0.0, all wheels are now built against libjpeg-turbo, which leads to the CI failures that are currently happening. Thus, these tests will now fail for every OS or at least only succeed by chance.

There are multiple ways out in descending order of preference by me:

1. Install Pillow from conda-forge rather from PyPI to enforce the same JPEG library for torchvision and Pillow.
2. Pin Pillow<9 for unittest CI. This basically preserves the behavior we had before.
3. Remove these tests completely.

The only downside of 1. that I see is that Pillow on conda-forge is released later than on PyPI. IIUC, the conda-forge package is community managed and they only start building when it is released on PyPI. If there are build failures it might take some time until the versions converge again. In the meantime, our tests would not pick up on any incompatibilities with Pillow in general not just for JPEG I/O. Thus, we would need to rely on user feedback that something is not working.

Given that this is an dependency issue, I would be fine with waiting for user reports in the time of Pillow version divergence between conda-forge and PyPI. If we don't want to accept that, we could also hide all JPEG compatibility tests behind a runtime flag and add another CI job that only runs these tests while enforcing 1. from above.

cc @pmeier @NicolasHug @datumbox

[NicolasHug]

Thanks a lot for the summary and for opening this issue @pmeier

In #5169 I temporarily skipped the offending tests unconditionally, for the CI to go back to green.

Paraphrasing what you mentioned about option 1: there's the risk that PIL releases a new version, and that we also release torchvision shortly after, before the conda-forge PIL channel gets updated with PIL's latest. In this case, we might miss bugs (related to jpeg, or to other parts where torchvision interacts with PIL, like the transforms).

It's also possible that the conda-forge channel decides to build against libjpeg-turbo instead of libjpeg one day, in which case we would be back to our original problem again (although at least for PIL 9 they're sticking to libjpeg it seems https://github.com/conda-forge/pillow-feedstock/pull/104/files)

Another option that we discussed internally is

4. build torchvision against libjpeg-turbo instead of libjpeg.

which could come with the benefit of speeding up the jpeg decoding. But there might be implications w.r.t. the PIL binary compatibility so this is something we'd have to plan with the release engineers.

As a temporary solution, would it be hard to pin one CI job (e.g. linux 3.8) to use libjpeg instead of libjpeg-turbo? Either by making it rely on conda, or by pinning it to PIL 8? This way we would be able to run offending tests on this one CI job, while still relying on the latest PIL for all other tests.

[pmeier]

would it be hard to pin one CI job [...] ?

Unfortunately, yes. All CI jobs use .circleci/unittest/{linux, windows}/environment.yml as base. It would be a lot easier to add a one-off job as explained in my top comment:

we could also hide all JPEG compatibility tests behind a runtime flag and add another CI job that only runs these tests while enforcing 1. from above.

[NicolasHug]

Thanks for the feedback, ok then, I'm happy to try and see what a dedicated job would look like. Would you be interested in submitting a PR?