Skip to content

remove explicit install of Pillow throughout CI #5950

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
May 10, 2022
Merged

remove explicit install of Pillow throughout CI #5950

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
d6f5d79
remove explicit install of Pillow throughout CI
pmeier May 5, 2022
cd69c86
explicitly install zlib in windows conda env
pmeier May 5, 2022
7b1efd8
check pillow on Windows
pmeier May 6, 2022
45d91d8
move check into conda env
pmeier May 6, 2022
b1c9f72
fix
pmeier May 6, 2022
77aabf7
more debug output
pmeier May 6, 2022
a3621e3
more debug
pmeier May 6, 2022
14c6c4c
add debug also to linux
pmeier May 6, 2022
2d34b3d
retry w/ and w/o packaging third party library
pmeier May 6, 2022
fcc944a
tag debug
pmeier May 6, 2022
a2ba9f5
more debug
pmeier May 6, 2022
1866da8
pin setuptools < 61
pmeier May 6, 2022
6585225
install setuptools from pip
pmeier May 6, 2022
2b416ab
cleanup
pmeier May 9, 2022
9ebdba6
try remove one more
pmeier May 9, 2022
4a0d6c7
Merge branch 'main' into pillow
datumbox May 9, 2022
44f3cde
Merge branch 'main' into pillow
pmeier May 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .circleci/config.yml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion .circleci/config.yml.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -691,7 +691,6 @@ jobs:
conda env remove -n python${PYTHON_VERSION} || true
conda create -yn python${PYTHON_VERSION} python=${PYTHON_VERSION}
conda activate python${PYTHON_VERSION}
conda install "Pillow>=5.3.0,!=8.3.*"
conda install -v -y -c pytorch-nightly pytorch
conda install -v -y $(ls ~/workspace/torchvision*.tar.bz2)
- run:
Expand Down
4 changes: 1 addition & 3 deletions .circleci/smoke_test/docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,7 @@ ENV PATH /opt/conda/bin:$PATH
RUN conda create -y --name python3.7 python=3.7
RUN conda create -y --name python3.8 python=3.8
RUN conda create -y --name python3.9 python=3.9
RUN conda create -y --name python3.10 python=3.10
SHELL [ "/bin/bash", "-c" ]
RUN echo "source /usr/local/etc/profile.d/conda.sh" >> ~/.bashrc
RUN source /usr/local/etc/profile.d/conda.sh && conda activate python3.7 && conda install -y "Pillow>=5.3.0,!=8.3.*"
RUN source /usr/local/etc/profile.d/conda.sh && conda activate python3.8 && conda install -y "Pillow>=5.3.0,!=8.3.*"
RUN source /usr/local/etc/profile.d/conda.sh && conda activate python3.9 && conda install -y "Pillow>=5.3.0,!=8.3.*"
CMD [ "/bin/bash"]
1 change: 0 additions & 1 deletion .circleci/unittest/linux/scripts/environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,5 @@ dependencies:
- h5py
- pip:
- future
- pillow >=5.3.0, !=8.3.*
- scipy
- av
3 changes: 1 addition & 2 deletions .circleci/unittest/windows/scripts/environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,9 @@ dependencies:
- jpeg
- ca-certificates
- hdf5
- setuptools == 58.0.4
- setuptools
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a refinement as proposed in #5868 (comment). See the next comment for details.

- pip:
- future
- pillow >=5.3.0, !=8.3.*
- scipy
- av != 9.1.1
- dataclasses
Expand Down
6 changes: 6 additions & 0 deletions .circleci/unittest/windows/scripts/setup_env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,9 @@ conda activate "${env_dir}"
# 3. Install Conda dependencies
printf "* Installing dependencies (except PyTorch)\n"
conda env update --file "${this_dir}/environment.yml" --prune

# 4. Downgrade setuptools on Python 3.7.
# See https://github.com/pytorch/vision/pull/5868
if [[ "${PYTHON_VERSION}" == '3.7' ]]; then
pip install --upgrade setuptools==58.0.4
fi
Comment on lines +43 to +45
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially, setuptools and similar packages used cp3_10 as the interpreter tag for Python 3.10. After PEP 641 was rejected, they changed to cp310. This patch was included in setuptools==59.1.0. With this tag set incorrectly, python setup.py develop ignores wheels on Python 3.10 and tries to use the source version. That is fine for pure python libraries, but means that it will try to compile Pillow from source.

Unfortunately, the defaults channel provides nothing between setuptools==58.0.4 and setuptools==61.2.0. The former has the problems described above and the latter has problems on Python 3.7 (see #5868). Thus, I opted to use the latest release in general and special case Python 3.7 here. When we hard-depend on Python 3.8, we can simply remove this.

2 changes: 1 addition & 1 deletion packaging/build_cmake.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ setup_conda_pytorch_constraint
setup_conda_cudatoolkit_plain_constraint

if [[ "$OSTYPE" == "msys" ]]; then
conda install -yq conda-build cmake "pillow>=5.3.0,!=8.3.*" future
conda install -yq conda-build cmake future
pip install dataclasses
fi

Expand Down