Skip to content

Commit 81d1b03

Browse files
mayeutgrzanka
mayeut
authored and
grzanka
committed
[WIP] Manylinux2010 (pypa#279)
* Minimal usable merge of markrwilliams:manylinux2 into current pypa:master - The dockerfiles build successfully. - Compared to current master, there's a new intermediate image for glibc. As the x86_64 build depends on it, this must have a predefined name. In the PR the "naming prefix" is 'markrwilliams/manylinux2', therefore it's assumed for all images. - There's a problem with libcurl which has been (temporarily) resolved with script which redirects yum. See docker/build_scripts/build.sh for details. The following script is proven to work: pushd docker/glibc/ docker build -t markrwilliams/manylinux2:centos-6.9-no-vsyscall popd docker/build_scripts/prefetch.sh curl openssl docker build -t markrwilliams/manylinux2:x86_64 \ -f docker/Dockerfile-x86_64 docker/ docker build -t markrwilliams/manylinux2:i686 \ -f docker/Dockerfile-i686 docker/ * Clean-up and adjustments for manylinux2010 - docker/build_scripts/build.sh: - removed libncurses.devel; not part of PEP 571 - removed gpg as dependency, because yum already requires it - removed some /dev/null redirections, because they hinder debugging - docker/build_scripts/manylinux1-check.py - renamed to manylinux-check.py - changed names/comments to new version - the check now targets glibc 2.12 instead of 2.5 - .travis.yml - only changed the image name - TODO: x86_64 still needs a script for the no-vsyscall image dependency - docker/deploy.sh - changed image name - note: .rst files have not been touched * Fix for new release of pip (v10.0.0) breaking the build script - The freshly released version of pip breaks build_utils.sh which tries to get the latest and greatest. Added a condition to instead use the latest version below v10. * Merge upstream changes (4a20e4b); remove get-pip hack - Incorporated the lastest changes from the official repository. - That cleanup removed two packages from 'yum erase ...' which looks like an oversight. Readded. - Removed the hack which fixed the broken build when the new pip was released. This has been fixed upstream, see here: pypa/get-pip#19 * Re-add the two /dev/null redirections in build.sh * Merge upstream changes (6f3bc0b) * Merge upstream changes (3dd4551) * Merge upstream changes (4132165) * Adapt build.sh for the x86_64 two-stage build - Add platform-dependent clause - Assign the same prefix to the the no-vsyscall image (quay.io/pypa/...) - The dependent docker/Dockerfile-x86_64 needs this prefix in FROM ... * Getting Travis to play along - Increase in `travis_wait` time - Shunting build output off into a log file which can be inspected on failure. * Drop 32-bit support and upgrade to devtoolset-7 * Update README.rst manylinux2 -> manylinux2010 * Add comment regarding MANYLINUX2010_DEPS * Add comment for Software collection & EPEL usage * Move expat-devel to PYTHON_COMPILE_DEPS * Use local gpgkey for RPM-GPG-KEY-CentOS-6 * Update to devtoolset-8 * Replace travis wait by custom command Prevent stdout timeout. The global travis timeout is the only one to apply now. * Use travis-ci build stages and cache glibc build glibc patch is unlikely to change very often, use travis-ci cache mechanism in order not to rebuild it every time. also, use build stages so that when it does change, glibc build is run as a pre-step, it will probably help not reaching the time limit, or at least, glibc build will be cached for job restart.
1 parent 9b3f193 commit 81d1b03

16 files changed

+690
-91
lines changed

‎.travis.yml

Lines changed: 30 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -10,25 +10,40 @@ branches:
1010
except:
1111
- /^pyup-/
1212

13+
cache:
14+
directories:
15+
- $HOME/docker
16+
1317
env:
1418
global:
1519
# QUAY_USERNAME and QUAY_PASSWORD for docker image upload
1620
- secure: "lKaTzEL6UNiEfp+BWLOUILG9BMtjwEMpwt6Yag0cQGHix7qJ/ElZ0t3oFw6ZwuDmA5qceAXIdxHLUK9HGVI2MloLk8czGhjvtfJ4XhOxtEJRQ0VkDGPsKN4cfhB4ZjGo6GAPtNqStMyNiY7BZuTrZa7coDLCoUeYcOmTpi6pmd1rrkk725B9QCTuhFHbPhkuL2yu/Jk6WxkHJBKjmuZek+iQa7lRItgMrG0/319PXLvwIGGl00nLFy+Ly5Ciwzux4wuHLTySZQKu0H9FX81A7smM0FW/42kg3ckGa2qLxRw/Pi8Nm/aIk8LD0QXzI5N7HhFfidOTgDS8Mt1HgfxmTk4wUXZ/KvCCshqjimzMc/s9i9wPZX9UqqcfrpZkmwz8dzhm1bndN45ZOCy6xAYT6dzf8T4mLMDjVWSW4+DUoW4sYHRLVujjcMk7ybcwGV43VruPTJnc8XVAhT+VIMQkoPjhQmTOn8h82LRNGYtLa5RReCh9OPKVYB2Quz18FXMWgFt7A6VWudL0c7/8CusLvuo+pLcxt9pnV40rvu1YEohpEj8qR/qTSaDUBZM0J9SVf5zrZR80pZUnXkDF8nm+mcLOTley3YWipU19lCR7dzVyCAiQdVAuNPdnyem3Yk8enGkAJbfLd6eaIDs+p73D0JXh1Nx1px1movVLQH3ohIw="
1721
- secure: "w1614pomHLltkBhqWM2bOvbymFWIWKqSqqIBDvaNn9tbQScioItJoELBT7g7+cD7nyU7OvpQ1U2fk0xVkCeNvYU0xS1vP4o/VnZRpup7f7Tkiq+2rf4fjwYr3HHnJjwak1l9bsw6FkgzKaVvSdiUJHMVxiIuLd3fVozR7qjBBhTDxSlWGOpSgd+ttpgMZwU5zQjdaVQr1D7E8M0979ZnWMrNRyLiAUeHaPILS815b+ijgqR+i5nmu0/FTCGM9Ik4KIzIfWq8AdfPdbRiq8c+LrrTPfyKcIQJaHmfduYRM4LycGWwzkXFBNtLrJ7uFLG9RDVemOHuHOWIJX8qCUIV4XuESXxH3fUQr6r+yxquTJbzXxNtoaLa6tBOTQWKDrRjT4z9Mf9Im14F2V59EUDoQowHx5bjunOH5wg3ruYNKYYBFRYra5kx0CkKrqFBzyl8fTUEQLyx1HWTVUC1WTXEeD/aFKOSIxW5DxZr5W4LLlW2+Raa52ZzY28Q6AdueFQCRzoJ70/GsJRlSsBdWNOHN4gSp1cZuToLWY15y64QhAMVDpikB+V4hmkbceLiTqeWzTStNL1sa32RHr6i/9zeFZw1pMD1+eOg9x6fgODfh2sqr/zPbu2oONsHnc4D2jwsEax4o+Dv5QHLvK7jdyWUmu47a9QReoexXK60jZXs3CA="
1822

19-
20-
matrix:
23+
jobs:
2124
include:
22-
- env: PLATFORM="i686"
23-
- env: PLATFORM="x86_64"
24-
25-
script:
26-
- PLATFORM=$PLATFORM TRAVIS_COMMIT=$TRAVIS_COMMIT ./build.sh
27-
28-
29-
deploy:
30-
provider: script
31-
script: docker/deploy.sh
32-
on:
33-
branch: master
34-
repo: pypa/manylinux
25+
- stage: "Patch glibc"
26+
env:
27+
- PLATFORM="x86_64"
28+
before_install:
29+
# Load cached docker images
30+
- if [[ -d $HOME/docker ]]; then ls $HOME/docker/*.tar.gz | xargs -I {file} sh -c "zcat {file} | docker load"; fi
31+
script:
32+
- PLATFORM=$PLATFORM TRAVIS_COMMIT=$TRAVIS_COMMIT ./build.sh glibc_only
33+
before_cache:
34+
# Save tagged docker images
35+
- mkdir -p $HOME/docker && docker images -a --filter='dangling=false' --format '{{.Repository}}:{{.Tag}} {{.ID}}' | grep 'centos-with-vsyscall:latest' | xargs -n 2 -t sh -c 'test -e $HOME/docker/$1.tar.gz || docker save $0 | gzip -2 > $HOME/docker/$1.tar.gz'
36+
- stage: "Build manylinux images"
37+
env:
38+
- PLATFORM="x86_64"
39+
before_install:
40+
# Load cached docker images
41+
- if [[ -d $HOME/docker ]]; then ls $HOME/docker/*.tar.gz | xargs -I {file} sh -c "zcat {file} | docker load"; fi
42+
script:
43+
- PLATFORM=$PLATFORM TRAVIS_COMMIT=$TRAVIS_COMMIT ./build.sh
44+
deploy:
45+
provider: script
46+
script: docker/deploy.sh
47+
on:
48+
branch: master
49+
repo: pypa/manylinux

‎build.sh

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,18 @@
33
# Stop at any error, show all commands
44
set -ex
55

6+
7+
if [ $PLATFORM == x86_64 ] || [ "$1" == "glibc_only" ]; then
8+
# Output something every 10 minutes or Travis kills the job
9+
while sleep 9m; do echo -n -e " \b"; done &
10+
docker build --rm -t centos-with-vsyscall:latest --cache-from centos-with-vsyscall:latest --target centos-with-vsyscall -f docker/glibc/Dockerfile docker/glibc/
11+
# Killing background sleep loop
12+
kill %1
13+
if [ "$1" == "glibc_only" ]; then
14+
exit 0
15+
fi
16+
docker build --rm -t quay.io/pypa/manylinux2010_centos-6-no-vsyscall --cache-from quay.io/pypa/manylinux2010_centos-6-no-vsyscall:latest --cache-from centos-with-vsyscall:latest -f docker/glibc/Dockerfile docker/glibc/
17+
fi
18+
619
docker/build_scripts/prefetch.sh openssl curl
7-
docker build --rm -t quay.io/pypa/manylinux1_$PLATFORM:$TRAVIS_COMMIT -f docker/Dockerfile-$PLATFORM docker/
20+
docker build --rm -t quay.io/pypa/manylinux2010_$PLATFORM:$TRAVIS_COMMIT -f docker/Dockerfile-$PLATFORM docker/

‎docker/Dockerfile-i686

Lines changed: 0 additions & 20 deletions
This file was deleted.

‎docker/Dockerfile-x86_64

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,13 @@
1-
FROM centos:5
2-
MAINTAINER The ManyLinux project
1+
# See docker/glibc/
2+
FROM quay.io/pypa/manylinux2010_centos-6-no-vsyscall
3+
LABEL maintainer="The ManyLinux project"
34

45
ENV LC_ALL en_US.UTF-8
56
ENV LANG en_US.UTF-8
67
ENV LANGUAGE en_US.UTF-8
7-
ENV PATH /opt/rh/devtoolset-2/root/usr/bin:$PATH
8-
ENV LD_LIBRARY_PATH /opt/rh/devtoolset-2/root/usr/lib64:/opt/rh/devtoolset-2/root/usr/lib:/usr/local/lib64:/usr/local/lib
9-
ENV PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
8+
ENV PATH /opt/rh/devtoolset-8/root/usr/bin:$PATH
9+
ENV LD_LIBRARY_PATH /opt/rh/devtoolset-8/root/usr/lib64:/opt/rh/devtoolset-8/root/usr/lib:/opt/rh/devtoolset-8/root/usr/lib64/dyninst:/opt/rh/devtoolset-8/root/usr/lib/dyninst:/usr/local/lib64:/usr/local/lib
10+
ENV PKG_CONFIG_PATH /usr/local/lib/pkgconfig
1011

1112
COPY build_scripts /build_scripts
1213
COPY sources /

‎docker/build_scripts/build.sh

Lines changed: 52 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -10,20 +10,25 @@ MY_DIR=$(dirname "${BASH_SOURCE[0]}")
1010

1111
# Dependencies for compiling Python that we want to remove from
1212
# the final image after compiling Python
13-
# GPG installed to verify signatures on Python source tarballs.
14-
PYTHON_COMPILE_DEPS="zlib-devel bzip2-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel gpg libffi-devel"
15-
16-
# Libraries that are allowed as part of the manylinux1 profile
17-
MANYLINUX1_DEPS="glibc-devel libstdc++-devel glib2-devel libX11-devel libXext-devel libXrender-devel mesa-libGL-devel libICE-devel libSM-devel ncurses-devel"
18-
19-
# Centos 5 is EOL and is no longer available from the usual mirrors, so switch
20-
# to http://vault.centos.org
21-
# From: https://github.com/rust-lang/rust/pull/41045
22-
# The location for version 5 was also removed, so now only the specific release
23-
# (5.11) can be referenced.
24-
sed -i 's/enabled=1/enabled=0/' /etc/yum/pluginconf.d/fastestmirror.conf
25-
sed -i 's/mirrorlist/#mirrorlist/' /etc/yum.repos.d/*.repo
26-
sed -i 's/#\(baseurl.*\)mirror.centos.org\/centos\/$releasever/\1vault.centos.org\/5.11/' /etc/yum.repos.d/*.repo
13+
PYTHON_COMPILE_DEPS="zlib-devel bzip2-devel expat-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel"
14+
15+
# Libraries that are allowed as part of the manylinux2010 profile
16+
# Extract from PEP: https://www.python.org/dev/peps/pep-0571/#the-manylinux2010-policy
17+
# On RPM-based systems, they are provided by these packages:
18+
# Package: Libraries
19+
# glib2: libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0
20+
# glibc: libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1, libcrypt.so.1, libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6
21+
# libICE: libICE.so.6
22+
# libX11: libX11.so.6
23+
# libXext: libXext.so.6
24+
# libXrender: libXrender.so.1
25+
# libgcc: libgcc_s.so.1
26+
# libstdc++: libstdc++.so.6
27+
# mesa: libGL.so.1
28+
#
29+
# PEP is missing the package for libSM.so.6 for RPM based system
30+
# Install development packages (except for libgcc which is provided by gcc install)
31+
MANYLINUX2010_DEPS="glibc-devel libstdc++-devel glib2-devel libX11-devel libXext-devel libXrender-devel mesa-libGL-devel libICE-devel libSM-devel"
2732

2833
# Get build utilities
2934
source $MY_DIR/build_utils.sh
@@ -41,39 +46,27 @@ echo "multilib_policy=best" >> /etc/yum.conf
4146
# Decided not to clean at this point: https://github.com/pypa/manylinux/pull/129
4247
yum -y update
4348

44-
# EPEL support
45-
yum -y install wget
46-
# https://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
47-
cp $MY_DIR/epel-release-5-4.noarch.rpm .
48-
check_sha256sum epel-release-5-4.noarch.rpm $EPEL_RPM_HASH
49-
50-
# Dev toolset (for LLVM and other projects requiring C++11 support)
51-
wget -q http://people.centos.org/tru/devtools-2/devtools-2.repo
52-
check_sha256sum devtools-2.repo $DEVTOOLS_HASH
53-
mv devtools-2.repo /etc/yum.repos.d/devtools-2.repo
54-
rpm -Uvh --replacepkgs epel-release-5*.rpm
55-
rm -f epel-release-5*.rpm
56-
57-
# from now on, we shall only use curl to retrieve files
58-
yum -y erase wget
49+
# Software collection (for devtoolset-8) and EPEL support (for cmake28 & yasm)
50+
yum -y install centos-release-scl https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
5951

6052
# Development tools and libraries
6153
yum -y install \
6254
automake \
6355
bison \
6456
bzip2 \
6557
cmake28 \
66-
devtoolset-2-binutils \
67-
devtoolset-2-gcc \
68-
devtoolset-2-gcc-c++ \
69-
devtoolset-2-gcc-gfortran \
58+
devtoolset-8-binutils \
59+
devtoolset-8-gcc \
60+
devtoolset-8-gcc-c++ \
61+
devtoolset-8-gcc-gfortran \
7062
diffutils \
71-
expat-devel \
7263
gettext \
73-
kernel-devel-`uname -r` \
7464
file \
65+
kernel-devel-`uname -r` \
66+
libffi-devel \
7567
make \
7668
patch \
69+
perl-devel \
7770
unzip \
7871
which \
7972
yasm \
@@ -147,18 +140,39 @@ rm -rf patchelf.tar.gz patchelf-$PATCHELF_VERSION
147140

148141
ln -s $PY36_BIN/auditwheel /usr/local/bin/auditwheel
149142

143+
# HACK: The newly compiled and installed curl messes with the system's
144+
# py2.6 installation, on which yum depends. Work around it by
145+
# rewiring libcurl.so specifically for yum. /usr/local/bin/ has higher
146+
# priority on the PATH than /usr/bin/
147+
cat <<'EOF' > /usr/local/bin/yum && chmod +x /usr/local/bin/yum
148+
#!/bin/bash
149+
if [ "x$(arch)" != xi686 ]; then
150+
LD_PRELOAD=/usr/lib64/libcurl.so.4
151+
else
152+
LD_PRELOAD=/usr/lib/libcurl.so.4
153+
fi
154+
export LD_PRELOAD
155+
/usr/bin/yum "$@"
156+
EOF
157+
# the above might not shadow the real yum just yet, so call hash to be
158+
# sure:
159+
type yum
160+
hash yum
161+
162+
150163
# Clean up development headers and other unnecessary stuff for
151164
# final image
152165
yum -y erase \
153166
avahi \
154167
bitstream-vera-fonts \
155168
freetype \
169+
gettext \
156170
gtk2 \
157171
hicolor-icon-theme \
158172
libX11 \
159173
wireless-tools \
160-
${PYTHON_COMPILE_DEPS} > /dev/null 2>&1
161-
yum -y install ${MANYLINUX1_DEPS}
174+
${PYTHON_COMPILE_DEPS} > /dev/null 2>&1
175+
yum -y install ${MANYLINUX2010_DEPS}
162176
yum -y clean all > /dev/null 2>&1
163177
yum list installed
164178

@@ -180,7 +194,7 @@ find /opt/_internal -depth \
180194
for PYTHON in /opt/python/*/bin/python; do
181195
# Smoke test to make sure that our Pythons work, and do indeed detect as
182196
# being manylinux compatible:
183-
$PYTHON $MY_DIR/manylinux1-check.py
197+
$PYTHON $MY_DIR/manylinux-check.py
184198
# Make sure that SSL cert checking works
185199
$PYTHON $MY_DIR/ssl-check.py
186200
done

‎docker/build_scripts/build_env.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,5 +37,5 @@ GIT_HASH=ba2fed9d02e424b735e035c4f2b0bdb168ef0df7e35156b5051d900dc7247787
3737
GIT_DOWNLOAD_URL=https://github.com/git/git/archive
3838

3939
GET_PIP_URL=https://bootstrap.pypa.io/get-pip.py
40-
EPEL_RPM_HASH=0dcc89f9bf67a2a515bad64569b7a9615edc5e018f676a578d5fd0f17d3c81d4
40+
EPEL_RPM_HASH=e5ed9ecf22d0c4279e92075a64c757ad2b38049bcf5c16c4f2b75d5f6860dc0d
4141
DEVTOOLS_HASH=a8ebeb4bed624700f727179e6ef771dafe47651131a00a78b342251415646acc

‎docker/build_scripts/build_utils.sh

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -54,10 +54,7 @@ function do_cpython_build {
5454
if [ -e ${prefix}/bin/python3 ]; then
5555
ln -s python3 ${prefix}/bin/python
5656
fi
57-
# --force-reinstall is to work around:
58-
# https://github.com/pypa/pip/issues/5220
59-
# https://github.com/pypa/get-pip/issues/19
60-
${prefix}/bin/python get-pip.py --force-reinstall
57+
${prefix}/bin/python get-pip.py
6158
if [ -e ${prefix}/bin/pip3 ] && [ ! -e ${prefix}/bin/pip ]; then
6259
ln -s pip3 ${prefix}/bin/pip
6360
fi

‎docker/build_scripts/epel-release-5-4.noarch.rpm

-11.9 KB
Binary file not shown.

‎docker/build_scripts/manylinux1-check.py renamed to ‎docker/build_scripts/manylinux-check.py

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Logic copied from PEP 513
22

3-
def is_manylinux1_compatible():
3+
def is_manylinux2010_compatible():
44
# Only Linux, and only x86-64 / i686
55
from distutils.util import get_platform
66
if get_platform() not in ["linux-x86_64", "linux-i686"]:
@@ -14,8 +14,8 @@ def is_manylinux1_compatible():
1414
# Fall through to heuristic check below
1515
pass
1616

17-
# Check glibc version. CentOS 5 uses glibc 2.5.
18-
return have_compatible_glibc(2, 5)
17+
# Check glibc version. CentOS 6 uses glibc 2.12.
18+
return have_compatible_glibc(2, 12)
1919

2020
def have_compatible_glibc(major, minimum_minor):
2121
import ctypes
@@ -45,9 +45,9 @@ def have_compatible_glibc(major, minimum_minor):
4545
return True
4646

4747
import sys
48-
if is_manylinux1_compatible():
49-
print("%s is manylinux1 compatible" % (sys.executable,))
48+
if is_manylinux2010_compatible():
49+
print("%s is manylinux2010 compatible" % (sys.executable,))
5050
sys.exit(0)
5151
else:
52-
print("%s is NOT manylinux1 compatible" % (sys.executable,))
52+
print("%s is NOT manylinux2010 compatible" % (sys.executable,))
5353
sys.exit(1)

‎docker/deploy.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/bin/bash
22
docker login -u $QUAY_USERNAME -p $QUAY_PASSWORD quay.io
3-
tag="quay.io/pypa/manylinux1_$PLATFORM"
3+
tag="quay.io/pypa/manylinux2010_$PLATFORM"
44
docker tag ${tag}:${TRAVIS_COMMIT} ${tag}:latest
55
docker push ${tag}:latest

‎docker/glibc/Dockerfile

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
FROM centos:6 as centos-with-vsyscall
2+
3+
COPY ./build_scripts /build_scripts
4+
RUN bash /build_scripts/rebuild-glibc-without-vsyscall.sh
5+
6+
FROM centos:6
7+
LABEL maintainer="The Manylinux project"
8+
9+
COPY --from=centos-with-vsyscall /rpms /rpms
10+
11+
RUN yum -y install /rpms/* && rm -rf /rpms

0 commit comments

Comments
 (0)