Skip to content

[manylinux2010] glibc update & glibc cache/image size reduction #874

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Dec 13, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ jobs:
script:
- PLATFORM=$PLATFORM docker/glibc/build.sh all
before_cache:
- travis-ci/cache_images.sh combined
- travis-ci/cache_images.sh combined_build
- <<: *manylinux-build
env:
- PLATFORM="x86_64"
Expand Down
57 changes: 34 additions & 23 deletions docker/glibc/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,29 +1,28 @@
FROM centos:6
LABEL maintainer="The Manylinux project"
FROM centos:6 AS manylinux2010_centos-6-no-vsyscall-build

# do not install debuginfo and what x86_64 already provides
COPY --from=quay.io/pypa/manylinux2010_centos-6-with-vsyscall32:latest \
/rpms/glibc-2.12-1.212.1.el6.i686.rpm \
#/rpms/glibc-common-2.12-1.212.1.el6.i686.rpm \
#/rpms/glibc-debuginfo-2.12-1.212.1.el6.i686.rpm \
#/rpms/glibc-debuginfo-common-2.12-1.212.1.el6.i686.rpm \
/rpms/glibc-devel-2.12-1.212.1.el6.i686.rpm \
#/rpms/glibc-headers-2.12-1.212.1.el6.i686.rpm \
/rpms/glibc-static-2.12-1.212.1.el6.i686.rpm \
#/rpms/glibc-utils-2.12-1.212.1.el6.i686.rpm \
#/rpms/nscd-2.12-1.212.1.el6.i686.rpm \
/rpms/glibc-2.12-1.212.1.el6_10.3.i686.rpm \
#/rpms/glibc-common-2.12-1.212.1.el6_10.3.i686.rpm \
#/rpms/glibc-debuginfo-2.12-1.212.1.el6_10.3.i686.rpm \
#/rpms/glibc-debuginfo-common-2.12-1.212.1.el6_10.3.i686.rpm \
/rpms/glibc-devel-2.12-1.212.1.el6_10.3.i686.rpm \
#/rpms/glibc-headers-2.12-1.212.1.el6_10.3.i686.rpm \
/rpms/glibc-static-2.12-1.212.1.el6_10.3.i686.rpm \
#/rpms/glibc-utils-2.12-1.212.1.el6_10.3.i686.rpm \
#/rpms/nscd-2.12-1.212.1.el6_10.3.i686.rpm \
/rpms/
# do not install debuginfo
COPY --from=quay.io/pypa/manylinux2010_centos-6-with-vsyscall64:latest \
/rpms/glibc-2.12-1.212.1.el6.x86_64.rpm \
/rpms/glibc-common-2.12-1.212.1.el6.x86_64.rpm \
#/rpms/glibc-debuginfo-2.12-1.212.1.el6.x86_64.rpm \
#/rpms/glibc-debuginfo-common-2.12-1.212.1.el6.x86_64.rpm \
/rpms/glibc-devel-2.12-1.212.1.el6.x86_64.rpm \
/rpms/glibc-headers-2.12-1.212.1.el6.x86_64.rpm \
/rpms/glibc-static-2.12-1.212.1.el6.x86_64.rpm \
/rpms/glibc-utils-2.12-1.212.1.el6.x86_64.rpm \
/rpms/nscd-2.12-1.212.1.el6.x86_64.rpm \
/rpms/glibc-2.12-1.212.1.el6_10.3.x86_64.rpm \
/rpms/glibc-common-2.12-1.212.1.el6_10.3.x86_64.rpm \
#/rpms/glibc-debuginfo-2.12-1.212.1.el6_10.3.x86_64.rpm \
#/rpms/glibc-debuginfo-common-2.12-1.212.1.el6_10.3.x86_64.rpm \
/rpms/glibc-devel-2.12-1.212.1.el6_10.3.x86_64.rpm \
/rpms/glibc-headers-2.12-1.212.1.el6_10.3.x86_64.rpm \
/rpms/glibc-static-2.12-1.212.1.el6_10.3.x86_64.rpm \
/rpms/glibc-utils-2.12-1.212.1.el6_10.3.x86_64.rpm \
/rpms/nscd-2.12-1.212.1.el6_10.3.x86_64.rpm \
/rpms/

# Centos 6 is EOL and is no longer available from the usual mirrors, so switch
Expand All @@ -32,10 +31,22 @@ RUN sed -i 's/enabled=1/enabled=0/g' /etc/yum/pluginconf.d/fastestmirror.conf &&
sed -i 's/^mirrorlist/#mirrorlist/g' /etc/yum.repos.d/*.repo && \
sed -i 's;^#baseurl=http://mirror;baseurl=https://vault;g' /etc/yum.repos.d/*.repo

RUN yum install -y glibc.i686 glibc-devel.i686 glibc-static.i686 glibc.x86_64 glibc-devel.x86_64 glibc-static.x86_64 && \
yum -y install /rpms/* && rm -rf /rpms && yum -y clean all && rm -rf /var/cache/yum/* && \
RUN yum update -y && \
yum install -y glibc.i686 glibc-devel.i686 glibc-static.i686 glibc.x86_64 glibc-devel.x86_64 glibc-static.x86_64 && \
yum -y install /rpms/* && rm -rf /rpms && rpm --rebuilddb && yum -y clean all && rm -rf /var/cache/yum/* && \
# if we updated glibc, we need to strip locales again...
localedef --list-archive | grep -v -i ^en_US.utf8 | xargs localedef --delete-from-archive && \
mv -f /usr/lib/locale/locale-archive /usr/lib/locale/locale-archive.tmpl && \
build-locale-archive && \
find /usr/share/locale -mindepth 1 -maxdepth 1 -not \( -name 'en*' -or -name 'locale.alias' \) | xargs rm -rf
find /usr/share/locale -mindepth 1 -maxdepth 1 -not \( -name 'en*' -or -name 'locale.alias' \) | xargs rm -rf && \
rm -rf /root/* /tmp/* /var/log/*

RUN ln -sf cracklib-small.pwi /usr/share/cracklib/pw_dict.pwi && \
ln -sf cracklib-small.pwd /usr/share/cracklib/pw_dict.pwd && \
rm -rf /var/lib/yum/history/* && \
find /usr/lib64/python2.6 \( -type f -a -name '*.pyc' -o -name '*.pyo' \) -delete

FROM scratch
LABEL maintainer="The Manylinux project"
COPY --from=manylinux2010_centos-6-no-vsyscall-build / /
CMD ["/bin/bash"]
5 changes: 3 additions & 2 deletions docker/glibc/Dockerfile-i686
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ RUN sed -i 's/enabled=1/enabled=0/g' /etc/yum/pluginconf.d/fastestmirror.conf &&
sed -i 's;^#baseurl=http://mirror;baseurl=https://vault;g' /etc/yum.repos.d/*.repo

RUN echo "i386" > /etc/yum/vars/basearch
RUN yum -y update && \
yum install -y util-linux-ng
RUN yum install -y util-linux-ng && \
yum -y clean all && \
rm -rf /var/cache/yum/*
COPY ./build_scripts /build_scripts
RUN linux32 bash /build_scripts/rebuild-glibc-without-vsyscall.sh
7 changes: 6 additions & 1 deletion docker/glibc/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,14 @@ case "${1-}" in
;;
all)
docker_build \
-t quay.io/pypa/manylinux2010_centos-6-no-vsyscall:latest \
--target manylinux2010_centos-6-no-vsyscall-build \
-t quay.io/pypa/manylinux2010_centos-6-no-vsyscall-build:latest \
--cache-from quay.io/pypa/manylinux2010_centos-6-with-vsyscall32:latest \
--cache-from quay.io/pypa/manylinux2010_centos-6-with-vsyscall64:latest \
--cache-from quay.io/pypa/manylinux2010_centos-6-no-vsyscall-build:latest
docker_build \
-t quay.io/pypa/manylinux2010_centos-6-no-vsyscall:latest \
--cache-from quay.io/pypa/manylinux2010_centos-6-no-vsyscall-build:latest \
--cache-from quay.io/pypa/manylinux2010_centos-6-no-vsyscall:latest
;;
*)
Expand Down
12 changes: 10 additions & 2 deletions docker/glibc/build_scripts/CentOS-source.repo
Original file line number Diff line number Diff line change
@@ -1,7 +1,15 @@
[base-source]
name=CentOS-6.10 - Base SRPMS
[os-source]
name=CentOS-6.10 - OS SRPMS
baseurl=https://vault.centos.org/6.10/os/Source/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
priority=1
enabled=1

[updates-source]
name=CentOS-6.10 - Updates SRPMS
baseurl=https://vault.centos.org/6.10/updates/Source/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
priority=1
enabled=1
28 changes: 14 additions & 14 deletions docker/glibc/build_scripts/glibc.spec.patch
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,25 @@ index 9bd07c9..c389711 100644
@@ -1,6 +1,6 @@
%define glibcsrcdir glibc-2.12-2-gc4ccff1
%define glibcversion 2.12
-%define glibcrelease 1.212%{?dist}
+%define glibcrelease 1.212.1%{?dist}
-%define glibcrelease 1.212%{?dist}.3
+%define glibcrelease 1.212.1%{?dist}.3
%define run_glibc_tests 1
%define auxarches athlon sparcv9v sparc64v alphaev6
%define xenarches i686 athlon
@@ -279,6 +279,7 @@
Patch247: glibc-rh1452717-4.patch
Patch248: glibc-rh1504810-1.patch
Patch249: glibc-rh1504810-2.patch
+Patch250: remove-vsyscall.patch
@@ -283,6 +283,7 @@
Patch251: glibc-rh1555930-1.patch
Patch252: glibc-rh1555930-2.patch
Patch253: glibc-rh1577437.patch
+Patch254: remove-vsyscall.patch

Buildroot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
Obsoletes: glibc-profile < 2.4
@@ -731,6 +732,7 @@
%patch247 -p1
%patch248 -p1
%patch249 -p1
+%patch250 -E -p3
@@ -739,6 +739,7 @@
%patch251 -p1
%patch252 -p1
%patch253 -p1
+%patch254 -E -p3

# A lot of programs still misuse memcpy when they have to use
# memmove. The memcpy implementation below is not tolerant at

10 changes: 8 additions & 2 deletions docker/glibc/build_scripts/rebuild-glibc-without-vsyscall.sh
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ mkdir $DOWNLOADED_SRPMS
adduser mockbuild
# yumdownloader assumes the current working directory
(cd $DOWNLOADED_SRPMS && yumdownloader --source glibc)
rpm -ivh $DOWNLOADED_SRPMS/glibc-$ORIGINAL_GLIBC_VERSION.el6.src.rpm
rpm -ivh $DOWNLOADED_SRPMS/glibc-$ORIGINAL_GLIBC_VERSION.el6_10.3.src.rpm
# Prepare the source by applying Red Hat and CentOS patches
rpmbuild -bp $SRPM_TOPDIR/SPECS/glibc.spec

Expand All @@ -40,6 +40,9 @@ cp $MY_DIR/remove-vsyscall.patch $SRPM_TOPDIR/SOURCES
# Patch the RPM spec file so that it uses the vsyscall removal patch
(cd $SRPM_TOPDIR/SPECS && patch -p2 < $MY_DIR/glibc.spec.patch)

# Use dist .el6_10 to mimic replaced glibc
sed -i 's/.el6$/.el6_10/g' /etc/rpm/macros.dist

# Build the RPMS
# In case of error, you can `docker commit` to inspect the build.log
rpmbuild -ba $SRPM_TOPDIR/SPECS/glibc.spec >> /var/log/build.log
Expand All @@ -49,4 +52,7 @@ mv $SRPM_TOPDIR/RPMS/* /rpms/
# Show us what happened last before cleaning up the log
echo ~~~~~~~~~~~~~~~~~~~~~ final lines of the build log ~~~~~~~~~~~~~~~~~~~~~ >/dev/null
tail -n30 /var/log/build.log
rm /var/log/build.log

# We don't need to keep anything execept /rpms/*
rm -rf /root /tmp /var /sbin
rm -rf /usr /lib*
9 changes: 8 additions & 1 deletion travis-ci/tags.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,25 @@ case "${1-}" in
64)
tags=( quay.io/pypa/manylinux2010_centos-6-with-vsyscall64:latest )
;;
combined_build)
tags=(
quay.io/pypa/manylinux2010_centos-6-no-vsyscall-build:latest
quay.io/pypa/manylinux2010_centos-6-no-vsyscall:latest
)
;;
combined)
tags=( quay.io/pypa/manylinux2010_centos-6-no-vsyscall:latest )
;;
all)
tags=(
quay.io/pypa/manylinux2010_centos-6-with-vsyscall32:latest
quay.io/pypa/manylinux2010_centos-6-with-vsyscall64:latest
quay.io/pypa/manylinux2010_centos-6-no-vsyscall-build:latest
quay.io/pypa/manylinux2010_centos-6-no-vsyscall:latest
)
;;
*)
echo "Usage: $0 {32|64|combined|all}" >&2
echo "Usage: $0 {32|64|combined_build|combined|all}" >&2
exit 1
;;
esac