[WIP] Manylinux2010 #279

mayeut · 2019-03-19T21:16:48Z

A new Manylinux 2010 PR based on #182 with master merged c.f. #179 (comment)

Remaining items:

Thorough review (already began in Manylinux2010 #182)
Merge [WIP] Reduce image size #210 in master (or at least the commit allowing to get rid of yum workaround) then rebase. This requires some review. This leads to yum workaround avoided, proper isolation of curl & openssl and a reduced image (faster docker pull). This could be done in a second step but this is the occasion to get this merged in.
drop 32bit image, update to devtoolset-8

…ster - The dockerfiles build successfully. - Compared to current master, there's a new intermediate image for glibc. As the x86_64 build depends on it, this must have a predefined name. In the PR the "naming prefix" is 'markrwilliams/manylinux2', therefore it's assumed for all images. - There's a problem with libcurl which has been (temporarily) resolved with script which redirects yum. See docker/build_scripts/build.sh for details. The following script is proven to work: pushd docker/glibc/ docker build -t markrwilliams/manylinux2:centos-6.9-no-vsyscall popd docker/build_scripts/prefetch.sh curl openssl docker build -t markrwilliams/manylinux2:x86_64 \ -f docker/Dockerfile-x86_64 docker/ docker build -t markrwilliams/manylinux2:i686 \ -f docker/Dockerfile-i686 docker/

- docker/build_scripts/build.sh: - removed libncurses.devel; not part of PEP 571 - removed gpg as dependency, because yum already requires it - removed some /dev/null redirections, because they hinder debugging - docker/build_scripts/manylinux1-check.py - renamed to manylinux-check.py - changed names/comments to new version - the check now targets glibc 2.12 instead of 2.5 - .travis.yml - only changed the image name - TODO: x86_64 still needs a script for the no-vsyscall image dependency - docker/deploy.sh - changed image name - note: .rst files have not been touched

- The freshly released version of pip breaks build_utils.sh which tries to get the latest and greatest. Added a condition to instead use the latest version below v10.

- Incorporated the lastest changes from the official repository. - That cleanup removed two packages from 'yum erase ...' which looks like an oversight. Readded. - Removed the hack which fixed the broken build when the new pip was released. This has been fixed upstream, see here: pypa/get-pip#19

- Add platform-dependent clause - Assign the same prefix to the the no-vsyscall image (quay.io/pypa/...) - The dependent docker/Dockerfile-x86_64 needs this prefix in FROM ...

- Increase in `travis_wait` time - Shunting build output off into a log file which can be inspected on failure.

is merge is necessary,

- Similar to gpg the libffi-devel package can't be removed after the build, but it looks like this is because it was improperly packaged. - Moved libffi to the "Development tools and libraries" section.

Like with CentOS 5 there's a CentOS 6 Docker image which is more up to date (tag :6) than the latest release (tag :6.9). See: pypa#206

Previously, vault.centos.org was missing some repository information, which prevented moving to 6.10. This is now available on CentOS 6.10 and it includes a glibc v1.212 (up from 1.209). All changes made are only to bump this version number.

docker/build_scripts/build.sh

docker/glibc/build_scripts/glibc.spec.patch

docker/build_scripts/build.sh

illume · 2019-03-24T15:24:57Z

docker/glibc/build_scripts/remove-vsyscall.patch

@@ -0,0 +1,401 @@
+diff --git a/BUILD/glibc-2.12-2-gc4ccff1/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S b/BUILD/glibc-2.12-2-gc4ccff1/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S


Who wrote this patch?

I found this commit... is it from here? markrwilliams@e9493d5#diff-3eda4130bdba562657f3ec7c1b3f5720

I'm not sure who wrote it, if they need credit, or if it needs review.

I guess @markrwilliams wrote it ? in #152

I did, largely backporting vsyscall-less code from newer glibcs.

docker/glibc/README.rst

manylinux2 -> manylinux2010

.travis.yml

Prevent stdout timeout. The global travis timeout is the only one to apply now.

glibc patch is unlikely to change very often, use travis-ci cache mechanism in order not to rebuild it every time. also, use build stages so that when it does change, glibc build is run as a pre-step, it will probably help not reaching the time limit, or at least, glibc build will be cached for job restart.

takluyver · 2019-04-06T13:50:01Z

I had to restart a timed-out job before, so I guess it was right on the cusp of the 50 minute timeout. If splitting it into two stages gives us 50 minutes for each stage, that should make it much more reliable.

takluyver · 2019-04-09T10:42:53Z

@mayeut what level of confidence would you have in images built by this branch as it stands? There don't seem to be a lot of people stepping up to do review. Would you be happy if it were merged, so the images are posted to quay.io and people start using them, and we try to fix problems as they come up? Or do you think we should keep it on hold for further review?

JonasVautherin · 2019-04-09T13:12:25Z

I cannot review this, but I am building stuff with manylinux1 and I could definitely try with manylinux2010 and report issues.

* Minimal usable merge of markrwilliams:manylinux2 into current pypa:master - The dockerfiles build successfully. - Compared to current master, there's a new intermediate image for glibc. As the x86_64 build depends on it, this must have a predefined name. In the PR the "naming prefix" is 'markrwilliams/manylinux2', therefore it's assumed for all images. - There's a problem with libcurl which has been (temporarily) resolved with script which redirects yum. See docker/build_scripts/build.sh for details. The following script is proven to work: pushd docker/glibc/ docker build -t markrwilliams/manylinux2:centos-6.9-no-vsyscall popd docker/build_scripts/prefetch.sh curl openssl docker build -t markrwilliams/manylinux2:x86_64 \ -f docker/Dockerfile-x86_64 docker/ docker build -t markrwilliams/manylinux2:i686 \ -f docker/Dockerfile-i686 docker/ * Clean-up and adjustments for manylinux2010 - docker/build_scripts/build.sh: - removed libncurses.devel; not part of PEP 571 - removed gpg as dependency, because yum already requires it - removed some /dev/null redirections, because they hinder debugging - docker/build_scripts/manylinux1-check.py - renamed to manylinux-check.py - changed names/comments to new version - the check now targets glibc 2.12 instead of 2.5 - .travis.yml - only changed the image name - TODO: x86_64 still needs a script for the no-vsyscall image dependency - docker/deploy.sh - changed image name - note: .rst files have not been touched * Fix for new release of pip (v10.0.0) breaking the build script - The freshly released version of pip breaks build_utils.sh which tries to get the latest and greatest. Added a condition to instead use the latest version below v10. * Merge upstream changes (4a20e4b); remove get-pip hack - Incorporated the lastest changes from the official repository. - That cleanup removed two packages from 'yum erase ...' which looks like an oversight. Readded. - Removed the hack which fixed the broken build when the new pip was released. This has been fixed upstream, see here: pypa/get-pip#19 * Re-add the two /dev/null redirections in build.sh * Merge upstream changes (6f3bc0b) * Merge upstream changes (3dd4551) * Merge upstream changes (4132165) * Adapt build.sh for the x86_64 two-stage build - Add platform-dependent clause - Assign the same prefix to the the no-vsyscall image (quay.io/pypa/...) - The dependent docker/Dockerfile-x86_64 needs this prefix in FROM ... * Getting Travis to play along - Increase in `travis_wait` time - Shunting build output off into a log file which can be inspected on failure. * Drop 32-bit support and upgrade to devtoolset-7 * Update README.rst manylinux2 -> manylinux2010 * Add comment regarding MANYLINUX2010_DEPS * Add comment for Software collection & EPEL usage * Move expat-devel to PYTHON_COMPILE_DEPS * Use local gpgkey for RPM-GPG-KEY-CentOS-6 * Update to devtoolset-8 * Replace travis wait by custom command Prevent stdout timeout. The global travis timeout is the only one to apply now. * Use travis-ci build stages and cache glibc build glibc patch is unlikely to change very often, use travis-ci cache mechanism in order not to rebuild it every time. also, use build stages so that when it does change, glibc build is run as a pre-step, it will probably help not reaching the time limit, or at least, glibc build will be cached for job restart.

dolang and others added 28 commits April 13, 2018 06:16

Fix for new release of pip (v10.0.0) breaking the build script

27e7332

- The freshly released version of pip breaks build_utils.sh which tries to get the latest and greatest. Added a condition to instead use the latest version below v10.

Re-add the two /dev/null redirections in build.sh

54b3c84

Merge upstream changes (6f3bc0b)

dcd4acf

Merge upstream changes (3dd4551)

8f868fb

Merge upstream changes (4132165)

50f4eac

Merge upstream changes (0566338)

0a6c6ee

Merge upstream changes (9f3b52a)

55b3514

Adapt build.sh for the x86_64 two-stage build

87886d4

- Add platform-dependent clause - Assign the same prefix to the the no-vsyscall image (quay.io/pypa/...) - The dependent docker/Dockerfile-x86_64 needs this prefix in FROM ...

Getting Travis to play along

2664c85

- Increase in `travis_wait` time - Shunting build output off into a log file which can be inspected on failure.

Merge upstream changes (e893a63)

67ef9a0

Merge upstream changes (d837450)

05098c8

Merge upstream changes (1971a37)

2ec5c38

is merge is necessary,

Merge upstream changes (96b47a2); move libff-devel

e075849

- Similar to gpg the libffi-devel package can't be removed after the build, but it looks like this is because it was improperly packaged. - Moved libffi to the "Development tools and libraries" section.

Merge upstream changes (46241e9)

24082fe

Merge a0d46fa; mirror base image change :6.9 -> :6

0a31334

Like with CentOS 5 there's a CentOS 6 Docker image which is more up to date (tag :6) than the latest release (tag :6.9). See: pypa#206

Merge upstream changes (9e82cb0)

e05c365

Merge 46e2d66; update to CentOS 6.10

c39783f

Previously, vault.centos.org was missing some repository information, which prevented moving to 6.10. This is now available on CentOS 6.10 and it includes a glibc v1.212 (up from 1.209). All changes made are only to bump this version number.

Merge branch 'master' into manylinux2010

ca7a542

Merge branch 'master' into manylinux2010

2011765

Merge branch 'master' into manylinux2010

ca12203

Merge branch 'master' into manylinux2010

931aa02

Merge branch 'master' into manylinux2010

f2f22d3

Merge branch 'master' into manylinux2010

5b12bfe

Merge branch 'master' into manylinux2010

f5792ce

Drop 32-bit support and upgrade to devtoolset-7

daa1a5f

takluyver mentioned this pull request Mar 23, 2019

Tracking issue for manylinux2010 rollout #179

Closed

14 tasks

illume reviewed Mar 24, 2019

View reviewed changes

docker/build_scripts/build.sh Show resolved Hide resolved

illume reviewed Mar 24, 2019

View reviewed changes

docker/glibc/build_scripts/glibc.spec.patch Show resolved Hide resolved

illume reviewed Mar 24, 2019

View reviewed changes

docker/build_scripts/build.sh Outdated Show resolved Hide resolved

illume reviewed Mar 24, 2019

View reviewed changes

takluyver reviewed Mar 26, 2019

View reviewed changes

docker/glibc/README.rst Outdated Show resolved Hide resolved

JonasVautherin mentioned this pull request Mar 28, 2019

Update README.rst mayeut/manylinux#1

Merged

JonasVautherin and others added 7 commits March 31, 2019 14:32

Update README.rst

4b59d3a

manylinux2 -> manylinux2010

Add comment regarding MANYLINUX2010_DEPS

db3b9d3

Add comment for Software collection & EPEL usage

3d899ab

Move expat-devel to PYTHON_COMPILE_DEPS

20db20d

Use local gpgkey for RPM-GPG-KEY-CentOS-6

356df2d

Merge branch 'master' into manylinux2010

f6fc85a

Update to devtoolset-8

39892a0

dralley mentioned this pull request Apr 3, 2019

wheels on PyPI rpm-software-management/createrepo_c#145

Closed

Merge branch 'master' into manylinux2010

bbfe5a4

takluyver reviewed Apr 5, 2019

View reviewed changes

.travis.yml Show resolved Hide resolved

mayeut added 3 commits April 5, 2019 22:41

Replace travis wait by custom command

6765234

Prevent stdout timeout. The global travis timeout is the only one to apply now.

Merge branch 'master' into manylinux2010

bc2ee54

trishankatdatadog merged commit 749d394 into pypa:master Apr 10, 2019

mayeut deleted the manylinux2010 branch April 10, 2019 21:00

matham mentioned this pull request Apr 11, 2019

[packaging][highlight]Add manylinux2010 support kivy/kivy#6248

Merged

This was referenced Apr 12, 2019

[WIP] manylinux2010 with devtoolset 7 #252

Closed

Manylinux2010 #182

Closed

Manylinux2 #152

Closed

Emulate the vsyscall page in userspace in the x86_64 Docker image #158

Closed

Mizux mentioned this pull request May 2, 2019

WHEEL file size mismatches for ortools-7.0.6546-cp36-cp36m-manylinux1_x86_64.whl google/or-tools#1218

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Manylinux2010 #279

[WIP] Manylinux2010 #279

mayeut commented Mar 19, 2019 •

edited

Loading

illume Mar 24, 2019

mayeut Mar 31, 2019

markrwilliams Mar 31, 2019

takluyver commented Apr 6, 2019

takluyver commented Apr 9, 2019

JonasVautherin commented Apr 9, 2019

		@@ -0,0 +1,401 @@
		diff --git a/BUILD/glibc-2.12-2-gc4ccff1/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S b/BUILD/glibc-2.12-2-gc4ccff1/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S

[WIP] Manylinux2010 #279

[WIP] Manylinux2010 #279

Conversation

mayeut commented Mar 19, 2019 • edited Loading

illume Mar 24, 2019

Choose a reason for hiding this comment

mayeut Mar 31, 2019

Choose a reason for hiding this comment

markrwilliams Mar 31, 2019

Choose a reason for hiding this comment

takluyver commented Apr 6, 2019

takluyver commented Apr 9, 2019

JonasVautherin commented Apr 9, 2019

mayeut commented Mar 19, 2019 •

edited

Loading