Add cross-builds for ARM

[vadimcn]

This enables cross-building for aarch64-unknown-linux-gnu and arm-unknown-linux-gnueabihf.
Because Python runs various scripts in the course of install image generation, a host arch Python distribution needs to be built first.

Areas that still need work:

• I've changed the base image to ubuntu:xenial, because it has cross-toolchains readily available. Obviously, a Python distro built that way may not run on older Linuxes, e.g. CentOS 6 and the like.
• Not all Python tests pass. But then they didn't seem to pass before my changes either. Am I running tests the wrong way? -
  (PYTHONHOME=$(pwd)/install LD_LIBRARY_PATH=$(pwd)/install/lib $(pwd)/install/bin/python3 build/run_tests.py from the root of extracted zstd package).
• I could not get the pip module to work on arm - python3 -m ensurepip fails because of missing _sysconfigdata__linux_aarch64-linux-gnu module. Not sure why it isn't getting built...

[vadimcn]

The last one should be fixed now.

[indygreg]

Thank you for following up on our email thread and contributing this!

As we discussed on that thread, my request was for you to contribute something quick and dirty that produced working results and I would figure out a path to incorporating it into this project so it could coexist with existing code. From a quick glance at this PR, I believe you've done that!

I can't land this PR as is because the changes to the base Docker image would break the existing builds. But, there is plenty in here to build a working integration from. I'll definitely look at incorporating this the next time I set down to hack on python-build-standalone (which could be several weeks away).

In the interim, if you want to incrementally add parts of your patch as separate PRs that can be landed without breaking existing builds, go for it! I think a good place to start would be to teach the build-*.sh scripts to pass --host/--target to configure. I'd prefer to do that unconditionally so we're explicit about our build/host targets in the configure invocation. But if passing these confuses some configure scripts, we can make it optional. Other build-*.sh changes (such as the normalization to $ROOT) would also make good incremental contributions.

The goal here would be to incrementally decrease the size of the diff to support ARM so that the scope of work required to implement that in a robust manner is as small as possible.

If you want to submit these incremental PRs, go for it. Otherwise I'll rework this PR the next time I sit down to hack on this project.

Thank you again for the contribution! Can't wait to incorporate this and get cross-builds working!

[vadimcn]

You're welcome!

I'm in no particular hurry to land this. I've got what I needed - a portable Python distro for ARM. Feel free to massage this PR into the shape you like when you have time.

I can't land this PR as is because the changes to the base Docker image would break the existing builds.

Curiously enough, building on xenial did not change the compatibility list that pyoxidizer analyze reports for libpython. The minimal glibc version is still 2.17.

[vadimcn]

Forgot another todo item:

• I had to exclude _decimal from static modules, because it requires different compilation flags depending on the platform (use assembly on x86, pure C otherwise). This needs a better solution, unless you want to use C code on all platforms.

[indygreg]

Curiously enough, building on xenial did not change the compatibility list that pyoxidizer analyze reports for libpython. The minimal glibc version is still 2.17.

Oh, nice!

FWIW the reason I use Debian as the base image is that http://snapshot.debian.org/ provides point-in-time snapshots of the Apt package repository. By pointing the environment at a specific snapshot, we ensure that apt-get update and apt-get install are deterministic and reproducible over time. This gives us a nice property of having reproducible builds/Docker images. I don't believe Ubuntu offers a similar snapshotted Apt server. If it did, I might consider using Ubuntu for the base image!

[vadimcn]

Debian Stretch might work as well; I went for Ubuntu Xenial just because it seemed to have a slightly lower version of glibc (2.23 vs 2.24).
In the end, it seems to be not so important, as the lowest compatible version is determined by which glibc exports the program actually uses. Maybe you could save yourself the trouble of having to compile cross-toolchains...

[indygreg]

I started incorporating parts of this patch into the main branch, which is why GitHub is now complaining about merge conflicts.

Essentially, I've got a parallel branch of yours going in my local development environment and I'm slowly porting pieces over as I grok them. I'm not sure when I'll finish this. Don't feel obliged to rebase and work through the merge conflicts if you don't want to!

[vadimcn]

@indygreg I've tried rebasing this branch on top of your main, but there are lots of conflicts. :(

You've added arm arch to the docker image, so I was wondering whether you are planning to enable Linux cross-builds anytime soon?

[indygreg]

I was experimenting with Linux ARM cross-compiles a few days ago. It is something I'd very much like to implement. I may look at it again in the days ahead.

[indygreg]

I just pushed some cross-compiling support. See #60 (comment) and https://python-build-standalone.readthedocs.io/en/latest/status.html#target-notes for more context.

The code in this PR was very helpful in implementing cross-compiling support. First for Apple targets then Linux. Thank you for helping to steer me in the right direction!

[vadimcn]

Thanks!
I've a question though: why are you using clang to cross-build on Linux platforms?

[indygreg]

I've a question though: why are you using clang to cross-build on Linux platforms?

Clang is likely required for cross builds on Apple targets. GCC might be possible. But I have no intention of trying to get that working... because Apple.

Only the i686 cross builds are currently using Clang, everything else is using the gcc crosstools packages in Debian Stretch.

However, I would like to get everything using Clang if possible that way we have a single compiler for all UNIX-like platforms.

Is there a good reason to use GCC over Clang? If there is, we should consider using a more modern GCC. Debian Stretch is version 6.3.

Sticking with 6.3 is probably fine. I just like keeping things modern. Everything else is using a modern toolchain.

[vadimcn]

Oh, sorry, I actually meant to ask "why aren't you using clang to cross build...", but accidentally the most important word in the sentence :)