[Python] ARM Linux Wheels (Raspberry PI)

[ShadowJonathan]

I'm unsure if GHA CI has Arm linux runners, but something that came up during the workshop was installing soar to raspberry PIs, which imo is a good use-case.

Adding support should be "as simple as" just adding a CI runner for linux ARM, though I remember the variety of ARM versions that raspberry pis are spread over, so looking at whether to support only ARM64 (aarch64) or older versions/variants (v6, v7, v8, hardware float, etc.) is something that needs to be done.

[ShadowJonathan]

Currently still researching, but I've found something interesting; https://www.piwheels.org/

This is a project that builds Python wheels for raspberry Pi, unfortunately, it only works for when/if there are sdists available on pypi, something we've kinda opted out of.

It is also an external project.

---

While GitHub only offers ARM64 (aarch64 (ARMV8-A)) Linux runners, this did give me an idea: to have self-hosted raspberry Pi CI runners (1, 2, 3, 3B(?)), maybe at MIT, that would handle these builds.

I'm not sure how tenable that is, but I'm throwing the idea out there. :)

[ShadowJonathan]

Alright, after a bit of research I've figured out the following:
(will update as I refine/figure out additional things)

There are a few "major" "streams" of ARM processors;

• armv5 (32-bit), aka arm32v5
• armv6 (32-bit), aka arm32v6/armel
• armv7 (32-bit), aka arm32v7/armhf
• armv8 (64-bit & 32-bit), aka arm64v8/aarch64/arm64
  • Has armv7 32-bit compatibility

(Some data from here)

(There's some finagling between them, such as l (little-endian) implicitly required and strewn between them, but lets imagine a more perfect picture for now)

armv8 (or more specifically, armv8a), aka aarch64 (aka ARM64) is the most desktop-ready variant of the bunch, and the one that Github CI can run on.

v6, v7, and v8 are not inter-compatible.

---

According to this stackexchange answer, the major raspberry pi versions have these architectures:

• Raspberry Pi 1, Model A, B, B+, the Compute Module, and the Raspberry Pi Zero: armv6
• Raspberry Pi 2 Model B, rev 1.1: armv7a
• Raspberry Pi 2, rev 1.2, and Raspberry Pi 3: armv8a
• Raspberry Pi 3 A+ and B+: armv8a

However, its worth noting that 2, 3A, and 3B were running in 32-bit armv7 mode for a long time, until raspberry pi announced a 64-bit version of their OS, which breaks compatibility with armv7, instead using v8 proper, or "aarch64/arm64"

---

Also, after researching this, and reading the piwheels FAQ, they say the following;

I'm a package maintainer. Can I upload an Arm wheel to PyPI?

PyPI now allows uploading Armv6 and Armv7 wheels, but we recommend that you don't upload Arm wheels to PyPI for incompatibility reasons and other complications.

Wheels built on a Raspberry Pi 3, although tagged armv7l are not truly Armv7, and may be incompatible with other Arm platforms. [...]

Why are the wheel files are tagged with Armv6 and Armv7?

While the hardware in Raspberry Pi 1 and Zero is Armv6, Pi 2 is Armv7, Pi 3, 4, 400 and Zero 2 are Armv8, the (32-bit) operating system images provided on raspberrypi.com are made to support all three architectures (and are compatible with all Raspberry Pi products to date) by operating in Armv6 userland.

However, wheels built on a Raspberry Pi 2/3/4 running the 32-bit OS are tagged armv7l. Since wheels built on a Raspberry Pi 3/4 will work on a Pi 5, 4, 3, 2, 1, Zero or Zero 2, we simply provide Pi 3/4-built wheels renamed armv6l, with a few exceptions (some packages, like opencv and tensorflow, are built with optimisations available on Pi 3/4).

In other words: Shenanigans needed to be performed to get wheels to build properly on raspberry pi, and provide "armv6l" and "armv7l" wheels.

---

In conclusion: Since Raspberry Pi is already moving away from v6 and v7, I propose that we focus on getting CI to build a aarch64 linux wheel, and then test if installation of that works properly on a raspberry pi 3. Not that we bother with a v7 wheel (due to the above-mentioned compatibility issues with other machines running v7), unless it turns out that its required for better compatibility.

I think I still have one sitting in my closet, either that or I'm using it for a project I half-forgot about, so I'll dig that up, PR a CI addition, and report if installation/testing works.

However, if there's a possibility to add a self-hosted raspberry pi runner to the test matrix (hopefully with a spare one that's laying in a closet at MIT somewhere, maybe), then that'd be swell, so we know that we're not breaking the wheel in some way.

[ShadowJonathan]

There's also this i just found;

• a specific comment saying that allowing arm7l onto pypi was "a mistake"
• a comment declaring that linux_arm7l&linux_armv6l be considered raspbian compatibility tags, not generic ARM
• a comment talking about more piwheels shenanigans stemming from raspberry pi's armv6 userland
• this issue which talks about how piwheels' wheels are actually armv6, even though they're tagged armv7
• "linux_armv?l tags aren't Pi specific, but if someone built a wheel on a Pi, it may be incompatible with other Arm boards."

[ShadowJonathan]

I got curious, and so did a test run;

• Running it via QEMU on Linux resulted in a build time blowup to 40 minutes, as the host machine needed to emulate aarch64 (which is slow).
  • This is only the build part, not even the test part, which'd add more to this total, probably making a single CI run take over an hour to complete entirely.
• Trying to run it on MacOS ARM64, via docker revealed to me that docker isn't installed on macos-14, and that it is not supported, as it'd required nested virtualisation, which - while the M2/M3 supports it - apple hasn't given software support for it yet.

This'd unironically would make having a self-hosted raspberry pi runner be the best CI option for linux aarch64.

Edit: or an Apple ARM64 Mac (such as a Mac Mini) running Asahi Linux, but that'd be a tad more expensive :)