feat(android): add Rust cross-compilation setup for Android environment #2650

ririv · 2025-11-05T02:59:24Z

Like platform windows

cibuildwheel/cibuildwheel/platforms/windows.py

Lines 206 to 232 in c53e541

    
           def setup_rust_cross_compile( 
        
               tmp: Path,  # noqa: ARG001 
        
               python_configuration: PythonConfiguration, 
        
               python_libs_base: Path,  # noqa: ARG001 
        
               env: MutableMapping[str, str], 
        
           ) -> None: 
        
               # Assume that MSVC will be used, because we already know that we are 
        
               # cross-compiling. MinGW users can set CARGO_BUILD_TARGET themselves 
        
               # and we will respect the existing value. 
        
               cargo_target = { 
        
                   "64": "x86_64-pc-windows-msvc", 
        
                   "32": "i686-pc-windows-msvc", 
        
                   "ARM64": "aarch64-pc-windows-msvc", 
        
               }.get(python_configuration.arch) 
        
               # CARGO_BUILD_TARGET is the variable used by Cargo and setuptools_rust 
        
               if env.get("CARGO_BUILD_TARGET"): 
        
                   if env["CARGO_BUILD_TARGET"] != cargo_target: 
        
                       log.notice("Not overriding CARGO_BUILD_TARGET as it has already been set") 
        
                   # No message if it was set to what we were planning to set it to 
        
               elif cargo_target: 
        
                   log.notice(f"Setting CARGO_BUILD_TARGET={cargo_target} for cross-compilation") 
        
                   env["CARGO_BUILD_TARGET"] = cargo_target 
        
               else: 
        
                   log.warning( 
        
                       f"Unable to configure Rust cross-compilation for architecture {python_configuration.arch}" 
        
                   )

Add Rust cross-compilation setup for Android environment, it can fix the cross-compilation problem for the rust project built with setuptools-rust

ririv · 2025-11-05T03:03:34Z

For Rust projects built with Maturin, it currently does not work. Maturin has some cross-compilation issues on Android that still need to be fixed.

For details, please see PyO3/maturin#2810

joerick · 2025-11-07T12:06:52Z

This looks nice and neat! Is there an example of it working in a real project that you've been testing it with?

ririv · 2025-11-07T12:52:33Z

I want to test， but I did not know how to run this without the github action，if you know how to do, you can tell me，and I will test it.

This commit is a modification I made based on the actual missing environment configuration built on Android using cbuildwheel.

Of course, I manually configured it externally in a non-isolated environment, and after the configuration, I built the wheel correctly and it can run on Android

joerick · 2025-11-07T14:47:02Z

You can test a project using something like this in a github workflow-

      - name: Build wheels
        uses: ririv/cibuildwheel@main

…_DIR to link against libpython3.x.so explicitly

ririv · 2025-11-10T07:08:38Z

Thanks, I have tested it, and made some new commits. Now, it works.

The CARGO_TARGET_<triple>_LINKER must be set. Otherwise, it will use the host's linker cc, which will result in an error.
The PYO3_CROSS_LIB_DIR must be set to link against libpython3.x.so explicitly.
See: https://peps.python.org/pep-0738/#linkage and https://pyo3.rs/v0.27.1/building-and-distribution.html#cross-compiling

The PYO3_CROSS_LIB_DIR environment variable should be set here rather than requiring users to specify it manually, for the following reasons:

PYO3 has become the de facto standard for Python extension modules built with Rust. Both setuptools-rust and maturin are tools developed by the PYO3 team specifically to serve PYO3.
The value set for PYO3_CROSS_LIB_DIR requires the directory path of target_python. This path is already configured within cbuildwheel's android.py but cannot be directly accessed externally. External configuration would necessitate manually constructing the path.
PYO3's documentation for Android cross-compilation is insufficiently detailed. If users attempt to set this variable manually, they are likely to become confused and unsure how to proceed.

ririv · 2025-11-10T07:11:35Z

https://github.com/ririv/android-wheels/blob/main/.github/workflows/cbuildweel-repair-test.yml
This is the workflow file. It performed some conversions.
For the maturin project, it was converted to a setuptools-rust project. This is because maturin has some bugs in Android cross-compilation. I've submitted a PR to fix it, but the maturin maintainer hasn't responded yet.

Here are some logs:

pydantic-core | maturin convert to setuptools-rust
https://github.com/ririv/android-wheels/actions/runs/19223626338/job/54946144941
tiktoken | based on setuptools-rust
https://github.com/ririv/android-wheels/actions/runs/19223649349/job/54946212372

joerick · 2025-11-28T15:52:21Z

Looks good to me! @mhsmith what do you think?

mhsmith · 2025-12-02T13:16:40Z

Sorry for the delay, I'll look at this as soon as I can.

cibuildwheel/platforms/android.py

mhsmith · 2025-12-21T01:24:14Z

cibuildwheel/platforms/android.py

+        log.notice("Not overriding PYO3_CROSS_LIB_DIR as it has already been set")
+    else:
+        env["PYO3_CROSS_LIB_DIR"] = str(python_dir / "prefix" / "lib")
+        log.notice("Setting PYO3_CROSS_LIB_DIR for PyO3 cross-compilation")


This logging is unnecessary.

mhsmith · 2025-12-21T01:29:25Z

cibuildwheel/platforms/android.py

+    # CARGO_BUILD_TARGET is the variable used by Cargo and setuptools_rust
+    if env.get("CARGO_BUILD_TARGET"):
+        if env["CARGO_BUILD_TARGET"] != cargo_target:
+            log.notice("Not overriding CARGO_BUILD_TARGET as it has already been set")


I can't think of a plausible reason for the user to override any of these variables, so let's set them all unconditionally. This will make both functions so much shorter, that they might as well be combined into a single setup_rust_cross_compile function.

Or even just setup_rust, since everything in this file is cross compiling by definition.

mhsmith · 2025-12-21T01:29:40Z

cibuildwheel/platforms/android.py

+    python_dir: Path,
+    env: MutableMapping[str, str],
+) -> None:
+    # All Python extension modules must therefore be explicitly linked against libpython3.x.so when building for Android.


Suggested change

# All Python extension modules must therefore be explicitly linked against libpython3.x.so when building for Android.

# All Python extension modules must be explicitly linked against libpython3.x.so when building for Android.

mhsmith · 2025-12-21T01:46:17Z

cibuildwheel/platforms/android.py

+    env: MutableMapping[str, str],
+) -> None:
+    cargo_target = android_triplet(python_configuration.identifier)
+    call("rustup", "target", "add", cargo_target)


This will fail if rustup isn't installed, so that must be handled.

Even if it is installed, it would be a waste of time to download Android Rust files for packages that don't actually use Rust. How long will this take? If it's more more than a couple of seconds, I don't think that's acceptable.

Is the rustup target add command still necessary even if CARGO_BUILD_TARGET is set? If so, Is there some way we could trigger the command only for packages that use Rust?

For example, if calling rustup is part of the build process (I don't know whether it is), then we could add a rustup script to the environment's bin directory, and have it run rustup target add before forwarding the command to the real rustup. I was thinking of doing something similar to install the Android Fortran compiler on demand for those packages that attempt to use it.

Copilot adds:

the Windows implementation (lines 206-232 in windows.py) does not call rustup at all - it only sets environment variables. This inconsistency should be evaluated to determine if rustup installation is actually necessary

Copilot

Pull request overview

This pull request adds Rust cross-compilation setup for the Android platform, mirroring similar functionality that exists for Windows. The implementation configures environment variables necessary for building Rust-based Python packages (using setuptools-rust and PyO3) when cross-compiling for Android targets.

Key Changes:

Added setup_rust_cross_compile() function to configure Cargo build target and linker environment variables
Added setup_PYO3_cross_compile() function to set PyO3 cross-compilation library directory
Integrated both functions into the Android environment setup process

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-21T04:58:07Z

cibuildwheel/platforms/android.py

+        )
+        env["CARGO_BUILD_TARGET"] = cargo_target
+        # CC has already been set by calling android.py (it calls android-env.sh)
+        env[f"{cargo_target_linker_env_name}"] = env["CC"]


Unnecessary f-string formatting. The variable cargo_target_linker_env_name is already a string, so wrapping it in an f-string is redundant. Change env[f"{cargo_target_linker_env_name}"] to env[cargo_target_linker_env_name] for cleaner code.

Suggested change

env[f"{cargo_target_linker_env_name}"] = env["CC"]

env[cargo_target_linker_env_name] = env["CC"]

Copilot · 2025-12-21T04:58:07Z

cibuildwheel/platforms/android.py

+    else:
+        log.warning(f"Unable to configure Rust cross-compilation for architecture {cargo_target}")


The else branch on line 428 is unreachable. The android_triplet function always returns a non-empty string from the ANDROID_TRIPLET dictionary (see lines 39-42, 54-55). If the identifier is invalid, parse_identifier would raise a ValueError before reaching this point, so cargo_target will always be truthy. This dead code should be removed, or if there's a possibility of empty/None values, the logic needs to be adjusted.

Copilot · 2025-12-21T04:58:07Z

cibuildwheel/platforms/android.py

+def setup_rust_cross_compile(
+    python_configuration: PythonConfiguration,
+    env: MutableMapping[str, str],
+) -> None:
+    cargo_target = android_triplet(python_configuration.identifier)
+    call("rustup", "target", "add", cargo_target)
+
+    # CARGO_BUILD_TARGET is the variable used by Cargo and setuptools_rust
+    if env.get("CARGO_BUILD_TARGET"):
+        if env["CARGO_BUILD_TARGET"] != cargo_target:
+            log.notice("Not overriding CARGO_BUILD_TARGET as it has already been set")
+        # No message if it was set to what we were planning to set it to
+    elif cargo_target:
+        cargo_target_linker_env_name = (
+            f"CARGO_TARGET_{cargo_target.upper().replace('-', '_')}_LINKER"
+        )
+        log.notice(
+            f"Setting CARGO_BUILD_TARGET={cargo_target} and {cargo_target_linker_env_name} for cross-compilation"
+        )
+        env["CARGO_BUILD_TARGET"] = cargo_target
+        # CC has already been set by calling android.py (it calls android-env.sh)
+        env[f"{cargo_target_linker_env_name}"] = env["CC"]
+    else:
+        log.warning(f"Unable to configure Rust cross-compilation for architecture {cargo_target}")
+
+
+def setup_PYO3_cross_compile(
+    python_dir: Path,
+    env: MutableMapping[str, str],
+) -> None:
+    # All Python extension modules must therefore be explicitly linked against libpython3.x.so when building for Android.
+    # See: https://peps.python.org/pep-0738/#linkage
+    # For projects using PyO3, this requires setting PYO3_CROSS_LIB_DIR to the directory containing libpython3.x.so.
+    # See: https://pyo3.rs/v0.27.1/building-and-distribution.html#cross-compiling
+    if env.get("PYO3_CROSS_LIB_DIR"):
+        log.notice("Not overriding PYO3_CROSS_LIB_DIR as it has already been set")
+    else:
+        env["PYO3_CROSS_LIB_DIR"] = str(python_dir / "prefix" / "lib")
+        log.notice("Setting PYO3_CROSS_LIB_DIR for PyO3 cross-compilation")


The new Rust cross-compilation setup functions lack test coverage. Given that test/test_android.py has comprehensive test coverage for other Android functionality, tests should be added to verify that setup_rust_cross_compile and setup_PYO3_cross_compile correctly set the expected environment variables (CARGO_BUILD_TARGET, CARGO_TARGET_*_LINKER, and PYO3_CROSS_LIB_DIR) under various conditions.

Rather than checking for the environment variables directly, it would be better to actually build and run a simple Rust project, just as we currently do for C++ in test_android.py::test_libcxx.

Copilot · 2025-12-21T04:58:07Z

cibuildwheel/platforms/android.py

    for key in ["CFLAGS", "CXXFLAGS"]:
        android_env[key] += " " + opt

+    # Cargo target linker need to be specified after CC is set


Typo in comment: "need" should be "needs". The comment should read "Cargo target linker needs to be specified after CC is set".

Suggested change

# Cargo target linker need to be specified after CC is set

# Cargo target linker needs to be specified after CC is set

cibuildwheel/platforms/android.py

… cargo and rustup

…s-rust

for more information, see https://pre-commit.ci

ririv · 2025-12-22T09:52:58Z

Thanks for the suggestions. I've updated the PR to address the feedback:

Lazy Rust Setup: I implemented the shim/wrapper approach for cargo and rustup as suggested. This ensures zero overhead for non-Rust projects, as the Android target is only installed when a Rust command is actually triggered. I used absolute paths in the shims to avoid recursion.
Platform Alignment: Cleaned up the redundant log.notice calls and removed the defensive CC check to follow the android-env.sh contract.
New Integration Test: Added test_setuptools_rust to test_android.py. It builds a minimal Rust extension and runs it on the emulator to verify the whole flow.

ririv · 2025-12-22T10:03:08Z

Since PyO3/maturin#2825 has been merged into main, we can now use cibuildwheel to test Rust projects built with maturin. However, as a new version hasn't been released yet, we have to build maturin from source for now.

This is the GItHub Action script:

https://github.com/ririv/android-wheels/blob/main/.github/workflows/test-cbuildwheel-repair-maturin-source.yml

This is the logs:

zstandard | non-Rust project
https://github.com/ririv/android-wheels/actions/runs/20427696018
tiktoken | based on setuptools-rust
https://github.com/ririv/android-wheels/actions/runs/20427678027

Sorry, there are still some issues with the maturin integration: it's building for Linux instead of Android. This might be a problem with the build module; I will investigate and fix it.

pydantic_core | based on maturin
https://github.com/ririv/android-wheels/actions/runs/20427674411

However, we can still use a conversion tool to transform maturin projects into setuptools-rust projects and successfully build Android wheels.

pydantic_core | maturin convert to setuptools-rust
https://github.com/ririv/android-wheels/actions/runs/20429779899

ririv · 2025-12-26T07:44:48Z

I have investigated the issue with the wheel generated by maturin. It actually produced the correct wheel, but the tag was incorrect.
This occurred because maturin does not yet support Android platform tags, causing it to mistakenly identify Android as Linux and generate an incorrect tag.
I have submitted a PR to maturin to fix this. Details can be found at PyO3/maturin#2900

This issue is unrelated to the current PR. So, no modification is necessary.
Please review, thanks. @mhsmith

mhsmith · 2025-12-26T15:44:34Z

Thanks; I'm on vacation at the moment, but I'll look at this in the new year.

ririv requested a review from mhsmith as a code owner November 5, 2025 02:59

feat(android): add Rust cross-compilation setup for Android environment

32dcc38

ririv force-pushed the main branch from 5f31eac to 32dcc38 Compare November 5, 2025 04:28

ririv force-pushed the main branch 10 times, most recently from 1384753 to 99ee18a Compare November 10, 2025 06:01

feat(android): set the Cargo target linker

22fe90b

ririv force-pushed the main branch from 99ee18a to 778dc5a Compare November 10, 2025 06:52

feat(android): add PYO3 cross-compilation setup to set PYO3_CROSS_LIB…

2a6798e

…_DIR to link against libpython3.x.so explicitly

ririv force-pushed the main branch from 9812d7f to 2a6798e Compare November 10, 2025 06:58

Merge branch 'main' into main

237399b

mhsmith reviewed Dec 21, 2025

View reviewed changes

Merge branch 'main' into main

0c6b76e

Copilot AI review requested due to automatic review settings December 21, 2025 04:55

Copilot started reviewing on behalf of ririv December 21, 2025 04:55 View session

Copilot AI reviewed Dec 21, 2025

View reviewed changes

ririv and others added 4 commits December 22, 2025 15:38

feat(android): streamline Rust cross-compilation setup

1b97365

feat(android): improve rustup target handling and add shim script for…

c92d716

… cargo and rustup

feat(android): add test for Android cross-compilation using setuptool…

9136ed3

…s-rust

[pre-commit.ci] auto fixes from pre-commit.com hooks

f5966be

for more information, see https://pre-commit.ci

henryiii mentioned this pull request Dec 24, 2025

No release of cibuildwheel available that installs 3.14.2 #2700

Open

	def setup_rust_cross_compile(
	tmp: Path, # noqa: ARG001
	python_configuration: PythonConfiguration,
	python_libs_base: Path, # noqa: ARG001
	env: MutableMapping[str, str],
	) -> None:
	# Assume that MSVC will be used, because we already know that we are
	# cross-compiling. MinGW users can set CARGO_BUILD_TARGET themselves
	# and we will respect the existing value.
	cargo_target = {
	"64": "x86_64-pc-windows-msvc",
	"32": "i686-pc-windows-msvc",
	"ARM64": "aarch64-pc-windows-msvc",
	}.get(python_configuration.arch)

	# CARGO_BUILD_TARGET is the variable used by Cargo and setuptools_rust
	if env.get("CARGO_BUILD_TARGET"):
	if env["CARGO_BUILD_TARGET"] != cargo_target:
	log.notice("Not overriding CARGO_BUILD_TARGET as it has already been set")
	# No message if it was set to what we were planning to set it to
	elif cargo_target:
	log.notice(f"Setting CARGO_BUILD_TARGET={cargo_target} for cross-compilation")
	env["CARGO_BUILD_TARGET"] = cargo_target
	else:
	log.warning(
	f"Unable to configure Rust cross-compilation for architecture {python_configuration.arch}"
	)

	# All Python extension modules must therefore be explicitly linked against libpython3.x.so when building for Android.
	# All Python extension modules must be explicitly linked against libpython3.x.so when building for Android.

	env[f"{cargo_target_linker_env_name}"] = env["CC"]
	env[cargo_target_linker_env_name] = env["CC"]

		else:
		log.warning(f"Unable to configure Rust cross-compilation for architecture {cargo_target}")

	# Cargo target linker need to be specified after CC is set
	# Cargo target linker needs to be specified after CC is set

feat(android): add Rust cross-compilation setup for Android environment #2650

Are you sure you want to change the base?

feat(android): add Rust cross-compilation setup for Android environment #2650

Conversation

ririv commented Nov 5, 2025

Uh oh!

ririv commented Nov 5, 2025

Uh oh!

joerick commented Nov 7, 2025

Uh oh!

ririv commented Nov 7, 2025

Uh oh!

joerick commented Nov 7, 2025

Uh oh!

ririv commented Nov 10, 2025

Uh oh!

ririv commented Nov 10, 2025

Uh oh!

joerick commented Nov 28, 2025

Uh oh!

mhsmith commented Dec 2, 2025

Uh oh!

Uh oh!

mhsmith Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

mhsmith Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

mhsmith Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

mhsmith Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

mhsmith Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

mhsmith Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

mhsmith Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ririv commented Dec 22, 2025

Uh oh!

ririv commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ririv commented Dec 26, 2025

Uh oh!

mhsmith commented Dec 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ririv commented Dec 22, 2025 •

edited

Loading