Remove Rust runtime dependency of maturin

[Flamefire]

#22257 introduced a CI check for Rust usage and enforces use of the CargoPythonBundle easyblock as cargo, if used, might otherwise use the cargo Config in the users home which makes the installation in-deterministic and might also fail

Pulling out Rust from the runtime dependencies of maturin would make it explicit when Rust is indeed used and not just required for building maturin.

We might need to fix/retest some easyconfigs but a current develop grep shows only 24 which might need changing. It is possible that those do not require Rust to build and hence don't need to be updated.
Also as maturin is a build dependency the impact is IMO low

@Micket any thoughts?

[pavelToman]

Could you help me clarify my understanding?

First, when using an easyconfig with the PythonBundle easyblock, I typically specify Python dependencies within the exts_list, and these dependencies are installed via pip install .. For Rust-based Python packages, the compilation involves maturin, which compiles Rust code into Python modules. My understanding is that once built, the resulting Python package does not require maturin or Rust at runtime. However, during the build phase, both maturin and Rust must be there as build dependencies. Could you confirm whether this understanding is correct? If yes, it makes sense, that maturin has a Rust as dependency and it is used in 'PythonBundle' as build dependency.

Currently, due to the CI checks, I can not include maturin as a build dependency within easyconfigs based on PythonBundle—it triggers the error: use CargoPythonBundle easyblock instead. Consequently, I must create a separate easyconfig specifically for extensions that depend on maturin. This new easyconfig cannot use PythonPackage or PythonBundle, but it must be CargoPythonBundle or CargoPythonPackage and have crates dependencies. However, my understanding is that maturin itself handles crate management internally, meaning explicitly installing crates might be redundant? Could you confirm whether the explicit installation of crates in the easyconfig is intentional or preferable compared to allowing maturin to manage crates directly?

I am dealing with this in my PR now:

• {bio}[foss/2023a] SCENICplus v1.0a2-20250113, pybigtools v0.1.2, scatac_fragment_tools v0.1.0 #22800

[Micket]

I cannot include maturin as a build dependency within easyconfigs based on PythonBundle—it triggers the error: use CargoPythonBundle easyblock instead

Well, yes that seems correct to me. It should be a CargoPythonBundle yes.

Consequently, I must create a separate easyconfig specifically for extensions that depend on maturin

No that does not follow at all from the previous statement. One can make a CargoPythonBundle.

In summary; if we do anything with Rust, we should specify all the cache all the crates yes (instead of it downloading them from the internet, and change over time, breaking the builds). That is the goal, making them reproducible.

I don't know what you mean by maturin handling crates, or what that would even mean. I don't see it doing anything that affects caching or mirroring crates.io.

whether the explicit installation of crates

there is no such concept as "installation of crates" (not these ones at least; these sources are not installed; they are used for the build of the software in question)

@Micket any thoughts?

I mean, sure that sounds fine I guess. We could just also check for maturin/setuptools-rust in the CI as well if we don't want to rock the boat to much?

[Flamefire]

I can try to clarify the points:

• maturin is a build tool written in Rust and hence requires Rust to be built
• A Python package may use any language and hence could need either C++ or Rust.
• A Python package may use maturin to build, independently of the language used. This is done by a setting in pyproject.toml which is used by pip install .
• maturin is (usually) not required at runtime
• AFAIK a package built with Rust does not require the Rust module at runtime

So Rust might not be required when building a Python package not using Rust but still using maturin. Hence my suggestion to decouple it

must be CargoPythonBundle or CargoPythonPackage and have crates dependencies.

You don't have to specify any crates, do you?

maturin itself handles crate management internally, meaning explicitly installing crates might be redundant

The crates are just sources. Internally we download a tarball for each specified crate and place it at a known location. Then we instruct Cargo/Rust to use those sources instead of downloading them during the build for security and reliability. maturing is not involved in all that. In the and maturin just calls cargo build which will then search or download the crate sources.
Hence we are not installing crates manually. We just provide sources for the later build to be used when necessary.
So yes this is intentional for all software using Rust

Summary: If a software uses Rust/Cargo to build we definitely want a Cargo* easyblock for 2 reasons:

1. Allow to provide crates to avoid having them downloaded during the build
2. Set environment variables to ensure Cargo is not picking up user settings (e.g. ~/.cargo) influencing the build in unexpected ways

@Micket

We could just also check for maturin/setuptools-rust in the CI as well if we don't want to rock the boat to much?

See also the above: I think it is possible to use maturin without Rust and only add Rust as a builddependency if the package actually uses Rust. The current CI check detects this explicit use of Rust and suggests the Cargo easyblocks.

But if we require Cargo easyblocks for any package using maturin to build we are too restrictive as it would be not necessary for some.

[boegel]

So Rust might not be required when building a Python package not using Rust but still using maturin. Hence my suggestion to decouple it

I wasn't actually aware of that, I guess we assumed that using maturin as tool to get Python packages installed always involved Rust. If not, then decoupling makes sense, since that's essentially a bug.
@Flamefire Do you happen to have a particular example of a non-Rust Python package that uses maturin?

I think part of the confusion here is also that we don't have documentation on how to deal with easyconfigs that use CargoPythonBundle, like how to generate the crates list, what to do if there's no Cargo.lock, dealing with corner cases, etc.

Let's use this as an opportunity to finally write up some docs...

[Flamefire]

@Flamefire Do you happen to have a particular example of a non-Rust Python package that uses maturin?

I don't because our maturin EasyConfigs always pull in Rust so we don't notice if any doesn't need Rust. So it is possible, that all of our packages do use Rust. I just judged by the description of maturin:

Build and publish crates with pyo3, cffi and uniffi bindings as well as rust binaries as python packages

And I remembered proposing to test removing Rust from maturin for one EC build but it turned out that Rust is in fact required for one of the dependencies.

So it is indeed possible that all our maturin-using EasyConfigs use Rust for at least one package

[PetrKralCZ]

@Flamefire @boegel
Even if software which requires maturin also always required Rust, I would still advocate for decoupling these two.
The reason is it would give us more flexibility with Rust versions.
For example when some new version of such software required Rust >= 1.87.0 in the future, we would also need to create another maturin for that version of Rust.
I believe that is the reason why we have so many easyconfigs of maturin already.