Building wheels with GMP et al

[oscarbenjamin]

If this is a bad place to open this discussion then let me know and I'll take it elsewhere.

I've been working on python-flint which is another project that packages similar libraries to gmpy. Along with GMP and MPFR, python-flint also includes Flint and Arb so it is similar in some ways to gmpy but with a different overall scope. The main thing I've been working on is producing wheels (flintlib/python-flint#1) and I'm pretty close to having that working with some of that work inspired by the way that gmpy does things.

I wonder if there is scope for collaborating and learning from each other about exactly what is a good way to build something like this because it seems like a very much uphill struggle with current Python packaging infrastructure (although things are getting better over time).

In particular I have questions right now about how exactly gmpy makes wheels for Apple arm64. I see that recent commits include a patch to GMP for this platform. Is that necessary? Does something not work without it?

Has the gmpy project considered moving to meson or cmake etc because I see some suggestion that other scientific Python projects are moving that way. Could there be any scope to work together on moving towards new build systems?

[casevh]

Hi Oscar,

This is a fine forum for these discussions.

I skimmed your work that you've done on python-flint. I think you've encountered most of the same issues with complex builds.

Regarding Apple arm64 builds - yes, the patch is required. Apple reserved a register for their use. GMP 6.2.1 uses that register. The patch modifies GMP 6.2.1 to save/restore that register. (https://gmplib.org/list-archives/gmp-discuss/2022-July/006821.html) Without the patch, it will crash frequently. (See #350)

Some comments on Windows

Compiling gmpy for Windows has always been a challenge. -cmingw32 worked well for 32-bit versions of Python but required extensive hacking to use mingw64 compilers on 64-bit Windows. (I originally used MPIR and MSVC but MSVC didn't support --enable-fat. I didn't want to compile for the lowest common CPU type nor try to release CPU specific versions.)

I compile the Windows binaries locally. I just extracted a copy of the cygwincompiler.py file from numpy and replace the file that comes with Python with a slightly modified copy. I don't compile anything else on Windows so it works for me. It would be great if the numpy version of -cmingw32 could be added to setuptools as -cmingw64. That would eliminate any risk of breakage for existing code that patches -cmingw32 on the fly.

I know it is possible to compile Cython extensions on Windows using MSVC and link to the GMP/MPFR/MPC dlls. (See #320) At some point, I'd like to compile GMP/MPFR/MPC with a version of mingw64 that links to Microsoft's ucrt and then compile gmpy with MSVC. The ucrt dlls could be release as "wingmpy2-lib" package and become a basis for other projects.

Regarding meson or cmake, I'm open to anything that it easier and more predictable, especially over the long run.

Regards,
Case

[oscarbenjamin]

Without the patch, it will crash frequently. (See #350)

Thanks. That's what I was wondering. It seems this topic comes up monthly on the GMP mailing list. I wonder when there might be a release with this patch.

I don't compile anything else on Windows so it works for me. It would be great if the numpy version of -cmingw32 could be added to setuptools as -cmingw64.

I guess that's implicitly what we're depending on by using numpy.distutils in python-flint. I suspect that adding this to setuptools is not going to happen but I might be wrong. AFAICT Core Python and PyPA etc basically expect that MSVC will be used on Windows. That doesn't work for many key scientific packages (e.g. scipy) but it seems that their fix is to move away from setuptools to things like meson. I might be wrong but it seems like setuptools doesn't have resource to support non-MSVC compilers and the packages that do want other compilers are moving away from setuptools.
https://labs.quansight.org/blog/2021/07/moving-scipy-to-meson
pypa/setuptools#2372

The ucrt dlls could be release as "wingmpy2-lib" package and become a basis for other projects.

It would be a lot better if we could depend on something like that. How exactly would it work for building a downstream project that depended on these DLLs at the C level? I'd still need all headers etc to build python-flint but can that all be packaged up so that python-flint CI just installs the wingmpy2-lib wheel and then uses plain setuptools/MSVC to build a Cython extension based on that?

That would be good although I'd still need to then build Flint and Arb so I'm not sure how much it would simplify anything for making python-flint wheels... I guess not duplicating the DLLs on disk or in memory is also a good thing and not having to carry patches for GMP etc. So yes if it just works(!) then it would simplify things.

Then again it would be better if it was just easier to build stuff on Windows using either mingw64 or MSVC and have everything else taken care of. Ideally I'd just say in the cibuildwheel config what sort of compiler I'd want to use and everything else would be figured out by the build tools without me needing to manually code everything. It should be possible for both C projects and C-based Python extension modules to specify their build in a sufficiently platform independent way that which compiler is used is just a configuration variable or something but it definitely doesn't feel that simple right now...

Since you mention ucrt the python-flint wheels currently link against msvcrt.dll. They seem to work fine but I assume that's potentially problematic and it would be better to link against ucrt. I'm sure I saw somewhere that msys2's ucrt support was somehow experimental or not production ready or something but I can't find that now although there's some information about ucrt here:
https://www.msys2.org/docs/environments/
If I understand correctly it should be straight-forward to switch to ucrt by just changing this line in CI:
https://github.com/fredrik-johansson/python-flint/blob/e08d3dfc236f6b38fcc11bcde5c621993761a806/.github/workflows/buildwheel.yml#L24
Maybe I should test that...

More generally though it seems like you really need to be an expert in packaging to make any of this work and it's really not my domain of expertise. I know of this project to help with these general problems:
https://discuss.python.org/t/modern-way-to-build-packages-with-c-cython-extensions/15050/28
https://iscinumpy.dev/post/scikit-build-proposal/
Perhaps @henryiii do you have any advice about what might be a good path forwards for projects like these?

[oscarbenjamin]

I'm sure I saw somewhere that msys2's ucrt support was somehow experimental

It's here that I see the note about it being experimental but I'm not sure what exactly it means:
https://github.com/marketplace/actions/setup-msys2#build-matrix

[rgommers]

I guess that's implicitly what we're depending on by using numpy.distutils in python-flint. I suspect that adding this to setuptools is not going to happen but I might be wrong

AFAIK the setuptools maintainers are still open to any contributions that folks need, including any features that numpy.distutils has. That said, I don't think anyone is working on that and moving to another build system is probably easier than starting to add things you need to setuptools.

[oscarbenjamin]

Thanks Ralf.

I don't particularly want to become a maintainer of these parts of setuptools so if no one else is interested working on it then I'd rather switch towards tools that other projects are going to use and that people with better expertise than me are going to maintain.

[oscarbenjamin]

I have opened a more recent python-flint issue to discuss the Python 3.12 problem:
flintlib/python-flint#52

@casevh What are the current plans for gmpy2 in Python 3.12?

[casevh]

What are the current plans for gmpy2 in Python 3.12?

My current approach relies on hacking the local copies of setuptools by overwriting cygwincompiler.py. It is not a realistic option for anyone else to use.

The following would be nice but it's probably too late to be viable - add a new compiler type called mingw64.

• This would add the numpy.distutils support for the modern MinGW64 compilers.
• The changes to distutils are simple. There are no changes to the existing mingw32 compiler environment. Any of the existing hacks to use mingw32 will continue to work.
• Existing projects will need to opt-in by specifying the new compiler.

@rgommers Would such a change be acceptable for consideration into setuptools? (I won't be offended if the answer is no.)

@oscarbenjamin I will add comments to flintlib/python-flint#52 regarding GMP builds.

[rgommers]

@rgommers Would such a change be acceptable for consideration into setuptools? (I won't be offended if the answer is no.)

I'm not a setuptools maintainer, but my understand is that yes, they're open to such things. And I agree that adding support for a new compiler is fairly easy to do and non-intrusive. So it's probably a single PR with quite low associated maintenance costs after that. You'd have to make that PR though. setuptools releases quite frequently, so this can be available pretty quickly.

[casevh]

Current update:

• I was able to compile GMP/MPFR/MPC and link them to ucrt. Compiling gmpy2 was successful but the test suite crashed.
• I am repeat the build process with but using mscrt.

More details to follow.

[oscarbenjamin]

• I was able to compile GMP/MPFR/MPC and link them to ucrt. Compiling gmpy2 was successful but the test suite crashed.

This sounds like my epxperience in:
flintlib/python-flint#41

[casevh]

I've added the GMP, MPFR, and MPC DLLs and library files to the gmpy2 repository. Look in gmpy\mingw64\winlibs. Basic tests pass and I need to do some cleanup yet.

But no hacks or modifications are needed to compile with MSVC and link to those DLLs.

[oscarbenjamin]

Is this line correct:

gmpy/mingw64/msys2_build.txt

Line 105 in 8c58be9

py -3.12 setup.py build_ext -cmingw32 -Imingw64\gmpy2 -Lmingw64\gmpy2 -f bdist_wheel

It still has -cmingw32...