Merge cypari, cypari2; replace cysignal's optional PARI dependency by a general hook

[mkoeppe]

• Needs pyproject.toml to declare build-time dependency on cysignals cypari2#93 (Needs pyproject.toml to declare build-time dependency on cysignals)
• merge cypari and cypari2 cypari2#19 (comment) (merge cypari, cypari2)
• Replace compile-time dependency on pari by run-time dispatch cysignals#125 (replace cysignal's optional PARI dependency by a general hook)

Tickets:

• cysignals: Replace compile-time dependency on pari by run-time dispatch #32431 cysignals: Replace compile-time dependency on pari by run-time dispatch

CC: @videlec @NathanDunfield @dimpase @kliem

Component: packages: standard

Issue created by migration from https://trac.sagemath.org/ticket/32360

[mkoeppe]

Description changed:

--- 
+++ 
@@ -2,3 +2,5 @@
 - https://github.com/sagemath/cypari2/issues/19#issuecomment-803142279 (merge cypari, cypari2)
 - https://github.com/sagemath/cysignals/issues/125 (replace cysignal's optional PARI dependency by a general hook)
 
+Tickets:
+- #32431 cysignals: Replace compile-time dependency on pari by run-time dispatch

[mkoeppe]

comment:7

@videlec, what's the status here?

[videlec]

comment:8

replace cysignal's optional PARI dependency by a general hook: is in good shape thanks to J. Kliem (see cysignals #166 and cypari2 #130).

About cypari2 converging to cypari the main step is to have wheels bundling gmp+pari that works on windows. I am neither able to develop nor test that.

[dimpase]

where are we here now? This was supposed to move the building issue for cysignals forward.

[dimpase]

About cypari2 converging to cypari the main step is to have wheels bundling gmp+pari that works on windows.

Native Windows, or "Sage way", i.e. Cygwin? Given that Sage all but stopped Cygwin, then it's (semi)native, via msys64, according to https://github.com/3-manifolds/CyPari.

@culler @NathanDunfield - how about sorting this merge out? Is cypari (not cypari2) even used in Sage?

[culler]

Cypari runs on native Windows as an independent python module and provides binary wheels for Windows. Of course its existence precedes that of Cypari2, as the name suggests, but it was originally based on Sage's pari module.

The Windows build of CyPari uses mingw to compile Pari and gmp. And the Windows version of cysignals works very hard to enable the use of sig_on and sig_off on Windows. On Windows, a signal handler runs in a separate thread with its own stack. The Sage implementation of sig_on and sig_off using setjmp assumes that the signal handler uses the same stack as the process receiving the signal. This is the case on most UNIX systems, although it is not required by posix. It turns out that there is one signal (SIGFPE) which runs on the main stack in Windows. The version of cysignals shipped with cypari exploits this fact to make sig_on and sig_off work on Windows.

One can use Cypari in Sage or in plain python. We are forced to use Cypari rather than Cypari2 for some of our computations due to the egregious memory leaks in the current version of Cypari2, which were causing jobs to get terminated for excessive memory use on Nathan's cluster. These leaks were created by a change in strategy with later versions of Cypari2. The new strategy attempts to keep GENs on the pari stack as long as possible. The old approach was to move each GEN to the Pari heap as soon as its wrapping python object is instantiated. The python object would then free the GEN in its dealloc method. The implementation of the new strategy is incomplete and incorrect; it includes reference counting errors which create python objects that never get freed, as well as GENs on the pari heap which never get freed. While cypari used to sync with cypari2 on a regular basis, incorporating improvements added by the Sage developers, that is no longer the case due to the memory leaks. In cypari we have reverted to the old strategy and eliminated the leaks.

I would say that dealing with the memory leaks is now the biggest obstacle to moving forward with any aspect of CyPari2.

[NathanDunfield]

how about sorting this merge out? Is cypari (not cypari2) even used in Sage?

There is no progress on merging them. Indeed, as Marc says they've drifted farther apart due to the memory leaks in cypari2.

Usingcypari and cypari2 simultaneously sometimes results in segfaults, so using cypari inside Sage isn't a good idea. (IIRC, I've only observed this on Linux, not macOS.)

[dimpase]

hmm, sounds bad. By the way, why is cypari still on an old Pari version (2.11) ?

[NathanDunfield]

By the way, why is cypari still on an old Pari version (2.11) ?

Looks like we need to update the docs, the current version uses Pari 2.15.2.

[culler]

When the memory issues are resolved the next obstruction to merging involves code organization. In a nutshell:

• CyPari must support Windows, since it is shipped with SnapPy which must support Windows.
• Windows does not allow one dll to refer to symbols in another dll, which implies that in order to work on Windows the CyPari extension module must consist of a single shared library, That shared library must be statically linked with libpari and also with cysignals, due to the low-level code in cysignals.

[dimpase]

• Windows does not allow one dll to refer to symbols in another dll

this doesn't sound right to me. The difference w.r.t. Unix is that for a dll one need to explicitly export symbols to be used elsewhere.

It can be that Pari (libpari) lacks such an export file on Windows - this is something that would qualify as a Pari bug.

[dimpase]

Looks like we need to update the docs, the current version uses Pari 2.15.2.

there is also a dead link to bitbucket on its PyPI

[culler]

• Windows does not allow one dll to refer to symbols in another dll

this doesn't sound right to me. The difference w.r.t. Unix is that for a dll one need to explicitly export symbols to be used elsewhere.

It can be that Pari (libpari) lacks such an export file on Windows - this is something that would qualify as a Pari bug.

There is another side to this, though. While cypari2 has many different extension modules, as far as I know there is no context in which it would be useful to import any of them other than the main cypari2 module. If I am right, there is no benefit to breaking it up in this way. Cysignals may possibly be a different story. But now that it has been isolated from Pari it may be fine as a standalone.

[dimpase]

please check out https://pari.math.u-bordeaux.fr/cgi-bin/bugreport.cgi?bug=2488

it seems that Pari does not recommend using Pari on Windows natively, and to go with WSL instead.

[culler]

Maybe SnapPy will have to drop support for Windows one day. But until that happens Cypari will need to continue to support Windows. A bigger concern for me at the moment is that gmp does not build on the macOS Sonoma beta release. - Marc

…

On Fri, Sep 1, 2023, 8:55 AM Dima Pasechnik ***@***.***> wrote: please check out https://pari.math.u-bordeaux.fr/cgi-bin/bugreport.cgi?bug=2488 it seems that Pari does not recommend using Pari on Windows natively, and go with WSL instead. — Reply to this email directly, view it on GitHub <#32360 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ6CP7ITW7PYMSBJO3UMSDXYHZNJANCNFSM6AAAAAA3ZVC64A> . You are receiving this because you were mentioned.Message ID: ***@***.***>