Change MolienSeries to not hang on certain invalid inputs (#5919)

When a virtual character that is not an actual character was fed
to MolienSeries, that could lead to an infinite loop. The reason
in that case was that a quotient of two polynomials was computed
that did not produce a polynomial, but rather a rational function.
Calling `GcdRepresentation(F, f)` on that as a first step then
tried to compute a ring containing both arguments, which
ultimately caused the infinite loop.

We fix this by passing a parent ring to `GcdRepresentation` which
then raises an error in the above situation. The error message
of course is not perfect, but note that other invalid inputs
already can lead to different errors in other places in the
code. Essentially the user is at fault here for passing in garbage.
But we make this patch anyway because getting any error is certainly
more helpful than an infinite loop.

Author: fingolfin

lib/ctblmoli.gi

@@ -208,6 +208,7 @@ InstallGlobalFunction( MolienSeries, function( arg )
           chi,          # character of `tbl', optional third argument
           numers,       # list of numerators   of sum of polynomial quotients
           denoms,       # list of denominators of sum of polynomial quotients
+          R,
           x,            # indeterminate
           tblclasses,   # class lengths of `tbl'
           orders,       # representative orders of `tbl'
@@ -279,7 +280,8 @@ InstallGlobalFunction( MolienSeries, function( arg )
     # `pol' is an additive polynomial.
     numers:= [];
     denoms:= [];
-    x:= Indeterminate( Rationals );
+    R:= UnivariatePolynomialRing(Rationals);
+    x:= R.1;
     pol:= Zero( x );
 
     tblclasses:= SizesConjugacyClasses( tbl );
@@ -361,7 +363,7 @@ InstallGlobalFunction( MolienSeries, function( arg )
           # Split the summand into two summands, with denominators
           # the special factor `f' resp. the remaining factors `F'.
           f:= ( x ^ special[1] - 1 ) ^ special[2];
-          repr:= GcdRepresentation( F, f );
+          repr:= GcdRepresentation( R, F, f );
 
           # Reduce the numerators if possible.
           num:= numer * repr[1];
@@ -413,7 +415,7 @@ InstallGlobalFunction( MolienSeries, function( arg )
             # $\prod_{j>i} f_j$ is stored in `F', and $f_i$ is in `f'.
 
             # at first position $r_i$, at second position $q_i$
-            repr:= GcdRepresentation( F, f );
+            repr:= GcdRepresentation( R, F, f );
 
             # The numerator $p_i$.
             p:= q * repr[1];

tst/testinstall/ctblmoli.tst

@@ -53,5 +53,16 @@ true
 gap> ser = ser2;
 true
 
+#
+# MolienSeries used to hang when given an non-character as input
+# now an error is raised
+#
+gap> G:=Group((1,2,3,4,5,6,7),(5,6,7));;
+gap> MolienSeries(NaturalCharacter(G));
+( 1-z^3+z^6-z^9+z^12-z^15+z^18 ) / ( (1-z^7)*(1-z^5)*(1-z^4)*(1-z^3)^2*(1-z^2)\
+*(1-z) )
+gap> MolienSeries(-NaturalCharacter(G));
+Error, <ns> must be a subset of <R>
+
 #
 gap> STOP_TEST("ctblmoli.tst");