scipy test failures on x86_64 with openblas 0.3.6

[opoplawski]

Since the introduction of openblas 0.3.6 to Fedora rawhide, we're seeing test failures on scipy builds on x86_64. See https://apps.fedoraproject.org/koschei/package/scipy
and in particular: https://kojipkgs.fedoraproject.org/work/tasks/3609/34623609/build.log

Example:

______________________________ TestLSQ.test_lstsq ______________________________
self = <scipy.interpolate.tests.test_bsplines.TestLSQ object at 0x7ff594251cf8>
    def test_lstsq(self):
        # check LSQ construction vs a full matrix version
        x, y, t, k = self.x, self.y, self.t, self.k
    
        c0, AY = make_lsq_full_matrix(x, y, t, k)
>       b = make_lsq_spline(x, y, t, k)
scipy/interpolate/tests/test_bsplines.py:1182: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
scipy/interpolate/_bsplines.py:1017: in make_lsq_spline
    check_finite=check_finite)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
ab = array([[1.00189868e+00, 7.52101832e-01, 5.38951295e-01, 6.89409012e-01,
        6.23893499e-01, 7.79037069e-01, 6.1263...e-04, 5.93585425e-04,
        2.05099058e-03, 2.58394962e-03, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00]])
overwrite_ab = True, lower = True, check_finite = True
    def cholesky_banded(ab, overwrite_ab=False, lower=False, check_finite=True):
        """
        Cholesky decompose a banded Hermitian positive-definite matrix
    
        The matrix a is stored in ab either in lower diagonal or upper
        diagonal ordered form::
    
            ab[u + i - j, j] == a[i,j]        (if upper form; i <= j)
            ab[    i - j, j] == a[i,j]        (if lower form; i >= j)
    
        Example of ab (shape of a is (6,6), u=2)::
    
            upper form:
            *   *   a02 a13 a24 a35
            *   a01 a12 a23 a34 a45
            a00 a11 a22 a33 a44 a55
    
            lower form:
            a00 a11 a22 a33 a44 a55
            a10 a21 a32 a43 a54 *
            a20 a31 a42 a53 *   *
    
        Parameters
        ----------
        ab : (u + 1, M) array_like
            Banded matrix
        overwrite_ab : bool, optional
            Discard data in ab (may enhance performance)
        lower : bool, optional
            Is the matrix in the lower form. (Default is upper form)
        check_finite : bool, optional
            Whether to check that the input matrix contains only finite numbers.
            Disabling may give a performance gain, but may result in problems
            (crashes, non-termination) if the inputs do contain infinities or NaNs.
    
        Returns
        -------
        c : (u + 1, M) ndarray
            Cholesky factorization of a, in the same banded format as ab
    
        See also
        --------
        cho_solve_banded : Solve a linear set equations, given the Cholesky factorization
                    of a banded hermitian.
    
        Examples
        --------
        >>> from scipy.linalg import cholesky_banded
        >>> from numpy import allclose, zeros, diag
        >>> Ab = np.array([[0, 0, 1j, 2, 3j], [0, -1, -2, 3, 4], [9, 8, 7, 6, 9]])
        >>> A = np.diag(Ab[0,2:], k=2) + np.diag(Ab[1,1:], k=1)
        >>> A = A + A.conj().T + np.diag(Ab[2, :])
        >>> c = cholesky_banded(Ab)
        >>> C = np.diag(c[0, 2:], k=2) + np.diag(c[1, 1:], k=1) + np.diag(c[2, :])
        >>> np.allclose(C.conj().T @ C - A, np.zeros((5, 5)))
        True
    
        """
        if check_finite:
            ab = asarray_chkfinite(ab)
        else:
            ab = asarray(ab)
    
        pbtrf, = get_lapack_funcs(('pbtrf',), (ab,))
>       c, info = pbtrf(ab, lower=lower, overwrite_ab=overwrite_ab)
E       SystemError: <fortran object> returned NULL without setting an error
scipy/linalg/decomp_cholesky.py:280: SystemError

Perhaps affected some other packages as well: https://apps.fedoraproject.org/koschei/affected-by/openblas-devel?epoch1=0&version1=0.3.5&release1=5.fc31&epoch2=0&version2=0.3.6&release2=1.fc31&collection=f31

[tylerjereddy]

This is for SciPy 1.2.1 or the just-released 1.3.0, or another version? Speaking from the SciPy side, we often have to be rather selective about the exact OpenBLAS version pairing for everything to work. Sometimes a patch or two is needed after a release. This will hopefully improve over time though!

[martin-frbg]

Is "SystemError: <fortran object> returned NULL without setting an error" indicative of an OpenBLAS error at all, or just a more general build failure ? (If it is, it would be helpful to know the cpu or OpenBLAS cpu target this is observed with )

[opoplawski]

This is with 1.2.1. We are also seeing segfaults with 1.3.0 - http://koji.fedoraproject.org/koji/taskinfo?taskID=34942727 - I haven't yet reported that to scipy.

[martin-frbg]

If I read the overview correctly, the build failure on x86_64 coincided with the OpenBLAS update (but apparently a few other packages were updated in the same timeframe ?), but other platforms started to fail much later, despite also linking against OpenBLAS ? Can you tell what cpu model your build host uses ?

[tylerjereddy]

I don't immediately recognize the error as something I've seen recently. Maybe @rgommers has seen it before.

I'm pretty sure that for SciPy 1.2.1 wheels we're using an OpenBLAS commit on the 0.3.5.dev line prior to the 0.3.6 release point. For SciPy 1.3.0, wheels are using an OpenBLAS commit from the 0.3.7.dev line to deal with recent SkylakeX kernel issues.

I suppose we'll have to do a 1.2.2 eventually with the 0.3.7.dev OpenBLAS as well for the same reason. See here for example: scipy/scipy#10145 Sadly, the Azure CI logs seem to have been purged already (!) so I can't check if the same failure was reported there.

[martin-frbg]

I'm pretty sure that for SciPy 1.2.1 wheels we

So can anybody please clarify what is the relation between "SciPy wheels" and whatever Fedora uses to build their packages. Is there any, or is Fedora compiling some sort of bare bones SciPy source with whatever OpenBLAS they see fit ? The lengthy build log originally linked above (and now gone) looked quite confusing to me, with what looked like a fortran-based BLAS getting built in between and a ton of warnings about out-of-bounds accesses in some fft library test code but no clear reference to a particular version of OpenBLAS.

[rgommers]

Is "SystemError: returned NULL without setting an error" indicative of an OpenBLAS error at all, or just a more general build failure

I have never seen that before. Very likely specific to OpenBLAS, even more so because the report says it's introduced with the upgrade to OpenBLAS 0.3.6

@opoplawski if you're able to bisect that to an OpenBLAS commit, that would probably be very useful.

[martin-frbg]

I gather this is a python error message, apparently either from using a wrong function prototype or a function actually not returning anything. Now in the case of OpenBLAS, I'd think it much more likely to receive an incorrect result rather than no result at all.

[brada4]

The code in question is here:
https://github.com/scipy/scipy/blob/02b0001af4d7125a390556c227578d6cfd06d4e2/scipy/linalg/decomp_cholesky.py#L280
Does it work with older python, it seems mainline python gets about as picky as pypy regarding external call prototypes.

[martin-frbg]

FWIW, the line 280 in decomp_cholesky.pc calls dpbtrf, which would imply either of SYRK,TRSM, GEMV or GEMM. If the build host for this was actually SkylakeX this would indeed be due to the incomplete reversion of the AVX512 DGEMM kernel in 0.3.6 (#1955, and accordingly only affect AVX512 hardware).

[tylerjereddy]

I did manage to dig up recent SciPy master branch test failures (5) under SkylakeX Intel SDE emulation on MacOS, with logs available from here: https://dev.azure.com/tylerjereddy/tyler-scipy-fork/_build/results?buildId=841

1. test_solve_discrete_are in scipy/linalg/tests/test_solvers.py
2. TestQR.test_random_tall_right in scipy/linalg/tests/test_decomp.py
3. TestQR.test_random_tall_left in scipy/linalg/tests/test_decomp.py
4. test_pinv_pinv2_comparison in scipy/linalg/tests/test_basic.py
5. TestLstsq.test_random_exact in scipy/linalg/tests/test_basic.py

Not sure how helpful that is, but we never saw the Linux failures on SKX because our manylinux1 glibc is too old to allow the AVX512 to go through. The PR I linked above can be used as a starting point to check if that's the same problem on Linux as well, although checking the machine hardware should be faster!

There's also the useful OPENBLAS_VERBOSE=2 and i.e., OPENBLAS_CORETYPE=haswell to try to avoid the kernel issue if you're bound to the same hardware & OpenBLAS version for some reason.

[opoplawski]

Unfortunately we're dealing with a couple of different build issues. The openblas 0.3.6 update appears to have broken the x86_64 build with test failures, but a later numpydoc 0.8.0 -> 0.9.1 update broke the documentation build on all arches.

Fedora builds scipy (or any package) against whatever libraries are in that Fedora release. Rawhide is the rolling development for the next Fedora release, so updates of any library can occur at any time. scipy's wheels are built with specific versions of libraries that they specify.

The builder reports:

model name	: Intel Core Processor (Haswell, no TSX, IBRS)
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti ibrs ibpb fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat

Unfortunately OPENBLAS_VERBOSE=2 does not appear to produce any usable output that I can discern. Scratch build here - https://koji.fedoraproject.org/koji/taskinfo?taskID=34966483

I don't think I'm going to have any time to be able to do a bisect of the openblas commits.

[brada4]

If you look google for SystemError: <fortran object> returned NULL you get a lot of hits against python 3.7 also in non-numeric libs all around foreign library imports. I am just wondering if problem is same in python 3.6 or not.

It says about returning NULL - the only return is INFO that is zero (might be interpreted as NULL in most cases) in case of success...

[martin-frbg]

Not sure what to make of this - Haswell is probably the most used platform, so any errors e.g. in the recent rewrite of assembly constraints following #2009 (that I just reviewed again) should have had more than enough time to show up in less complex environments. And I still do not understand how a numerical error in OpenBLAS could lead to that particular error message.

[martin-frbg]

Not reproducible by building scipy-1.2.1 with 0.3.6 on openSUSE 15 and running the provided testsuite via python runtests.py. (I do note that this build was done with gcc 7.3.1 while rawhide seems to be using some flavor of 9.1. I am not aware of any miscompilations with 9.x since #2009 was fixed - though you had seen some transient failures there back in february )