NumPy compiler warnings on Windows

[michaelosthege]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

Even fresh, minimal environments on Windows have a broken compiler toolchain, rendering them completely unusable.

This command creates a small environment:

conda create -n aesara-test -c conda-forge python=3.9 aesara -y

Upon import aesara there are lots of errors and warnings:

>>> import aesara
WARN: Could not locate executable g77
WARN: Could not locate executable f77
WARN: Could not locate executable ifort
WARN: Could not locate executable ifl
WARN: Could not locate executable f90
WARN: Could not locate executable DF
WARN: Could not locate executable efl
Using built-in specs.
COLLECT_GCC=C:\Users\osthege\AppData\Local\Continuum\miniconda3\envs\CARenv2\Library\mingw-w64\bin\gfortran.exe
COLLECT_LTO_WRAPPER=C:/Users/osthege/AppData/Local/Continuum/miniconda3/envs/CARenv2/Library/mingw-w64/bin/../lib/gcc/x86_64-w64-mingw32/5.3.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../gcc-5.3.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/include --libexecdir=/mingw64/lib --with-gxx-include-dir=/mingw64/include/c++/5.3.0 --enable-bootstrap --with-arch=x86-64 --with-tune=generic --enable-languages=c,lto,c++,objc,obj-c++,fortran,ada --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --enable-version-specific-runtime-libs --disable-isl-version-check --enable-lto --enable-libgomp --disable-multilib --enable-checking=release --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --with-isl=/mingw64 --with-pkgversion='Rev5, Built by MSYS2 project' --with-bugurl=https://sourceforge.net/projects/msys2 --with-gnu-as --with-gnu-ld
Thread model: posix
gcc version 5.3.0 (Rev5, Built by MSYS2 project)
WARNING (aesara.tensor.blas): Using NumPy C-API based implementation for BLAS functions.

At first I thought this was just one environment, but the problem appears in all new environments I create. And I'm running out of non-broken old environments to fall back to 😬

With these broken environments, doing a posterior predictive sampling with a GP is about 100x slower than in non-broken envs.

Installed packages

# packages in environment at C:\Users\osthege\AppData\Local\Continuum\miniconda3\envs\aesara-test:
#
# Name                    Version                   Build  Channel
aesara                    2.5.3            py39h7fcc1c2_0    conda-forge
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2021.10.8            h5b45459_0    conda-forge
cons                      0.4.5              pyhd8ed1ab_0    conda-forge
etuples                   0.3.5              pyhd8ed1ab_0    conda-forge
filelock                  3.6.0              pyhd8ed1ab_0    conda-forge
intel-openmp              2022.0.0          h57928b3_3663    conda-forge
libblas                   3.9.0              13_win64_mkl    conda-forge
libcblas                  3.9.0              13_win64_mkl    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
liblapack                 3.9.0              13_win64_mkl    conda-forge
libpython                 2.1              py39hcbf5309_1    conda-forge
libzlib                   1.2.11            h8ffe710_1014    conda-forge
logical-unification       0.4.5              pyhd8ed1ab_0    conda-forge
m2w64-binutils            2.25.1                        5    conda-forge
m2w64-bzip2               1.0.6                         6    conda-forge
m2w64-crt-git             5.0.0.4636.2595836               2    conda-forge
m2w64-gcc                 5.3.0                         6    conda-forge
m2w64-gcc-ada             5.3.0                         6    conda-forge
m2w64-gcc-fortran         5.3.0                         6    conda-forge
m2w64-gcc-libgfortran     5.3.0                         6    conda-forge
m2w64-gcc-libs            5.3.0                         7    conda-forge
m2w64-gcc-libs-core       5.3.0                         7    conda-forge
m2w64-gcc-objc            5.3.0                         6    conda-forge
m2w64-gmp                 6.1.0                         2    conda-forge
m2w64-headers-git         5.0.0.4636.c0ad18a               2    conda-forge
m2w64-isl                 0.16.1                        2    conda-forge
m2w64-libiconv            1.14                          6    conda-forge
m2w64-libmangle-git       5.0.0.4509.2e5a9a2               2    conda-forge
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    conda-forge
m2w64-make                4.1.2351.a80a8b8               2    conda-forge
m2w64-mpc                 1.0.3                         3    conda-forge
m2w64-mpfr                3.1.4                         4    conda-forge
m2w64-pkg-config          0.29.1                        2    conda-forge
m2w64-toolchain           5.3.0                         7    conda-forge
m2w64-toolchain_win-64    2.4.0                         0    conda-forge
m2w64-tools-git           5.0.0.4592.90b8472               2    conda-forge
m2w64-windows-default-manifest 6.4                           3    conda-forge
m2w64-winpthreads-git     5.0.0.4634.697f757               2    conda-forge
m2w64-zlib                1.2.8                        10    conda-forge
minikanren                1.0.3              pyhd8ed1ab_0    conda-forge
mkl                       2022.0.0           h0e2418a_796    conda-forge
msys2-conda-epoch         20160418                      1    conda-forge
multipledispatch          0.6.0                      py_0    conda-forge
numpy                     1.22.3           py39h6331f09_0    conda-forge
openssl                   3.0.2                h8ffe710_1    conda-forge
pip                       22.0.4             pyhd8ed1ab_0    conda-forge
python                    3.9.12          hcf16a7b_1_cpython    conda-forge
python_abi                3.9                      2_cp39    conda-forge
scipy                     1.7.3            py39hc0c34ad_0    conda-forge
setuptools                61.3.0           py39hcbf5309_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlite                    3.37.1               h8ffe710_0    conda-forge
tbb                       2021.5.0             h2d74725_0    conda-forge
tk                        8.6.12               h8ffe710_0    conda-forge
toolz                     0.11.2             pyhd8ed1ab_0    conda-forge
typing_extensions         4.1.1              pyha770c72_0    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
ucrt                      10.0.20348.0         h57928b3_0    conda-forge
vc                        14.2                 hb210afc_6    conda-forge
vs2015_runtime            14.29.30037          h902a5da_6    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h62dcd97_1    conda-forge

Environment info

active environment : aesara-test
    active env location : C:\Users\osthege\AppData\Local\Continuum\miniconda3\envs\aesara-test
            shell level : 1
       user config file : C:\Users\osthege\.condarc
 populated config files : C:\Users\osthege\.condarc
          conda version : 4.12.0
    conda-build version : 3.10.9
         python version : 3.7.2.final.0
       virtual packages : __cuda=11.0=0
                          __win=0=0
                          __archspec=1=x86_64
       base environment : C:\Users\osthege\AppData\Local\Continuum\miniconda3  (writable)
      conda av data dir : C:\Users\osthege\AppData\Local\Continuum\miniconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\osthege\AppData\Local\Continuum\miniconda3\pkgs
                          C:\Users\osthege\.conda\pkgs
                          C:\Users\osthege\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\osthege\AppData\Local\Continuum\miniconda3\envs
                          C:\Users\osthege\.conda\envs
                          C:\Users\osthege\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.12.0 requests/2.27.1 CPython/3.7.2 Windows/10 Windows/10.0.19041
          administrator : False
             netrc file : None
           offline mode : False

[michaelosthege]

Testing with the command from the PyMC Installation Guide (WIndows)

• ✔ mamba create -n pm3v4 -c conda-forge python=3.9 libpython mkl-service numba python-graphviz scipy arviz "aesara==2.2.6" "scipy<1.8.0"
• ✔ mamba create -n pm3v4 -c conda-forge python=3.9 libpython mkl-service numba python-graphviz scipy arviz "aesara==2.3.3" "scipy<1.8.0"
• ❌ mamba create -n pm3v4 -c conda-forge python=3.9 libpython mkl-service numba python-graphviz scipy arviz "aesara==2.3.4" "scipy<1.8.0"
• ❌ mamba create -n pm3v4 -c conda-forge python=3.9 libpython mkl-service numba python-graphviz scipy arviz "aesara==2.3.5" "scipy<1.8.0"
• ❌ mamba create -n pm3v4 -c conda-forge python=3.9 libpython mkl-service numba python-graphviz scipy arviz "aesara==2.3.8" "scipy<1.8.0"

And reproducing with a minimal env:

• ✔ mamba create -n aesara233 -c conda-forge python=3.9 "aesara==2.3.3" "scipy<1.8.0"
• ❌ mamba create -n aesara234 -c conda-forge python=3.9 "aesara==2.3.4" "scipy<1.8.0"

I compared the mamba list and the only difference is Aesara itself.

So the problem was introduced in aesara-devs/aesara@rel-2.3.3...rel-2.3.4

Upon further investigation it looks like I don't see performance differences between pymc==4.0.0b2 and pymc==4.0.0b6.
So maybe my performance problems were unrelated to the prints I'm seeing on import.

Still, the question remains if these prints are indicative of broken dependencies (and should be fixed via the feedstock), or if they are practically irrelevant and should be hidden.

[brandonwillard]

Doesn't a Windows installation require an additional Conda package like m2w64? The Aesara documentation mentions that here.

Aside from that, the warnings you're seeing look like the ones emitted by something like numpy.distutils.system_info.get_info. I believe those messages are constructed via logging, so their visibility can be affected by your local logging settings.

Regardless, it looks like the environments you're using/creating do not have build toolchains that NumPy and—as a result—Aesara can recognize/use.

[michaelosthege]

Including m2w64-toolchain doesn't help with these warnings.
Besides, shouldn't a "highly recommended" dependency included by the aesara-feedstock?

mamba create -n aesara253 -c conda-forge python=3.9 "aesara==2.5.3" "scipy<1.8.0" m2w64-toolchain -y

mamba logs

C:\>mamba create -n aesara253 -c conda-forge python=3.9 "aesara==2.5.3" "scipy<1.8.0" m2w64-toolchain -y

                  __    __    __    __
                 /  \  /  \  /  \  /  \
                /    \/    \/    \/    \
███████████████/  /██/  /██/  /██/  /████████████████████████
              /  / \   / \   / \   / \  \____
             /  /   \_/   \_/   \_/   \    o \__,
            / _/                       \_____/  `
            |/
        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗
        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║
        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║
        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝

        mamba (0.22.1) supported by @QuantStack

        GitHub:  https://github.com/mamba-org/mamba
        Twitter: https://twitter.com/QuantStack

█████████████████████████████████████████████████████████████


Looking for: ['python=3.9', 'aesara==2.5.3', "scipy[version='<1.8.0']", 'm2w64-toolchain']

pkgs/main/noarch                                   806.2kB @   2.2MB/s  0.4s
pkgs/msys2/noarch                                  111.0 B @ 261.0 B/s  0.1s
pkgs/r/win-64                                      743.0kB @   1.7MB/s  0.5s
pkgs/msys2/win-64                                   39.8kB @  79.4kB/s  0.1s
pkgs/r/noarch                                      482.4kB @ 793.0kB/s  0.2s
pkgs/main/win-64                                     3.9MB @   3.8MB/s  1.4s
conda-forge/noarch                                   7.8MB @   3.4MB/s  3.0s
conda-forge/win-64                                  14.1MB @   3.4MB/s  5.5s
Transaction

  Prefix: C:\Users\osthege\AppData\Local\Continuum\miniconda3\envs\aesara253

  Updating specs:

   - python=3.9
   - aesara==2.5.3
   - scipy[version='<1.8.0']
   - m2w64-toolchain


  Package                                      Version  Build               Channel                  Size
-----------------------------------------------------------------------------------------------------------
  Install:
-----------------------------------------------------------------------------------------------------------

  + aesara                                       2.5.3  py39h7fcc1c2_0      conda-forge/win-64     Cached
  + bzip2                                        1.0.8  h8ffe710_4          conda-forge/win-64     Cached
  + ca-certificates                          2021.10.8  h5b45459_0          conda-forge/win-64     Cached
  + cons                                         0.4.5  pyhd8ed1ab_0        conda-forge/noarch     Cached
  + etuples                                      0.3.5  pyhd8ed1ab_0        conda-forge/noarch     Cached
  + filelock                                     3.6.0  pyhd8ed1ab_0        conda-forge/noarch     Cached
  + intel-openmp                              2022.0.0  h57928b3_3663       conda-forge/win-64     Cached
  + libblas                                      3.9.0  13_win64_mkl        conda-forge/win-64     Cached
  + libcblas                                     3.9.0  13_win64_mkl        conda-forge/win-64     Cached
  + libffi                                       3.4.2  h8ffe710_5          conda-forge/win-64     Cached
  + liblapack                                    3.9.0  13_win64_mkl        conda-forge/win-64     Cached
  + libpython                                      2.1  py39hcbf5309_1      conda-forge/win-64     Cached
  + libzlib                                     1.2.11  h8ffe710_1014       conda-forge/win-64     Cached
  + logical-unification                          0.4.5  pyhd8ed1ab_0        conda-forge/noarch     Cached
  + m2w64-binutils                              2.25.1  5                   conda-forge/win-64     Cached
  + m2w64-bzip2                                  1.0.6  6                   conda-forge/win-64     Cached
  + m2w64-crt-git                   5.0.0.4636.2595836  2                   conda-forge/win-64     Cached
  + m2w64-gcc                                    5.3.0  6                   conda-forge/win-64     Cached
  + m2w64-gcc-ada                                5.3.0  6                   conda-forge/win-64     Cached
  + m2w64-gcc-fortran                            5.3.0  6                   conda-forge/win-64     Cached
  + m2w64-gcc-libgfortran                        5.3.0  6                   conda-forge/win-64     Cached
  + m2w64-gcc-libs                               5.3.0  7                   conda-forge/win-64     Cached
  + m2w64-gcc-libs-core                          5.3.0  7                   conda-forge/win-64     Cached
  + m2w64-gcc-objc                               5.3.0  6                   conda-forge/win-64     Cached
  + m2w64-gmp                                    6.1.0  2                   conda-forge/win-64     Cached
  + m2w64-headers-git               5.0.0.4636.c0ad18a  2                   conda-forge/win-64     Cached
  + m2w64-isl                                   0.16.1  2                   conda-forge/win-64     Cached
  + m2w64-libiconv                                1.14  6                   conda-forge/win-64     Cached
  + m2w64-libmangle-git             5.0.0.4509.2e5a9a2  2                   conda-forge/win-64     Cached
  + m2w64-libwinpthread-git         5.0.0.4634.697f757  2                   conda-forge/win-64     Cached
  + m2w64-make                        4.1.2351.a80a8b8  2                   conda-forge/win-64     Cached
  + m2w64-mpc                                    1.0.3  3                   conda-forge/win-64     Cached
  + m2w64-mpfr                                   3.1.4  4                   conda-forge/win-64     Cached
  + m2w64-pkg-config                            0.29.1  2                   conda-forge/win-64     Cached
  + m2w64-toolchain                              5.3.0  7                   conda-forge/win-64     Cached
  + m2w64-toolchain_win-64                       2.4.0  0                   conda-forge/win-64     Cached
  + m2w64-tools-git                 5.0.0.4592.90b8472  2                   conda-forge/win-64     Cached
  + m2w64-windows-default-manifest                 6.4  3                   conda-forge/win-64     Cached
  + m2w64-winpthreads-git           5.0.0.4634.697f757  2                   conda-forge/win-64     Cached
  + m2w64-zlib                                   1.2.8  10                  conda-forge/win-64     Cached
  + minikanren                                   1.0.3  pyhd8ed1ab_0        conda-forge/noarch     Cached
  + mkl                                       2022.0.0  h0e2418a_796        conda-forge/win-64     Cached
  + msys2-conda-epoch                         20160418  1                   conda-forge/win-64     Cached
  + multipledispatch                             0.6.0  py_0                conda-forge/noarch     Cached
  + numpy                                       1.22.3  py39h6331f09_0      conda-forge/win-64     Cached
  + openssl                                      3.0.2  h8ffe710_1          conda-forge/win-64     Cached
  + pip                                         22.0.4  pyhd8ed1ab_0        conda-forge/noarch     Cached
  + python                                      3.9.12  hcf16a7b_1_cpython  conda-forge/win-64     Cached
  + python_abi                                     3.9  2_cp39              conda-forge/win-64     Cached
  + scipy                                        1.7.3  py39hc0c34ad_0      conda-forge/win-64     Cached
  + setuptools                                  61.3.1  py39hcbf5309_0      conda-forge/win-64        1MB
  + six                                         1.16.0  pyh6c4a22f_0        conda-forge/noarch     Cached
  + sqlite                                      3.37.1  h8ffe710_0          conda-forge/win-64     Cached
  + tbb                                       2021.5.0  h2d74725_1          conda-forge/win-64      152kB
  + tk                                          8.6.12  h8ffe710_0          conda-forge/win-64     Cached
  + toolz                                       0.11.2  pyhd8ed1ab_0        conda-forge/noarch     Cached
  + typing_extensions                            4.1.1  pyha770c72_0        conda-forge/noarch     Cached
  + tzdata                                       2022a  h191b570_0          conda-forge/noarch     Cached
  + ucrt                                  10.0.20348.0  h57928b3_0          conda-forge/win-64     Cached
  + vc                                            14.2  hb210afc_6          conda-forge/win-64     Cached
  + vs2015_runtime                         14.29.30037  h902a5da_6          conda-forge/win-64     Cached
  + wheel                                       0.37.1  pyhd8ed1ab_0        conda-forge/noarch     Cached
  + xz                                           5.2.5  h62dcd97_1          conda-forge/win-64     Cached

  Summary:

  Install: 63 packages

  Total download: 1MB

-----------------------------------------------------------------------------------------------------------

tbb                                                151.9kB @ 762.7kB/s  0.2s
setuptools                                           1.3MB @   6.6MB/s  0.2s
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

To activate this environment, use

     $ mamba activate aesara253

To deactivate an active environment, use

     $ mamba deactivate


C:\>mamba activate aesara253

(aesara253) C:\>python -c "import aesara"
WARN: Could not locate executable g77
WARN: Could not locate executable f77
WARN: Could not locate executable ifort
WARN: Could not locate executable ifl
WARN: Could not locate executable f90
WARN: Could not locate executable DF
WARN: Could not locate executable efl
Using built-in specs.
COLLECT_GCC=C:\Users\osthege\AppData\Local\Continuum\miniconda3\envs\aesara253\Library\mingw-w64\bin\gfortran.exe
COLLECT_LTO_WRAPPER=C:/Users/osthege/AppData/Local/Continuum/miniconda3/envs/aesara253/Library/mingw-w64/bin/../lib/gcc/x86_64-w64-mingw32/5.3.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../gcc-5.3.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/include --libexecdir=/mingw64/lib --with-gxx-include-dir=/mingw64/include/c++/5.3.0 --enable-bootstrap --with-arch=x86-64 --with-tune=generic --enable-languages=c,lto,c++,objc,obj-c++,fortran,ada --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --enable-version-specific-runtime-libs --disable-isl-version-check --enable-lto --enable-libgomp --disable-multilib --enable-checking=release --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --with-isl=/mingw64 --with-pkgversion='Rev5, Built by MSYS2 project' --with-bugurl=https://sourceforge.net/projects/msys2 --with-gnu-as --with-gnu-ld
Thread model: posix
gcc version 5.3.0 (Rev5, Built by MSYS2 project)
WARNING (aesara.tensor.blas): Using NumPy C-API based implementation for BLAS functions.

I believe those messages are constructed via logging, so their visibility can be affected by your local logging settings.

Yes, they are created by logging, with the WARNING level. I do not tweak my local logging settings. This is all with default behavior.

Regardless, it looks like the environments you're using/creating do not have build toolchains that NumPy and—as a result—Aesara can recognize/use.

Same with the NumPy build toolchains. What's the point of installing Aesara through conda-forge if it doesn't do a better job than pip when it comes to installing performance-relevant build dependencies?

[twiecki]

Well the fact that it's different between aesara versions is definitely a smoking gun.

These are the PRs on the feedstock that could be related, although I don't think it completely matches with the versioning:
#48
#29 (this one specifically)

[michaelosthege]

There was a post on Discourse where a one-liner setting in .aesararc made the prints go away.

https://discourse.pymc.io/t/pymc-installation-issue/9279/5

@brandonwillard is that a sensible solution? And if so, could it become the default?

[brandonwillard]

There was a post on Discourse where a one-liner setting in .aesararc made the prints go away.

https://discourse.pymc.io/t/pymc-installation-issue/9279/5

@brandonwillard is that a sensible solution? And if so, could it become the default?

Yeah, something like that could work. @dgerlanc has been looking into the Mac compilation issues; he might have precision answers to a few common Mac + architecture cases, as well.

[maresb]

To clarify, this is essentially two issues:

1. Numpy emits the following errors which should be filtered out. I hopefully have a solution with Catch Numpy "Could not locate executable" warnings aesara-devs/aesara#980

WARN: Could not locate executable g77
WARN: Could not locate executable f77
WARN: Could not locate executable ifort
WARN: Could not locate executable ifl
WARN: Could not locate executable f90
WARN: Could not locate executable DF
WARN: Could not locate executable efl

2. A quick BLAS has not been set up, resulting in the following very relevant warning:

WARNING (aesara.tensor.blas): Using NumPy C-API based implementation for BLAS functions.

This can be resolved by installing mkl-service, or in the case of Apple M1 where mkl-service is inapplicable, openblas can be used as a substitute.