Add _logger.debug statements to cmodule default blas ldflags #528

lucianopaz · 2023-12-01T10:08:49Z

closes #527

michaelosthege

flake8 or so taught me to do it like this..

michaelosthege · 2023-12-01T10:19:40Z

pytensor/link/c/cmodule.py

@@ -2718,6 +2718,7 @@ def check_required_file(paths, required_regexs):
                    found = True
                    break
            if not found:
+                _logger.debug(f"Required file {req} not found")


Suggested change

_logger.debug(f"Required file {req} not found")

_logger.debug("Required file '%s' not found", req)

michaelosthege · 2023-12-01T10:19:52Z

pytensor/link/c/cmodule.py

+                    check_mkl_openmp()
+                except Exception as e:
+                    _logger.debug(e)
+            _logger.debug(f"The following blas flags will be used {res}")


Suggested change

_logger.debug(f"The following blas flags will be used {res}")

_logger.debug("The following blas flags will be used: '%s'", res)

michaelosthege · 2023-12-01T10:20:13Z

pytensor/link/c/cmodule.py

            return res
        else:
+            _logger.debug(f"Supplied flags {res} failed to compile")


Suggested change

_logger.debug(f"Supplied flags {res} failed to compile")

_logger.debug("Supplied flags '%s' failed to compile", res)

michaelosthege · 2023-12-01T10:21:30Z

pytensor/link/c/cmodule.py

+    _logger.debug("Will search for BLAS libraries in the following directories:")
+    _logger.debug("{}".format("\n".join(searched_library_dirs)))


Suggested change

_logger.debug("Will search for BLAS libraries in the following directories:")

_logger.debug("{}".format("\n".join(searched_library_dirs)))

search_dirs = "\n".join(searched_library_dirs)

_logger.debug("Will search for BLAS libraries in the following directories:\n%s", search_dirs)

lucianopaz · 2023-12-01T10:25:12Z

flake8 or so taught me to do it like this..

Do you really want to use old style string interpolation instead of the newer f-strings?

michaelosthege · 2023-12-01T10:28:47Z

flake8 or so taught me to do it like this..

Do you really want to use old style string interpolation instead of the newer f-strings?

It's because this way, if there's an error it gets logged properly

lucianopaz · 2023-12-01T21:28:30Z

Out of curiosity, I looked a bit into why one would want to use old style string interpolation in logs. It seems like both formats will work fine and log properly, but the one you are suggested is only interpolated when the message is actually logged, while the modern interpolations are performed regardless of whether the message needs to be logged or not. I'll apply your suggestions and merge this.

codecov-commenter · 2023-12-01T21:55:20Z

Codecov Report

Merging #528 (db41c43) into main (7ecb9f8) will increase coverage by 0.01%.
Report is 3 commits behind head on main.
The diff coverage is 91.52%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #528      +/-   ##
==========================================
+ Coverage   80.82%   80.84%   +0.01%     
==========================================
  Files         162      162              
  Lines       46194    46269      +75     
  Branches    11288    11320      +32     
==========================================
+ Hits        37337    37404      +67     
- Misses       6630     6637       +7     
- Partials     2227     2228       +1

Files	Coverage Δ
pytensor/compile/function/types.py	`79.86% <100.00%> (+0.13%)`	⬆️
pytensor/compile/profiling.py	`76.19% <100.00%> (+0.06%)`	⬆️
pytensor/graph/replace.py	`84.84% <100.00%> (-0.31%)`	⬇️
pytensor/tensor/rewriting/basic.py	`94.09% <100.00%> (+0.08%)`	⬆️
pytensor/tensor/rewriting/math.py	`86.50% <100.00%> (+<0.01%)`	⬆️
pytensor/tensor/rewriting/shape.py	`81.43% <100.00%> (+0.41%)`	⬆️
pytensor/tensor/rewriting/subtensor.py	`88.80% <100.00%> (+0.07%)`	⬆️
pytensor/tensor/shape.py	`93.09% <86.66%> (+0.01%)`	⬆️
pytensor/link/c/cmodule.py	`55.74% <84.61%> (+0.21%)`	⬆️
pytensor/tensor/basic.py	`88.39% <87.50%> (-0.05%)`	⬇️

michaelosthege · 2023-12-01T22:20:57Z

Out of curiosity, I looked a bit into why one would want to use old style string interpolation in logs. It seems like both formats will work fine and log properly, but the one you are suggested is only interpolated when the message is actually logged, while the modern interpolations are performed regardless of whether the message needs to be logged or not. I'll apply your suggestions and merge this.

Thanks for the explanation!

I installed from this branch into my Docker image. The logger didn't show right away, and I had to set its level even though I had previously done a logging.basicConfig(level=logging.DEBUG). No Idea why.

>>> pytensor.link.c.cmodule._logger.setLevel(logging.DEBUG)
>>> pytensor.link.c.cmodule._logger.debug("test")
DEBUG:pytensor.link.c.cmodule:test
>>> pytensor.link.c.cmodule.default_blas_ldflags()
DEBUG:pytensor.link.c.cmodule:Will search for BLAS libraries in the following directories:
/opt/mamba/lib/gcc/x86_64-conda-linux-gnu/12.3.0
/opt/mamba/lib/gcc
/opt/mamba/x86_64-conda-linux-gnu/lib
/opt/mamba/lib
/opt/mamba/x86_64-conda-linux-gnu/sysroot/lib64
/opt/mamba/x86_64-conda-linux-gnu/sysroot/usr/lib64
/opt/mamba/x86_64-conda-linux-gnu/lib
/opt/mamba/lib
/opt/mamba/x86_64-conda-linux-gnu/sysroot/lib64
/opt/mamba/x86_64-conda-linux-gnu/sysroot/usr/lib64
/opt/mamba/lib
DEBUG:pytensor.link.c.cmodule:Checking MKL flags with intel threading
DEBUG:pytensor.link.c.cmodule:Compiling for 64 bit architecture
DEBUG:pytensor.link.c.cmodule:Supplied flags  failed to compile
DEBUG:pytensor.link.c.cmodule:Supplied flags '' failed to compile
DEBUG:pytensor.link.c.cmodule:Supplied flags ['-lmkl_core', '-lmkl_rt', '-lmkl_intel_thread', '-liomp5', '-lpthread', '-Wl,-rpath,/opt/mamba/lib'] failed to compile
DEBUG:pytensor.link.c.cmodule:Checking MKL flags with GNU OpenMP threading
DEBUG:pytensor.link.c.cmodule:Compiling for 64 bit architecture
DEBUG:pytensor.link.c.cmodule:Supplied flags  failed to compile
DEBUG:pytensor.link.c.cmodule:Supplied flags '' failed to compile
DEBUG:pytensor.link.c.cmodule:Supplied flags ['-lmkl_core', '-lmkl_rt', '-lmkl_gnu_thread', '-lgomp', '-lpthread', '-Wl,-rpath,/opt/mamba/lib'] failed to compile
DEBUG:pytensor.link.c.cmodule:Checking Lapack + blas
DEBUG:pytensor.link.c.cmodule:Compiling for 64 bit architecture
DEBUG:pytensor.link.c.cmodule:Supplied flags  failed to compile
DEBUG:pytensor.link.c.cmodule:Supplied flags '' failed to compile
DEBUG:pytensor.link.c.cmodule:Supplied flags ['-llapack', '-lblas', '-lcblas', '-lm', '-Wl,-rpath,/opt/mamba/lib'] failed to compile
DEBUG:pytensor.link.c.cmodule:Checking blas alone
DEBUG:pytensor.link.c.cmodule:Compiling for 64 bit architecture
DEBUG:pytensor.link.c.cmodule:Supplied flags  failed to compile
DEBUG:pytensor.link.c.cmodule:Supplied flags '' failed to compile
DEBUG:pytensor.link.c.cmodule:Supplied flags ['-lblas', '-lcblas', '-Wl,-rpath,/opt/mamba/lib'] failed to compile
DEBUG:pytensor.link.c.cmodule:Checking openblas
DEBUG:pytensor.link.c.cmodule:Compiling for 64 bit architecture
DEBUG:pytensor.link.c.cmodule:Supplied flags  failed to compile
DEBUG:pytensor.link.c.cmodule:Supplied flags '' failed to compile
DEBUG:pytensor.link.c.cmodule:Supplied flags ['-lopenblas', '-lgfortran', '-lgomp', '-lm', '-fopenmp', '-Wl,-rpath,/opt/mamba/lib'] failed to compile
DEBUG:pytensor.link.c.cmodule:Failed to identify blas ldflags. Will leave them empty.
''

michaelosthege · 2023-12-01T22:51:43Z

With a bit more detailed logging (https://github.com/michaelosthege/pytensor/tree/mode-cmodule-logging)

DEBUG:pytensor.link.c.cmodule:Checking MKL flags with intel threading
DEBUG:pytensor.link.c.cmodule:Compiling for 64 bit architecture
DEBUG:pytensor.link.c.cmodule:Test compilation with flags '['-lmkl_core', '-lmkl_rt', '-lmkl_intel_thread', '-liomp5', '-lpthread', '-Wl,-rpath,/opt/mamba/lib', '-L/opt/mamba/lib']' resulted in
(False, False)
DEBUG:pytensor.link.c.cmodule:Supplied flags '['-lmkl_core', '-lmkl_rt', '-lmkl_intel_thread', '-liomp5', '-lpthread', '-Wl,-rpath,/opt/mamba/lib']' failed to compile
DEBUG:pytensor.link.c.cmodule:Supplied flags ['-lmkl_core', '-lmkl_rt', '-lmkl_intel_thread', '-liomp5', '-lpthread', '-Wl,-rpath,/opt/mamba/lib'] failed to compile
DEBUG:pytensor.link.c.cmodule:Checking MKL flags with GNU OpenMP threading
DEBUG:pytensor.link.c.cmodule:Compiling for 64 bit architecture
DEBUG:pytensor.link.c.cmodule:Test compilation with flags '['-lmkl_core', '-lmkl_rt', '-lmkl_gnu_thread', '-lgomp', '-lpthread', '-Wl,-rpath,/opt/mamba/lib', '-L/opt/mamba/lib']' resulted in
(False, False)
DEBUG:pytensor.link.c.cmodule:Supplied flags '['-lmkl_core', '-lmkl_rt', '-lmkl_gnu_thread', '-lgomp', '-lpthread', '-Wl,-rpath,/opt/mamba/lib']' failed to compile
DEBUG:pytensor.link.c.cmodule:Supplied flags ['-lmkl_core', '-lmkl_rt', '-lmkl_gnu_thread', '-lgomp', '-lpthread', '-Wl,-rpath,/opt/mamba/lib'] failed to compile
DEBUG:pytensor.link.c.cmodule:Checking Lapack + blas
DEBUG:pytensor.link.c.cmodule:Compiling for 64 bit architecture
DEBUG:pytensor.link.c.cmodule:Test compilation with flags '['-llapack', '-lblas', '-lcblas', '-lm', '-Wl,-rpath,/opt/mamba/lib', '-L/opt/mamba/lib']' resulted in
(False, False)
DEBUG:pytensor.link.c.cmodule:Supplied flags '['-llapack', '-lblas', '-lcblas', '-lm', '-Wl,-rpath,/opt/mamba/lib']' failed to compile
DEBUG:pytensor.link.c.cmodule:Supplied flags ['-llapack', '-lblas', '-lcblas', '-lm', '-Wl,-rpath,/opt/mamba/lib'] failed to compile
DEBUG:pytensor.link.c.cmodule:Checking blas alone
DEBUG:pytensor.link.c.cmodule:Compiling for 64 bit architecture
DEBUG:pytensor.link.c.cmodule:Test compilation with flags '['-lblas', '-lcblas', '-Wl,-rpath,/opt/mamba/lib', '-L/opt/mamba/lib']' resulted in
(False, False)
DEBUG:pytensor.link.c.cmodule:Supplied flags '['-lblas', '-lcblas', '-Wl,-rpath,/opt/mamba/lib']' failed to compile
DEBUG:pytensor.link.c.cmodule:Supplied flags ['-lblas', '-lcblas', '-Wl,-rpath,/opt/mamba/lib'] failed to compile
DEBUG:pytensor.link.c.cmodule:Checking openblas
DEBUG:pytensor.link.c.cmodule:Compiling for 64 bit architecture
DEBUG:pytensor.link.c.cmodule:Test compilation with flags '['-lopenblas', '-lgfortran', '-lgomp', '-lm', '-fopenmp', '-Wl,-rpath,/opt/mamba/lib', '-L/opt/mamba/lib']' resulted in
(False, False)
DEBUG:pytensor.link.c.cmodule:Supplied flags '['-lopenblas', '-lgfortran', '-lgomp', '-lm', '-fopenmp', '-Wl,-rpath,/opt/mamba/lib']' failed to compile
DEBUG:pytensor.link.c.cmodule:Supplied flags ['-lopenblas', '-lgfortran', '-lgomp', '-lm', '-fopenmp', '-Wl,-rpath,/opt/mamba/lib'] failed to compile
DEBUG:pytensor.link.c.cmodule:Failed to identify blas ldflags. Will leave them empty.
''

lucianopaz added the maintenance label Dec 1, 2023

lucianopaz requested a review from michaelosthege December 1, 2023 10:09

michaelosthege approved these changes Dec 1, 2023

View reviewed changes

Add _logger.debug statements to cmodule default blas ldflags

db41c43

lucianopaz force-pushed the cmodule_logs branch from 9f1bbef to db41c43 Compare December 1, 2023 21:32

twiecki merged commit c38e7ba into pymc-devs:main Dec 1, 2023

michaelosthege mentioned this pull request Dec 3, 2023

Empty BLAS flags in mamba-installed v2.18.1 environment under Docker #523

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add _logger.debug statements to cmodule default blas ldflags #528

Add _logger.debug statements to cmodule default blas ldflags #528

lucianopaz commented Dec 1, 2023

michaelosthege left a comment

michaelosthege Dec 1, 2023

michaelosthege Dec 1, 2023

michaelosthege Dec 1, 2023

michaelosthege Dec 1, 2023

lucianopaz commented Dec 1, 2023

michaelosthege commented Dec 1, 2023

lucianopaz commented Dec 1, 2023

codecov-commenter commented Dec 1, 2023

michaelosthege commented Dec 1, 2023

michaelosthege commented Dec 1, 2023 •

edited

Loading

	_logger.debug(f"Required file {req} not found")
	_logger.debug("Required file '%s' not found", req)

	_logger.debug(f"The following blas flags will be used {res}")
	_logger.debug("The following blas flags will be used: '%s'", res)

	_logger.debug(f"Supplied flags {res} failed to compile")
	_logger.debug("Supplied flags '%s' failed to compile", res)

		_logger.debug("Will search for BLAS libraries in the following directories:")
		_logger.debug("{}".format("\n".join(searched_library_dirs)))

Add _logger.debug statements to cmodule default blas ldflags #528

Add _logger.debug statements to cmodule default blas ldflags #528

Conversation

lucianopaz commented Dec 1, 2023

michaelosthege left a comment

Choose a reason for hiding this comment

michaelosthege Dec 1, 2023

Choose a reason for hiding this comment

michaelosthege Dec 1, 2023

Choose a reason for hiding this comment

michaelosthege Dec 1, 2023

Choose a reason for hiding this comment

michaelosthege Dec 1, 2023

Choose a reason for hiding this comment

lucianopaz commented Dec 1, 2023

michaelosthege commented Dec 1, 2023

lucianopaz commented Dec 1, 2023

codecov-commenter commented Dec 1, 2023

Codecov Report

michaelosthege commented Dec 1, 2023

michaelosthege commented Dec 1, 2023 • edited Loading

michaelosthege commented Dec 1, 2023 •

edited

Loading