pybind11-2.13.6-GCC-14.2.0 sanity check command failed

[sassy-crick]

When I am installing software on our cluster, we are doing the build in Rocky-8.9 container and then we test using Alma-8.7. These tests are a simply sanity-check command like this:

$ eb --sanity-check-only --rebuild pybind11-2.13.6-GCC-14.2.0.eb

The --rebuild came from as the sanity check failed originally with:

ERROR: Shell command failed!
    full command              ->  python -c 'import sys; print("%s.%s.%s" % sys.version_info[:3])'
    exit code                 ->  127
    called from               ->  'det_python_version' function in /sw-eb/software/EasyBuild/5.1.1/lib/python3.6/site-packages/easybuild/easyblocks/generic/pythonpackage.py (line 81)

I, wrongly, assumed it tries to use the system-python, which is python3 and the single command python is not resolved. So I updated the container to Alma-8.9 and added a symlink from /usr/bin/python3 to /usr/bin/python`. Now the failure was like this:

== sanity checking...
  >> running shell command:
        None -c 'import pybind11; print(pybind11.get_include())'

Now that made even less sense, so I was digging a bit more. Outside a container, and that has been confirmed independently by @Thyre , I can reproduce the issue:

$ eb -d --sanity-check-only pybind11-2.13.6-GCC-14.2.0.eb 
== Running parse hook for GCC-14.2.0.eb...
== Running parse hook for GCC-14.2.0.eb...
== Running parse hook for GCCcore-14.2.0.eb...
== Running parse hook for binutils-2.42-GCCcore-14.2.0.eb...
== Running parse hook for GCCcore-14.2.0.eb...
== Running parse hook for M4-1.4.19.eb...
== Running parse hook for binutils-2.42.eb...
== resolving dependencies 
== processing EasyBuild easyconfig /sw-eb/software/EasyBuild/5.1.1/easybuild/easyconfigs/p/pybind11/pybind11-2.13.6-GCC-14.2.0.eb
== sanity checking...
∙∙∙ Installing pybind11/2.13.6-GCC-14.2.0: sanity checking (12 out of 17 steps done) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━ 0:00:00

ERROR: Shell command failed!
    full command              ->  python -c 'import sys; print("%s.%s.%s" % sys.version_info[:3])'
    exit code                 ->  127
    called from               ->  'det_python_version' function in /sw-eb/software/EasyBuild/5.1.1/lib/python3.6/site-packages/easybuild/easyblocks/generic/pythonpackage.py (line 81)
    working directory         ->  /rds/general/user/jsassman/home
    output (stdout + stderr)  ->  /dev/shm/jsassman/eb-8j2bdc6f/run-shell-cmd-output/python-k7v775v5/out.txt
    interactive shell script  ->  /dev/shm/jsassman/eb-8j2bdc6f/run-shell-cmd-output/python-k7v775v5/cmd.sh

with the content of out.txt:

/usr/bin/bash: python: command not found

and the content of cnd.sh

# Run this script to set up a shell environment that EasyBuild used to run the shell command
EB_SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
echo "# Shell for the command: '"'python -c '"'"'import sys; print("%s.%s.%s" % sys.version_info[:3])'"'"''"'"
echo "# Use command history, exit to stop"
bash --rcfile $EB_SCRIPT_DIR/env.sh -i "$@"

I have attached the env.sh file as env.txt

I got the feeling the Python-module is not loaded, as, when I was running the cmd.sh command from the failed build I got:

# Shell for the command: 'None -c 'import pybind11; print(pybind11.get_include())''
# Use command history, exit to stop
Apptainer> ml

Currently Loaded Modules:
  1) EasyBuild/5.1.1   2) GCCcore/14.2.0   3) zlib/1.3.1-GCCcore-14.2.0   4) binutils/2.42-GCCcore-14.2.0   5) GCC/14.2.0   6) pybind11/2.13.6-GCC-14.2.0

Again, that was verified independently.

The really odd thing is: during the installation the same command is being used, and here it works.

Anybody got any ideas? That is with EasyBuild-5.1.1, EasyBuild-5.1.0, EasyBuild-5.0.0, and also EasyBuild-4.9.4, which I find surprising.

[branfosj]

The currently loaded modules does not include Python, but Python is a dependency in pybind11-2.13.6-GCC-14.2.0.eb: https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/p/pybind11/pybind11-2.13.6-GCC-14.2.0.eb#L30

[branfosj]

But yes it fails...

[branfosj]

The fake module generated for the sanity check does not have the Boost and Python dependencies

help([==[

Description
===========
pybind11 is a lightweight header-only library that exposes C++ types in Python and vice versa,
 mainly to create Python bindings of existing C++ code.


More information
================
 - Homepage: https://pybind11.readthedocs.io
]==])

whatis([==[Description: pybind11 is a lightweight header-only library that exposes C++ types in Python and vice versa,
 mainly to create Python bindings of existing C++ code.]==])
whatis([==[Homepage: https://pybind11.readthedocs.io]==])
whatis([==[URL: https://pybind11.readthedocs.io]==])

local root = "/rds/projects/2017/branfosj-rse/easybuild/EL8-ice/software/pybind11/2.13.6-GCC-14.2.0"

conflict("pybind11")

depends_on("GCC/14.2.0")

prepend_path("CMAKE_PREFIX_PATH", root)
prepend_path("CPATH", pathJoin(root, "include"))
prepend_path("PATH", pathJoin(root, "bin"))
prepend_path("PKG_CONFIG_PATH", pathJoin(root, "share", "pkgconfig"))
prepend_path("XDG_DATA_DIRS", pathJoin(root, "share"))
setenv("EBROOTPYBIND11", root)
setenv("EBVERSIONPYBIND11", "2.13.6")
setenv("EBDEVELPYBIND11", pathJoin(root, "easybuild", "pybind11-2.13.6-GCC-14.2.0-easybuild-devel"))

prepend_path("PYTHONPATH", pathJoin(root, "lib", "python3.13", "site-packages"))
-- Built with EasyBuild version 5.1.2.dev0-rb9372bab75f88ead6fc2188e49f6ce4e386e327f

[sassy-crick]

Thanks, that confirms my suspicion. I assumed the 'fake' module is calling the 'real' module generation process, but is being put in a temporary directory. That way, so I assumed, we check also the correct module generation. Where would I find the fake module?

[boegel]

Thanks, that confirms my suspicion. I assumed the 'fake' module is calling the 'real' module generation process, but is being put in a temporary directory. That way, so I assumed, we check also the correct module generation. Where would I find the fake module?

Indeed, this fake (temporary would be a better name) module is generated in a temporary directory.
You should be able to figure out where it's located (and even see it's contents) from the EasyBuild (debug) log.

I'll take a closer look at this, starting with trying to reproduce this problem.

It's clear that the fake module is incorrect, it should definitely have all (non-build) dependencies, not just the toolchain, not sure why that happens...

[boegel]

In fact, I'm a bit surprised that --sanity-check-only actually relies on a fake module at all: there should be no need to, since there should be a proper module file already, so EasyBuild should just load that rather than generating a fake module and using that for the sanity check.

Generating a fake module and using that for the sanity check only makes sense in a full install, or when --module-only is used (since the sanity check must pass before the final module file is generated and put in place).

[Thyre]

In fact, I'm a bit surprised that --sanity-check-only actually relies on a fake module at all: there should be no need to, since there should be a proper module file already, so EasyBuild should just load that rather than generating a fake module and using that for the sanity check.

I think this comes from the pybind11 EasyBlock (see the sanity check):

        if not self.is_extension:
            # only load fake module for stand-alone installations (not for extensions),
            # since for extension the necessary modules should already be loaded at this point
            fake_mod_data = self.load_fake_module(purge=True)
        cmd = "%s -c 'import pybind11; print(pybind11.get_include())'" % self.python_cmd
        res = run_shell_cmd(cmd, fail_on_error=False)

That may also be the reason why we don't see this for other software.

[branfosj]

Based off what we do in cudacompat.py and making sure we define python_cmd then the following works:

    def sanity_check_step(self):
        """
        Custom sanity check for Python packages
        """
        # don't add user site directory to sys.path (equivalent to python -s)
        env.setvar('PYTHONNOUSERSITE', '1', verbose=False)
        # Get python includes
        fake_mod_data = None
        if not self.is_extension:
            # only needed for stand-alone installations (not for extensions),
            # since for extension the necessary modules should already be loaded at this point
            if build_option('sanity_check_only'):
                # skip loading of fake module when using --sanity-check-only, load real module instead
                self.load_module()
                if not self.python_cmd:
                    self.prepare_python()
            elif not self.dry_run:
                # only load fake module for stand-alone installations (not for extensions),
                fake_mod_data = self.load_fake_module(purge=True, verbose=True)
        cmd = "%s -c 'import pybind11; print(pybind11.get_include())'" % self.python_cmd
        res = run_shell_cmd(cmd, fail_on_error=False)
        if res.exit_code:
            raise EasyBuildError("Failed to get pybind11 includes!")
        python_include = res.output.strip()
        if fake_mod_data:
            self.clean_up_fake_module(fake_mod_data)

        # Check for CMake config and includes
        custom_paths = {
            'files': ['share/cmake/pybind11/pybind11Config.cmake'],
            'dirs': ['include/pybind11', os.path.join(python_include, 'pybind11')],
        }
        # Check for Python module
        return PythonPackage.sanity_check_step(self, custom_paths=custom_paths)

Unsure if there is a better way, as that looks more complicated than I'd like it to be.

[boegel]

We introduced sanity_check_load_module in easybuilders/easybuild-framework#4125, easyblocks should be using that.

Fixed for pybind11 in:

• use sanity_check_load_module in sanity check step of custom easyblock for pybind11 easybuild-easyblocks#3864