Skip to content

Conversation

@Thyre
Copy link
Collaborator

@Thyre Thyre commented Jun 26, 2025

When EasyBlocks are not using the deprecated make_module_req_guess, the Bundle EasyBlock would only collect the module_load_environment set outside of make_module_step. make_module_step was never called by Bundle, and hence additional module paths were missing.

To fix this, also run the make_module step for each component.

Fixes #3800

When EasyBlocks are not using the deprecated `make_module_req_guess`,
the Bundle EasyBlock would only collect the `module_load_environment`
set outside of `make_module_step`. `make_module_step` was never called
by Bundle, and hence additional module paths were missing.

To fix this, also run the `module` step for each component.

Signed-off-by: Jan André Reuter <[email protected]>
@Thyre
Copy link
Collaborator Author

Thyre commented Jun 26, 2025

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS intel-compilers-2025.2.0-CUDA-12.9.0.eb
  • SUCCESS intel-compilers-2025.0.0-CUDA-12.8.0.eb
  • SUCCESS intel-compilers-2024.2.0-CUDA-12.5.0.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
ZAM054 - Linux Zorin OS 17, x86_64, 12th Gen Intel(R) Core(TM) i7-1260P, 1 x NVIDIA NVIDIA GeForce MX550, 565.57.01, Python 3.10.12
See https://gist.github.com/Thyre/80fb7cf280538c046b2f1ec2c9ab3117 for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Jun 26, 2025

Test report by @Thyre

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 3 (3 easyconfigs in total)
ZAM054 - Linux Zorin OS 17, x86_64, 12th Gen Intel(R) Core(TM) i7-1260P, 1 x NVIDIA NVIDIA GeForce MX550, 565.57.01, Python 3.10.12
See https://gist.github.com/Thyre/2ee9724d3f7f5b8fe296478163943458 for a full test report.


Still fails --module-only. So we probably need the call later on anyway...

Call `make_module_step` for each component explicitly in Bundles own
`make_module_step`, since step list in `install_step` will not be
executed in a module-only build.

Signed-off-by: Jan André Reuter <[email protected]>
@Thyre
Copy link
Collaborator Author

Thyre commented Jun 26, 2025

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS intel-compilers-2025.2.0-CUDA-12.9.0.eb
  • SUCCESS intel-compilers-2025.0.0-CUDA-12.8.0.eb
  • SUCCESS intel-compilers-2024.2.0-CUDA-12.5.0.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
ZAM054 - Linux Zorin OS 17, x86_64, 12th Gen Intel(R) Core(TM) i7-1260P, 1 x NVIDIA NVIDIA GeForce MX550, 565.57.01, Python 3.10.12
See https://gist.github.com/Thyre/743811790962985fe97c16587d0cfef4 for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Jun 26, 2025

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS intel-compilers-2025.2.0-CUDA-12.9.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
ZAM054 - Linux Zorin OS 17, x86_64, 12th Gen Intel(R) Core(TM) i7-1260P, 1 x NVIDIA NVIDIA GeForce MX550, 565.57.01, Python 3.10.12
See https://gist.github.com/Thyre/92eef59cc368dee097188da1a11668b1 for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Jun 26, 2025

Looking at this else branch, we might need both approaches, right? A component might set its module_load_environment in make_module_step which we're missing at the moment. The else branch then tries to update the environment with incomplete information, which may fail for a later component requiring this information.

For the deprecated approach, we call make_module_req_guess explicitly.

else:
# Update current environment with component environment to ensure stuff provided
# by this component can be picked up by installation of subsequent components
# - this is a stripped down version of EasyBlock.make_module_req for fake modules
# - once bundle installation is complete, this is handled by the generated module as usual
for mod_envar, mod_paths in comp.module_load_environment.items():
# expand glob patterns in module load environment to existing absolute paths
mod_expand = mod_paths.expand_paths(self.installdir)
mod_expand = [os.path.join(self.installdir, path) for path in mod_expand]
# prepend to current environment variable if new stuff added to installation
curr_env = os.getenv(mod_envar, '')
curr_paths = [path for path in curr_env.split(os.pathsep) if path]
new_paths = nub(mod_expand + curr_paths)
new_env = os.pathsep.join(new_paths)
if new_env and new_env != curr_env:
env.setvar(mod_envar, new_env)

Are there any EasyConfigs we could test this with?

Also sets fake=True for both calls to avoid multiple modules
with the same name being created during the install process.
This should shouldn't have broken something, it's simply cleaner.

Signed-off-by: Jan André Reuter <[email protected]>
@Thyre
Copy link
Collaborator Author

Thyre commented Jun 26, 2025

Rebuild:

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS GCC-14.2.0.eb
  • SUCCESS Autotools-20240712-GCCcore-14.2.0.eb
  • SUCCESS OpenSSL-3.eb
  • SUCCESS gfbf-2025a.eb

Build succeeded for 4 out of 4 (4 easyconfigs in total)
Linux - Linux Arch Linux UNKNOWN, x86_64, AMD Ryzen 7 7800X3D 8-Core Processor, 1 x AMD Navi 48 [RX 9070/9070 XT] (device id: 0x7550, gfx: gfx1201, driver: 6.15.3-arch1-1), 1 x AMD Raphael (device id: 0x164e, gfx: gfx1036, driver: 6.15.3-arch1-1), Python 3.13.5
See https://gist.github.com/Thyre/95271548a982229e5219ffaa4cabf058 for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Jun 26, 2025

Module only:

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS GCC-14.2.0.eb
  • SUCCESS Autotools-20240712-GCCcore-14.2.0.eb
  • SUCCESS OpenSSL-3.eb
  • SUCCESS gfbf-2025a.eb

Build succeeded for 4 out of 4 (4 easyconfigs in total)
Linux - Linux Arch Linux UNKNOWN, x86_64, AMD Ryzen 7 7800X3D 8-Core Processor, 1 x AMD Navi 48 [RX 9070/9070 XT] (device id: 0x7550, gfx: gfx1201, driver: 6.15.3-arch1-1), 1 x AMD Raphael (device id: 0x164e, gfx: gfx1036, driver: 6.15.3-arch1-1), Python 3.13.5
See https://gist.github.com/Thyre/9a93b805c73a51c6e4c2c9d6938cf091 for a full test report.


Interestingly, --module only adds this to each module.

== comparing module file with backup /data/EasyBuild-develop/modules/all/Core/gfbf/2025a.bak_20250626231901_12722; diff is:
--- /data/EasyBuild-develop/modules/all/Core/gfbf/2025a.bak_20250626231901_12722
+++ /data/EasyBuild-develop/modules/all/Core/gfbf/2025a.lua
@@ -26,6 +26,7 @@
 
 depends_on("FFTW/3.3.10")
 
+prepend_path("CMAKE_PREFIX_PATH", root)
 setenv("EBROOTGFBF", root)
 setenv("EBVERSIONGFBF", "2025a")
 setenv("EBDEVELGFBF", pathJoin(root, "easybuild", "Core-gfbf-2025a-easybuild-devel"))

I need to check if this is because of this PR.

Edit: Also occurs without this PR.

@Thyre
Copy link
Collaborator Author

Thyre commented Jun 26, 2025

@boegelbot please test @ jsc-zen3
EB_ARGS="buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb gfbf-2024a.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb X11-20240607-GCCcore-13.3.0.eb --installpath=/tmp/$USER/ebpr-3801"

@boegelbot
Copy link

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3801 EB_ARGS="buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb gfbf-2024a.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb X11-20240607-GCCcore-13.3.0.eb --installpath=/tmp/$USER/ebpr-3801" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3801 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 7027

Test results coming soon (I hope)...

Details

- notification for comment with ID 3010092357 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS buildenv-default-foss-2023b.eb
  • SUCCESS OpenSSL-3.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
  • SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
  • SUCCESS gfbf-2024a.eb
  • SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
  • FAIL (build issue) jiter-0.4.1-GCCcore-12.3.0.eb (partial log available at https://gist.github.com/boegelbot/23eae1b96e9c78a901ed7f2445dac54b)
  • SUCCESS X11-20240607-GCCcore-13.3.0.eb

Build succeeded for 7 out of 8 (8 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/7f59db027b137d50cb4c31b2f1609f80 for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Jun 27, 2025

@boegelbot please test @ jsc-zen3
EB_ARGS="setuptools-rust-1.6.0-GCCcore-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb --installpath=/tmp/$USER/ebpr-3801"

@boegelbot
Copy link

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3801 EB_ARGS="setuptools-rust-1.6.0-GCCcore-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb --installpath=/tmp/$USER/ebpr-3801" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3801 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 7029

Test results coming soon (I hope)...

Details

- notification for comment with ID 3011700918 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

Build succeeded for 1 out of 2 (2 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/44b246ced2e741bede9f59b17767a2de for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Jun 27, 2025

Test report by boegelbot

Overview of tested easyconfigs (in order)

* **SUCCESS** _setuptools-rust-1.6.0-GCCcore-12.3.0.eb_

* **FAIL (build issue)** _jiter-0.4.1-GCCcore-12.3.0.eb_ (partial log available at https://gist.github.com/boegelbot/47da64ad3ca316b6d043b21cb72887b4)

Build succeeded for 1 out of 2 (2 easyconfigs in total) jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21 See https://gist.github.com/boegelbot/44b246ced2e741bede9f59b17767a2de for a full test report.

Where is setuptools-rust 1.11.1 requires semantic-version, which is not installed. coming from? I don't see any EasyConfig using this version.

I'll cross-check on jsc-zen3 to see if this is somehow introduced by this PR.

@Thyre
Copy link
Collaborator Author

Thyre commented Jun 27, 2025

Where is setuptools-rust 1.11.1 requires semantic-version, which is not installed. coming from? I don't see any EasyConfig using this version.

I'll cross-check on jsc-zen3 to see if this is somehow introduced by this PR.

Python/3.11.3 introduces the pip check error:

[reuter1@jsczen3l1 ~]$ module list
No modules loaded
[reuter1@jsczen3l1 ~]$ pip check
No broken requirements found.
[reuter1@jsczen3l1 ~]$ module load Python/3.11.3
[reuter1@jsczen3l1 ~]$ module list

Currently Loaded Modules:
  1) GCCcore/12.3.0               3) binutils/2.40-GCCcore-12.3.0   5) ncurses/6.4-GCCcore-12.3.0       7) Tcl/8.6.13-GCCcore-12.3.0      9) XZ/5.4.2-GCCcore-12.3.0      11) OpenSSL/1.1
  2) zlib/1.2.13-GCCcore-12.3.0   4) bzip2/1.0.8-GCCcore-12.3.0     6) libreadline/8.2-GCCcore-12.3.0   8) SQLite/3.42.0-GCCcore-12.3.0  10) libffi/3.4.4-GCCcore-12.3.0  12) Python/3.11.3-GCCcore-12.3.0



[reuter1@jsczen3l1 ~]$ pip list
Package           Version
----------------- -------
flann             1.9.2
flann             1.9.2
flit_core         3.9.0
packaging         23.1
pip               23.1.2
setuptools        67.7.2
setuptools-rust   1.11.1
setuptools-scm    7.1.0
tomli             2.0.1
typing_extensions 4.6.3
wheel             0.40.0

[notice] A new release of pip is available: 23.1.2 -> 25.1.1
[notice] To update, run: pip install --upgrade pip
[reuter1@jsczen3l1 ~]$ pip check
setuptools-rust 1.11.1 requires semantic-version, which is not installed.

I haven't touched Python-3.11.3-GCCcore-12.3.0.eb here, so I'd guess that this error is completely unrelated to this PR.

@Thyre
Copy link
Collaborator Author

Thyre commented Jun 27, 2025

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS setuptools-rust-1.6.0-GCCcore-12.3.0.eb
  • SUCCESS jiter-0.4.1-GCCcore-12.3.0.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3
See https://gist.github.com/Thyre/99394fc1d8bca95199af8a155ff3d954 for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Jun 27, 2025

@boegelbot please test @ jsc-zen3
EB_ARGS="Python-3.11.3-GCCcore-12.3.0.eb setuptools-rust-1.6.0-GCCcore-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb --installpath=/tmp/$USER/ebpr-3801"

@boegelbot
Copy link

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3801 EB_ARGS="Python-3.11.3-GCCcore-12.3.0.eb setuptools-rust-1.6.0-GCCcore-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb --installpath=/tmp/$USER/ebpr-3801" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3801 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 7032

Test results coming soon (I hope)...

Details

- notification for comment with ID 3012235149 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS Python-3.11.3-GCCcore-12.3.0.eb
  • SUCCESS setuptools-rust-1.6.0-GCCcore-12.3.0.eb
  • SUCCESS jiter-0.4.1-GCCcore-12.3.0.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/d24a018660d1d8210bfacbc7ccc6cf6b for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Jun 27, 2025

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS buildenv-default-foss-2023b.eb
  • SUCCESS OpenSSL-3.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
  • SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
  • SUCCESS gfbf-2024a.eb
  • SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
  • SUCCESS jiter-0.4.1-GCCcore-12.3.0.eb
  • SUCCESS X11-20240607-GCCcore-13.3.0.eb

Build succeeded for 8 out of 8 (8 easyconfigs in total)
datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3
See https://gist.github.com/Thyre/dc26f15cf5eb27a565eb8576e9d8856f for a full test report.

@boegel boegel added the bug fix label Jul 2, 2025
@boegel boegel added this to the 5.1.1 milestone Jul 2, 2025
@boegel boegel changed the title Bundle: Add missing call to make_module_step for components add missing call to make_module_step for components in Bundle easyblock Jul 2, 2025
@ocaisa
Copy link
Member

ocaisa commented Jul 2, 2025

@boegelbot please test @ jsc-zen3
EB_ARGS="purge_dups-1.2.6-foss-2023b.eb FlexiBLAS-3.4.5-GCC-14.2.0.eb matplotlib-3.8.2-gfbf-2023b.eb --installpath /tmp/$USER/ebpr-3775"

@boegelbot
Copy link

@ocaisa: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3801 EB_ARGS="purge_dups-1.2.6-foss-2023b.eb FlexiBLAS-3.4.5-GCC-14.2.0.eb matplotlib-3.8.2-gfbf-2023b.eb --installpath /tmp/$USER/ebpr-3775" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3801 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 7088

Test results coming soon (I hope)...

Details

- notification for comment with ID 3028000072 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

Copy link
Member

@ocaisa ocaisa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Certainly in principle these changes look fine to me, tests have not thrown up any problems AFAICT

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS FlexiBLAS-3.4.5-GCC-14.2.0.eb
  • SUCCESS matplotlib-3.8.2-gfbf-2023b.eb
  • SUCCESS purge_dups-1.2.6-foss-2023b.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
jsczen3c4.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/a540d2d4966ceb1c7540e179447b3eaf for a full test report.

@boegel
Copy link
Member

boegel commented Jul 3, 2025

Thanks a lot for figuring this out @Thyre!

@boegel boegel merged commit 6c750af into easybuilders:develop Jul 3, 2025
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bundle EasyBlock once again doesn't set PATH correctly for Bundle components

4 participants