Skip to content

fix fake_module_environment context manager: make sure that cleanup of fake module is always done, even if an error was raised #4895

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 26, 2025

Conversation

boegel
Copy link
Member

@boegel boegel commented May 26, 2025

This fixes a problem when --parallel-extensions-install is used when extensions are involved that can't actually be installed in parallel, like with Perl-5.34.1-GCCcore-11.3.0.eb:

$ eb Perl-5.34.1-GCCcore-11.3.0.eb --parallel-extensions-install
== installing extension File::Listing 6.15 (2/383)...
== ... (took 1 min 8 secs)
== FAILED: Installation ended unsuccessfully: Module command '/usr/share/lmod/lmod/libexec/lmod python load Perl/5.34.1-GCCcore-11.3.0' failed with exit code 1; stderr: Lmod has detected the following error: These module(s) or extension(s) exist but cannot be loaded as requested: "Perl/5.34.1-GCCcore-11.3.0"
   Try: "module spider Perl/5.34.1-GCCcore-11.3.0" to see how to load the module(s).

If you don't understand the warning or error, contact the helpdesk at [email protected]

EasyBuild will naively always try to install extensions in parallel when the --parallel-extensions-install configuration setting is enabled, and only give up on that when a NotImplementedError is raised; see EasyBlock.install_all_extensions.
A NotImplementedError will be raised by the install_extension_async method for extensions other than R packages.

We should probably look into changing that, and only start installing extensions in parallel (via EasyBlock.install_extensions_parallel) if all extensions that should be installed actually support it. If not, the extensions should just be installed sequentially (via EasyBlock.install_extensions_sequential).
Doing that correctly requires additional work though, probably an extra method in Extension that declares whether installing in parallel is supported, or not.

The acute problem here is that this leaves the environment in an inconsistent state, because the fake module that was loaded when the asynchronous installation of the 1st extension was being launched will not be cleaned up.

This makes Lmod quite unhappy as soon as an attempt is made to prepare the environment for the installation of the 2nd extension, see error above.

This fix in this PR makes sure that the fake module that was loaded is correctly cleaned up, and then there's no problem when the extensions are being installed.

edit: opened two issues to follow-up on things that were uncovered while working out this fix:

…f fake module is always done, even if an error was raised
@boegel boegel added this to the next release (5.1.0) milestone May 26, 2025
@boegel boegel added the bug fix label May 26, 2025
Copy link
Contributor

@lexming lexming left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lexming lexming enabled auto-merge May 26, 2025 08:26
@lexming lexming merged commit 8e6079e into easybuilders:develop May 26, 2025
40 of 57 checks passed
@boegel boegel deleted the fix_fake_module_environment branch May 26, 2025 10:38
boegel added a commit to boegel/easybuild-framework that referenced this pull request May 26, 2025
boegel added a commit to boegel/easybuild-framework that referenced this pull request May 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants