adds support for including 'extensions' statement in Lua modules with Lmod 8.2+ #3085

wpoely86 · 2019-11-15T09:23:35Z

Lmod 8.2 supports now extensions, this PR makes EB generate the needed lines in the module files for it.

* develop: (413 commits) bump version to 4.0.2dev tweak release notes and bump version for v4.0.1 release prepare release notes for eb401 extend test_build_easyconfigs_in_parallel_gc3pie to also check for raised error when a job failed improve error message for jobs that failed when using GC3Pie job backend appease the Hound fix join for Tcl module syntax in test_toy_ghost_installdir make removal of ghost installation directories opt-in remove installation directory used by module file that is replaced under --force, to avoid leaving behind a 'ghost' installation add ModuleGenerator.det_installdir method to determine installation directory used by a given module file use remove_dir + move_file functions in EasyBlock.make_dir Fix wrong function invocation in last commit. Report what build jobs actually failed. fix broken test for get_paths_for appease the Hound only consider resolved path to 'eb' script if desired subdir is not found relative to 'eb' script location in get_paths_for enhance test for get_paths_for define $EB_SCRIPT_PATH in 'eb' wrapper script, and consider it before location of 'eb' determined via $PATH in get_paths_for function Error out if some GC3Pie job failed. better filtering + logging in test for move_logs ...

* origin/develop: (74 commits) always include mandatory easyconfig parameters in dumped easyconfig file, regardless of whether default value is used or not add support for is_mandatory_key method in EasyConfig class silence deprecation warning for Python 2.6 when testing with Python 2.6 also test with Lmod 8.2.3 in GitHub CI ensure 'silence_deprecation_warnings' build option is correctly set in tests correctly silence deprecation warning for Lmod 6.x in Travis CI config add support for --silence-deprecation-warnings, and use it to avoid deprecation warnings breaking tests when testing with Lmod 6.x update help message for --external-modules-metadata fix trivial code style issue fix module version cache check keep testing single configuration in Travis CI with Lmod 6.5.1 don't test non-Lmod module tools with Python 3.8 also test with Python 3.8 in Travis CI take into account that platform.dist was removed in Python 3.8 in get_os_version, leverage distro package if available deprecate running EasyBuild with Python 2.6 via new check_python_version() function stop testing with Lmod 6.x in GitHub Actions stop testing with Lmod 6.x in Travis CI, update to Lmod 8.2.3, favor testing with Python 2.7 over 2.6 (+ cleanup) fix broken test_lmod_specific, use DEPR_VERSION rather than REQ_VERSION take into account that platform.linux_distribution was removed in Python 3.8 in get_os_name, leverage distro package if available deprecate support for using Lmod 6.x ...

boegel · 2019-11-15T10:25:25Z

easybuild/tools/module_generator.py

+        provide_list = self._generate_provides_list()
+        if self.modules_tool.supports_extensions and provide_list:
+            provide_list_str = ', '.join(['"%s"' % x for x in provide_list])
+            lines.extend(['', 'extensions(%s)' % provide_list_str])


We have two options here:

either we include this if the Lmod version used at installation time is recent enough (which is what you're doing here)

or we add a conditional in the module file itself, using LmodVersion() (that's what @rtmclay recommended)

The benefit of the latter approach is that the module files will still work when loaded with an older Lmod version, and there's basically no overhead in using the conditional.

It's a design choice, I don't have a strong preference for either.

But adding support in a module file for something that doesn't work with the current Lmod seems a bit weird to me? Especially as it's pretty easy to regenerate a module.

Also, it might surprise people. You update Lmod and suddenly the output of av etc changes

It's a protection mechanism too though. You may be installing software on a system with a slightly more recent version of Lmod (8.2+), and then consume the modules elsewhere with a slightly older Lmod (<8.2).
That wouldn't be very nice either... It's certainly less serious than being a bit careful and only using extensions conditionally.

Output of ml av will change anyway as soon as you install something with extensions after updating to Lmod 8.2, and you're probably not going to notice straight away (I certainly rarely check the generated module files mannually ;) ).

boegel · 2019-11-17T15:22:53Z

@wpoely86 Tests are still broken when using Lmod 8.2+ (@boegelbot only reports broken tests from Travis CI).

======================================================================
FAIL: test_module_extensions (test.framework.module_generator.LuaModuleGeneratorTest)
test the extensions() for extensions
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/framework/module_generator.py", line 585, in test_module_extensions
    self.assertTrue(re.search(pattern, desc, re.M), "Pattern '%s' found in: %s" % (pattern, desc))
AssertionError: None is not true : Pattern '^extensions\("bar/0.0", "barbar/0.0", "l/s", "toy/0.0"\)' found in: help([==[

Description
===========
Toy C program, 100% toy.

More information
================
 - Homepage: https://easybuilders.github.io/easybuild


Included extensions
===================
bar-0.0, barbar-0.0, l-s, toy-0.0
]==])

whatis([==[Description: Toy C program, 100% toy.]==])
whatis([==[Homepage: https://easybuilders.github.io/easybuild]==])
whatis([==[URL: https://easybuilders.github.io/easybuild]==])
whatis([==[Extensions: bar-0.0, barbar-0.0, l-s, toy-0.0]==])

local root = "/tmp/eb-CRdgXt/eb-Qajkhe/eb-FkUxoO/tmp9r3vc4/software/toy/0.0-gompi-2018a-test"

conflict("toy")

if convertToCanonical(LmodVersion()) > convertToCanonical("8.2.0") then
    extensions("bar/0.0", "barbar/0.0", "l/s", "toy/0.0")
end

 ----------------------------------------------------------------------
Ran 653 tests in 874.006s

FAILED (failures=1)
ERROR: Not all tests were successful.

We need a Lmod version check and it's simple a complete pain in Tcl.

boegel

@wpoely86 Some minor suggestions, it's pretty much good to go imho...

easybuild/tools/module_generator.py

…o modprovides * 'modprovides' of github:wpoely86/easybuild-framework: drop forgotten print statement

minor style fixes in module_generator.py w.r.t. extensions support

wpoely86 added 14 commits June 21, 2019 14:36

Add module_provides for extensions

d4bded7

Rework module_provides

3cf19f1

fix extensions in moduletool

ff92bb0

only add extensions for Lmod 8

5267f52

use correct Lmod version for extensions

400bce2

add first test for extensions

6f3deea

add empty lines for extensions

5e6b041

properly handle the case if extensions are supported

eb1499a

fix tests

a68d917

add all extensions in unittest

2bb070a

upgrade git workflow test for Lmod from 8.1.x to 8.2.x

8f244fc

fix tcl syntax of extensions

c3d29b7

wpoely86 requested a review from boegel November 15, 2019 09:23

fix tests: excape ( ) in regex

077c685

boegel reviewed Nov 15, 2019

View reviewed changes

wpoely86 added 3 commits November 15, 2019 17:54

add extensions always in Lua modules with Lmod version guard

d9984f1

fix typo

653c6ce

actually use the conditional statement

399bbd2

boegel added the enhancement label Nov 16, 2019

boegel added this to the next release (4.1.0) milestone Nov 16, 2019

easybuilders deleted a comment from boegelbot Nov 16, 2019

wpoely86 added 3 commits November 22, 2019 17:34

Limit extensions to lua modules only

9bebaae

We need a Lmod version check and it's simple a complete pain in Tcl.

drop unused changes

342d30b

drop forgotten print statement

7bb4838

boegel requested changes Nov 28, 2019

View reviewed changes

easybuilders deleted a comment from boegelbot Nov 29, 2019

fix remarks

44c0e8a

wpoely86 and others added 2 commits November 29, 2019 17:39

Merge branch 'modprovides' of github:wpoely86/easybuild-framework int…

6e2726f

…o modprovides * 'modprovides' of github:wpoely86/easybuild-framework: drop forgotten print statement

minor style fixes in module_generator.py w.r.t. extensions support

b3d2bac

boegel changed the title ~~Adds support for extensions in Lmod modules~~ adds support for including 'extensions' statement in Lua modules with Lmod 8.2+ Nov 29, 2019

Merge pull request #23 from boegel/modprovides

07e58f6

minor style fixes in module_generator.py w.r.t. extensions support

boegel approved these changes Nov 29, 2019

View reviewed changes

boegel merged commit 74a4c03 into easybuilders:develop Nov 29, 2019

boegel mentioned this pull request Nov 30, 2019

make including 'extensions' statement in generated Lua module files opt-in for now #3107

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

adds support for including 'extensions' statement in Lua modules with Lmod 8.2+ #3085

adds support for including 'extensions' statement in Lua modules with Lmod 8.2+ #3085

Uh oh!

wpoely86 commented Nov 15, 2019

Uh oh!

boegel Nov 15, 2019 •

edited

Loading

Uh oh!

wpoely86 Nov 15, 2019

Uh oh!

wpoely86 Nov 15, 2019

Uh oh!

boegel Nov 15, 2019

Uh oh!

boegel commented Nov 17, 2019

Uh oh!

boegel left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

adds support for including 'extensions' statement in Lua modules with Lmod 8.2+ #3085

adds support for including 'extensions' statement in Lua modules with Lmod 8.2+ #3085

Uh oh!

Conversation

wpoely86 commented Nov 15, 2019

Uh oh!

boegel Nov 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wpoely86 Nov 15, 2019

Choose a reason for hiding this comment

Uh oh!

wpoely86 Nov 15, 2019

Choose a reason for hiding this comment

Uh oh!

boegel Nov 15, 2019

Choose a reason for hiding this comment

Uh oh!

boegel commented Nov 17, 2019

Uh oh!

boegel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

boegel Nov 15, 2019 •

edited

Loading