Skip to content

take into account maxparallel for individual extensions #3811

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
boegel wants to merge 5 commits into
base: develop
Choose a base branch
from
Open

take into account maxparallel for individual extensions #3811

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7738d1b
take into account maxparallel for individual extensions
boegel Aug 30, 2021
45371a9
don't compare parallel vs maxparallel again when taking maxparallel i…
boegel Aug 30, 2021
bb6a606
Merge branch 'develop' into ext_maxparallel
boegel Aug 31, 2021
50030cd
Merge branch 'ext_maxparallel' of github.com:boegel/easybuild-framewo…
boegel Aug 31, 2021
57b053e
introduce get_parallel_ec_param_value function to determine value for…
boegel Aug 31, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions easybuild/framework/extension.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,10 @@ def __init__(self, mself, ext, extra_params=None):
self.log.debug("Skipping unknown custom easyconfig parameter '%s' for extension %s/%s: %s",
key, name, version, value)

if self.cfg['maxparallel'] and self.cfg['parallel'] > self.cfg['maxparallel']:
self.cfg['parallel'] = self.cfg['maxparallel']
self.log.info("Setting parallelism to %d for extension %s", self.cfg['parallel'], name)

self.sanity_check_fail_msgs = []

@property
Expand Down
16 changes: 16 additions & 0 deletions test/framework/easyblock.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1025,10 +1025,15 @@ def test_init_extensions(self):

test_ec = os.path.join(self.test_prefix, 'test.eb')
test_ec_txt = toy_ec_txt.replace("('barbar', '0.0', {", "('barbar', '0.0', {'easyblock': 'DummyExtension',")
test_ec_txt = test_ec_txt.replace("('barbar', '0.0', {", "('barbar', '0.0', {'maxparallel': 3,")
Copy link
Contributor

@Flamefire Flamefire Sep 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this changing the same as the line above? Maybe combine both into one? This looks like a C&P mistake

test_ec_txt += "\nparallel = 5"
write_file(test_ec, test_ec_txt)
ec = process_easyconfig(test_ec)[0]
eb = get_easyblock_instance(ec)

# require to trigger setting of 'parallel' value
Copy link
Contributor

@akesandgren akesandgren Aug 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"required"

Copy link
Contributor

@Flamefire Flamefire Sep 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you move that setting of orig-parallel to the ctor this is not really required anymore, unless you add a top-level maxparallel and test the mentioned case of an extension having a higher value for it and it should still work.

E.g. instead of the EC parallel (why do we have that anyway? Isn't that basically the same as maxparallel?) use a maxparallel of 2 and a build option parallel of 5, then barbar should still yield in a parallel value of 3

eb.check_readiness_step()

eb.prepare_for_extensions()
eb.init_ext_instances()
ext_inst_class_names = [x.__class__.__name__ for x in eb.ext_instances]
Expand All @@ -1040,6 +1045,17 @@ def test_init_extensions(self):
]
self.assertEqual(ext_inst_class_names, expected)

# default parallel is inherited from parent
self.assertEqual(eb.cfg['parallel'], 5)
self.assertEqual(eb.ext_instances[0].cfg['parallel'], 5)
self.assertEqual(eb.ext_instances[1].cfg['parallel'], 5)
self.assertEqual(eb.ext_instances[3].cfg['parallel'], 5)

barbar_ext = eb.ext_instances[2]
self.assertEqual(barbar_ext.name, 'barbar')
# parallel should be set to 3 for barbar, due to maxparallel
self.assertEqual(barbar_ext.cfg['parallel'], 3)

# check what happen if we specify an easyblock that doesn't derive from Extension,
# and hence can't be used to install extensions...
test_ec = os.path.join(self.test_prefix, 'test_broken.eb')
Expand Down