Subtoolchain searching #1306

ocaisa · 2015-06-26T13:57:29Z

Add support for automatically matching the lowest possible member of a toolchain hierarchy when resolving dependencies using a command line option --minimal-toolchains.
Requires a SUBTOOLCHAIN to be defined for each level in the hierarchy (see examples inside)

WIP @boegel ... please review
FYI @geimer

… problems, in particular it assumes that the ec's exist in top-level toolchain and then replaces them with subtoolchains, if they don't then it'll barf.

use minimally_resolve_dependencies from main.py

…framework into subtoolchain_searching

remove unused build_specs

…framework into subtoolchain_searching Conflicts: easybuild/tools/robot.py

make EasyConfig instances comparable and hashable

…framework into subtoolchain_searching

construct subtoolchains dict via search_toolchain, use nub

…sidering dummy toolchain

…ting-modules to --use-existing-modules

…oolchains

boegel · 2015-11-08T16:04:33Z

@ocaisa: with ocaisa#22 included, I think this is good to go in (but I need to take another close look once that PR is included here)

I also wrote up some documentation for this, see ocaisa#22 .

boegel · 2015-11-08T16:05:49Z

@ocaisa: also, there are some minor conflicts in easybuild/toolchains/ipsmpi.py and easybuild/toolchains/gpsmpi.py, can you look into updating your branch with current develop and fixing them?

…caisa/easybuild-framework into subtoolchain_searching Conflicts: easybuild/toolchains/gpsmpi.py easybuild/toolchains/ipsmpi.py

require --experimental for and add unit tests and fix bugs in functions supporting --minimal-toolchains

hpcugentbot · 2015-11-08T21:21:20Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2302/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

hpcugentbot · 2015-11-08T22:24:29Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2303/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

define SUBTOOLCHAIN for toolchains where it wasn't yet

boegel · 2015-11-09T16:20:16Z

easybuild/tools/options.py

@@ -244,6 +246,8 @@ def override_options(self):
                      None, 'store', None),
            'update-modules-tool-cache': ("Update modules tool cache file(s) after generating module file",
                                          None, 'store_true', False),
+            'use-existing_modules': ("Use existing modules when resolving dependencies with minimal toolchains",


crap, typo here, _ should be -

fix typo

hpcugentbot · 2015-11-09T16:24:22Z

EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2307/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

edit: test suite run cancelled by @boegel since another one is following it

hpcugentbot · 2015-11-09T18:00:10Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2308/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

boegel · 2015-11-09T18:01:37Z

Going in after extensive testing by rebuilding all HPL easyconfigs, right in time for EasyBuild v2.4.0.

Thanks @ocaisa!

Subtoolchain searching

Alan and others added 30 commits June 17, 2015 15:14

Added SUBTOOLCHAIN names to intel-para toolchain

9683cda

First effort at searching by subtoolchains...unfinished but number of…

49e846b

… problems, in particular it assumes that the ec's exist in top-level toolchain and then replaces them with subtoolchains, if they don't then it'll barf.

Bit more added to replace_toolchain_with_hierarchy

ff4201c

use minimally_resolve_dependencies from main.py

f4f6dd3

Merge pull request #3 from boegel/subtoolchain_searching

aed1020

use minimally_resolve_dependencies from main.py

Some comments

90edc95

Merge branch 'subtoolchain_searching' of github.com:ocaisa/easybuild-…

819ef97

…framework into subtoolchain_searching

remove unused build_specs

9eb446d

Merge pull request #4 from boegel/subtoolchain_searching

53252e0

remove unused build_specs

More commenting

b232830

Merge branch 'subtoolchain_searching' of github.com:ocaisa/easybuild-…

b9f94ee

…framework into subtoolchain_searching Conflicts: easybuild/tools/robot.py

make EasyConfig instances comparable and hashable

5878231

Merge pull request #5 from boegel/subtoolchain_searching

23c224f

make EasyConfig instances comparable and hashable

Start building the hierarchy

40655cc

Merge branch 'subtoolchain_searching' of github.com:ocaisa/easybuild-…

aaa9fb7

…framework into subtoolchain_searching

More edits for hierarchy

b1a56c1

construct subtoolchains dict via search_toolchain, use nub

76c8236

Merge pull request #6 from boegel/subtoolchain_searching

858fdb6

construct subtoolchains dict via search_toolchain, use nub

Complete initial subtoolchain support

e7ebc15

Update comment

0ae53db

Added some resilience and further checking

36a420a

Fixed missing logic in if statement

8301d54

Fixed bad logic for subtoolchains

5568232

More logic

488948f

Check that everything appears in the final list

47d5e85

Final logic...now to test

4b91fe3

Edit

02816d8

Fixed some errors

fcfeb1d

Fixed some bugs

f70ccbd

Bugs

93c15ef

boegel added 7 commits November 7, 2015 15:39

add extra unit test for find_minimally_resolved_modules with also con…

5581adc

…sidering dummy toolchain

also define full_mod_name and short_mod_name keys in toolchain dict

186d7b5

fix issues in find_minimally_resolved_modules + rename --use-any-exis…

ec64a75

…ting-modules to --use-existing-modules

add new test easyconfig required by tests checking enabling minimal t…

111ed29

…oolchains

reverse order of get_toolchain_hierarchy result

aec6b0b

require use of --experimental in find_minimally_resolved_modules

dbbf611

allow use of experimental code in robot tests

07a052e

boegel mentioned this pull request Nov 8, 2015

require --experimental for and add unit tests and fix bugs in functions supporting --minimal-toolchains ocaisa/easybuild-framework#22

Merged

Alan added 2 commits November 8, 2015 18:59

Merge branches 'develop' and 'subtoolchain_searching' of github.com:o…

e5832b2

…caisa/easybuild-framework into subtoolchain_searching Conflicts: easybuild/toolchains/gpsmpi.py easybuild/toolchains/ipsmpi.py

Merge pull request #22 from boegel/subtoolchain_searching

0d6f431

require --experimental for and add unit tests and fix bugs in functions supporting --minimal-toolchains

define SUBTOOLCHAIN for toolchains where it wasn't yet

cd898d7

boegel mentioned this pull request Nov 9, 2015

define SUBTOOLCHAIN for toolchains where it wasn't yet ocaisa/easybuild-framework#23

Merged

boegel and others added 2 commits November 9, 2015 17:14

use GCC as subtoolchain for gcccuda toolchain

df90823

Merge pull request #23 from boegel/subtoolchain_searching

6e752a5

define SUBTOOLCHAIN for toolchains where it wasn't yet

boegel reviewed Nov 9, 2015
View reviewed changes

fix typo

89fcbed

boegel mentioned this pull request Nov 9, 2015

fix typo ocaisa/easybuild-framework#24

Merged

Merge pull request #24 from boegel/subtoolchain_searching

f8f1bb8

fix typo

boegel mentioned this pull request Nov 9, 2015

document experimental support for minimal toolchains easybuilders/easybuild#166

Merged

boegel added a commit that referenced this pull request Nov 9, 2015

Merge pull request #1306 from ocaisa/subtoolchain_searching

316a37a

Subtoolchain searching

boegel merged commit 316a37a into easybuilders:develop Nov 9, 2015

ocaisa deleted the subtoolchain_searching branch November 19, 2015 15:13

boegel mentioned this pull request Jan 23, 2016

also try and resolve dependencies using subtoolchains #741

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Subtoolchain searching #1306

Subtoolchain searching #1306

Uh oh!

ocaisa commented Jun 26, 2015

Uh oh!

boegel commented Nov 8, 2015

Uh oh!

boegel commented Nov 8, 2015

Uh oh!

hpcugentbot commented Nov 8, 2015

Uh oh!

hpcugentbot commented Nov 8, 2015

Uh oh!

boegel Nov 9, 2015

Uh oh!

hpcugentbot commented Nov 9, 2015

Uh oh!

hpcugentbot commented Nov 9, 2015

Uh oh!

boegel commented Nov 9, 2015

Uh oh!

Uh oh!

Subtoolchain searching #1306

Subtoolchain searching #1306

Uh oh!

Conversation

ocaisa commented Jun 26, 2015

Uh oh!

boegel commented Nov 8, 2015

Uh oh!

boegel commented Nov 8, 2015

Uh oh!

hpcugentbot commented Nov 8, 2015

Uh oh!

hpcugentbot commented Nov 8, 2015

Uh oh!

boegel Nov 9, 2015

Choose a reason for hiding this comment

Uh oh!

hpcugentbot commented Nov 9, 2015

Uh oh!

hpcugentbot commented Nov 9, 2015

Uh oh!

boegel commented Nov 9, 2015

Uh oh!

Uh oh!