Skip to content

A simplified version of minimal toolchains that does the toolchain re… #1614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
boegel merged 22 commits into from
Feb 17, 2016
Merged

A simplified version of minimal toolchains that does the toolchain re… #1614

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
f454c22
A simplified version of minimal toolchains that does the toolchain re…
Feb 15, 2016
a224028
Remove old implementation
Feb 15, 2016
dc5c739
Tidy up a bit
Feb 15, 2016
5fcc121
Remove test
Feb 15, 2016
5ea7c77
Update tests
Feb 15, 2016
c3c276e
Update main
Feb 15, 2016
d38a984
Add #1576 to be more thorough when constructing the hierarchy
Feb 16, 2016
69d9308
Clean up a bit
Feb 16, 2016
5924be5
Fix some tests
Feb 16, 2016
793a0b4
Replace existing tests
Feb 16, 2016
09a19f4
Bug fixes
Feb 16, 2016
13617e3
Tidy up a bit
Feb 16, 2016
ef278e4
Add a new easyconfig for some tests
Feb 16, 2016
2dbd868
Tidy up a bit
Feb 16, 2016
1181e98
Typo
Feb 16, 2016
dc2312f
fix mocking + clear cache to fix broken test
boegel Feb 16, 2016
eb8d8ee
Merge pull request #29 from boegel/improved_minimal_toolchains
Feb 16, 2016
1a4e188
some refactoring and style fixes
boegel Feb 17, 2016
861c88a
Merge pull request #30 from boegel/improved_minimal_toolchains
Feb 17, 2016
ac7f8b7
Dump the parsed easyconfig to a reproducability directory and also to…
Feb 17, 2016
eb26475
refactor dumping easyconfig in reprod
boegel Feb 17, 2016
e6d2cd5
Merge pull request #31 from boegel/improved_minimal_toolchains
Feb 17, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
155 changes: 151 additions & 4 deletions easybuild/framework/easyconfig/easyconfig.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,11 @@
from easybuild.tools.module_naming_scheme import DEVEL_MODULE_SUFFIX
from easybuild.tools.module_naming_scheme.utilities import avail_module_naming_schemes, det_full_ec_version
from easybuild.tools.module_naming_scheme.utilities import det_hidden_modname, is_valid_module_name
from easybuild.tools.modules import get_software_root_env_var_name, get_software_version_env_var_name
from easybuild.tools.modules import get_software_root_env_var_name, get_software_version_env_var_name, modules_tool
from easybuild.tools.ordereddict import OrderedDict
from easybuild.tools.systemtools import check_os_dependency
from easybuild.tools.toolchain import DUMMY_TOOLCHAIN_NAME, DUMMY_TOOLCHAIN_VERSION
from easybuild.tools.toolchain.utilities import get_toolchain
from easybuild.tools.toolchain.utilities import get_toolchain, search_toolchain
from easybuild.tools.utilities import quote_py_str, quote_str, remove_unwanted_chars
from easybuild.framework.easyconfig import MANDATORY
from easybuild.framework.easyconfig.constants import EXTERNAL_MODULE_MARKER
Expand All @@ -66,7 +66,7 @@
from easybuild.framework.easyconfig.parser import DEPRECATED_PARAMETERS, REPLACED_PARAMETERS
from easybuild.framework.easyconfig.parser import EasyConfigParser, fetch_parameters_from_easyconfig
from easybuild.framework.easyconfig.templates import TEMPLATE_CONSTANTS, template_constant_dict

from easybuild.toolchains.gcccore import GCCcore
Copy link
Member

@boegel boegel Feb 17, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please keep imports alphabetically sorted

Copy link
Member Author

@ocaisa ocaisa Feb 17, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You say that, but I'm pretty sure f comes before t in the alphabet...

Copy link
Member

@boegel boegel Feb 17, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops :)


_log = fancylogger.getLogger('easyconfig.easyconfig', fname=False)

Expand Down Expand Up @@ -105,6 +105,109 @@ def new_ec_method(self, key, *args, **kwargs):
return new_ec_method


def module_is_available(full_mod_name, modtool, avail_modules, hidden):
Copy link
Member

@boegel boegel Feb 15, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should probably move this inside robot_find_minimal_toolchain_of_dependency as an inner function,

or just simply it to this in robot_find_minimal_toolchain_of_dependency:

module_exists = (full_mod_name in avail_modules) or (hidden and modtool.exist([full_mod_name])[0])

Copy link
Member Author

@ocaisa ocaisa Feb 15, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is needed by tools.py

Copy link
Member

@boegel boegel Feb 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, you can replace it with the same single line there too?

having a function like this here is kind of weird

if you want to keep it, it should be moved to easybuild.tools.modules instead

"""
Check whether specified module is available.

@param full_mod_name: full module name
@param modtool: ModulesTool instance that can be used to check whether (hidden) modules exist
@param avail_modules: list of available modules
@param hidden: hidden module
"""
resolved = full_mod_name in avail_modules
# hidden modules need special care, since they may not be included in list of available modules
if hidden:
resolved |= modtool.exist([full_mod_name])[0]
return resolved

def toolchain_hierarchy_cache(func):
"""Function decorator to cache (and retrieve cached) toolchain hierarchy queries."""
cache = {}

def cache_aware_func(toolchain):
"""Look up toolchain hierarchy in cache first, determine and cache it if not available yet."""
cache_key = (toolchain['name'], toolchain['version'])

# fetch from cache if available, cache it if it's not
if cache_key in cache:
_log.debug("Using cache to return hierarchy for toolchain %s: %s", str(toolchain), cache[cache_key])
return cache[cache_key]
else:
toolchain_hierarchy = func(toolchain)
cache[cache_key] = toolchain_hierarchy
return cache[cache_key]

return cache_aware_func


@toolchain_hierarchy_cache
def get_toolchain_hierarchy(parent_toolchain):
"""
Determine list of subtoolchains for specified parent toolchain.
Result starts with the most minimal subtoolchains first, ends with specified toolchain.

The dummy toolchain is considered the most minimal subtoolchain only if the add_dummy_to_minimal_toolchains
build option is enabled.

@param parent_toolchain: dictionary with name/version of parent toolchain
"""
# obtain list of all possible subtoolchains
_, all_tc_classes = search_toolchain('')
subtoolchains = dict((tc_class.NAME, getattr(tc_class, 'SUBTOOLCHAIN', None)) for tc_class in all_tc_classes)

current_tc_name, current_tc_version = parent_toolchain['name'], parent_toolchain['version']
subtoolchain_name, subtoolchain_version = subtoolchains[current_tc_name], None

# the parent toolchain is at the top of the hierarchy
toolchain_hierarchy = [parent_toolchain]

while subtoolchain_name:
# grab the easyconfig of the current toolchain and search the dependencies for a version of the subtoolchain
path = robot_find_easyconfig(current_tc_name, current_tc_version)
if path is None:
raise EasyBuildError("Could not find easyconfig for %(name)s toolchain version %(version)s",
current_tc_name, current_tc_version)

# parse the easyconfig
parsed_ec = process_easyconfig(path)[0]
# search the dependencies for the version of the subtoolchain
dep_tcs = [dep_toolchain['toolchain'] for dep_toolchain in parsed_ec['dependencies']
if dep_toolchain['toolchain']['name'] == subtoolchain_name]
unique_dep_tc_versions = set([dep_tc['version'] for dep_tc in dep_tcs])

if len(unique_dep_tc_versions) == 1:
subtoolchain_version = dep_tcs[0]['version']

elif len(unique_dep_tc_versions) == 0:
# only retain GCCcore as subtoolchain if version was found
if subtoolchain_name == GCCcore.NAME:
_log.info("No version found for %s; assuming legacy toolchain and skipping it as subtoolchain.",
subtoolchain_name)
subtoolchain_name = GCCcore.SUBTOOLCHAIN
subtoolchain_version = ''
# dummy toolchain: end of the line
elif subtoolchain_name == DUMMY_TOOLCHAIN_NAME:
subtoolchain_version = ''
else:
raise EasyBuildError("No version found for subtoolchain %s in dependencies of %s",
subtoolchain_name, current_tc_name)
else:
raise EasyBuildError("Multiple versions of %s found in dependencies of toolchain %s: %s",
subtoolchain_name, current_tc_name, unique_dep_tc_versions)

if subtoolchain_name == DUMMY_TOOLCHAIN_NAME and not build_option('add_dummy_to_minimal_toolchains'):
# we're done
break

# add to hierarchy and move to next
current_tc_name, current_tc_version = subtoolchain_name, subtoolchain_version
subtoolchain_name, subtoolchain_version = subtoolchains[current_tc_name], None
toolchain_hierarchy.insert(0, {'name': current_tc_name, 'version': current_tc_version})

_log.info("Found toolchain hierarchy for toolchain %s: %s", parent_toolchain, toolchain_hierarchy)
return toolchain_hierarchy


class EasyConfig(object):
"""
Class which handles loading, reading, validation of easyconfigs
Expand Down Expand Up @@ -707,7 +810,8 @@ def _parse_dependency(self, dep, hidden=False, build_only=False):
raise EasyBuildError("Found toolchain spec as dict with wrong keys (no name/version): %s", tc_spec)
else:
raise EasyBuildError("Unsupported type for toolchain spec encountered: %s (%s)", tc_spec, type(tc_spec))

elif build_option('minimal_toolchains'):
Copy link
Member

@boegel boegel Feb 15, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change the logic to make this more readable:

if tc_spec is None:
    if build_option('minimal_toolchains'):
        ...
    else:
        ... # ??
else:
    # (true) boolean value simply indicates that a dummy toolchain is used
    ...

tc = robot_find_minimal_toolchain_of_dependency(dependency, tc)
Copy link
Member

@boegel boegel Feb 15, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

empty line below to separate line that jumps back one indent level

self.log.debug("Derived toolchain to use for dependency %s, based on toolchain spec %s: %s", dep, tc_spec, tc)
dependency['toolchain'] = tc

Expand Down Expand Up @@ -1137,6 +1241,49 @@ def robot_find_easyconfig(name, version):

return res

def robot_find_minimal_toolchain_of_dependency(dependency, parent_toolchain):
Copy link
Member

@boegel boegel Feb 15, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we try to use two empty lines between top-level definitions

"""
Find the minimal toolchain of a dependency

@dependency: dependency target dict (long and short module names may not exist yet)
@parent_toolchain: toolchain from which to derive the toolchain hierarchy to search
@return: minimal toolchain for which an easyconfig exists for this dependency (and matches build_options)
"""
if build_option('use_existing_modules') and not build_option('retain_all_deps'):
existing_modules = modules_tool().available()
Copy link
Member

@boegel boegel Feb 15, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sure that existing_modules is always defined (also in case of else)

modtool = modules_tool()

newdep = copy.deepcopy(dependency)
toolchain_hierarchy = get_toolchain_hierarchy(parent_toolchain)
Copy link
Member

@boegel boegel Feb 15, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shorter variable names may make this easier to read

  • dependency -> dep
  • toolchain -> tc


possible_toolchains = []
# reversed search: start with subtoolchains first, i.e. first (dummy or) compiler-only toolchain, etc.
for toolchain in toolchain_hierarchy:
newdep['toolchain'] = toolchain
eb_file = robot_find_easyconfig(newdep['name'], det_full_ec_version(newdep))
if eb_file is not None:
module_exists = False
# if necessary check if module exists
if build_option('use_existing_modules') and not build_option('retain_all_deps'):
full_mod_name = ActiveMNS().det_full_module_name(newdep)
module_exists = module_is_available(full_mod_name, modtool, existing_modules, newdep['hidden'])
# Add the toolchain to list of possibilities
possible_toolchains.append({'toolchain': toolchain, 'module_exists': module_exists})

if not possible_toolchains:
raise EasyBuildError("Irresolvable dependency found (even with minimal toolchains): %s", dependency)

# Select the toolchain to return, defaulting to the first element (lowest possible toolchain)
minimal_toolchain = possible_toolchains[0]['toolchain']
if build_option('use_existing_modules') and not build_option('retain_all_deps'):
# Take the last element in the case of using existing modules (allows optimisation)
filtered_possibilities = [toolchain for toolchain in possible_toolchains if toolchain['module_exists']]
if filtered_possibilities:
# Take the last element (the maximum toolchain where a module exists already)
minimal_toolchain = filtered_possibilities[-1]['toolchain']

_log.info("Minimally resolving dependency %s using toolchain %s", dependency, toolchain)
return minimal_toolchain

def copy_easyconfigs(paths, target_dir):
"""
Expand Down
108 changes: 2 additions & 106 deletions easybuild/framework/easyconfig/tools.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@
from vsc.utils import fancylogger

from easybuild.framework.easyconfig import EASYCONFIGS_PKG_SUBDIR
from easybuild.framework.easyconfig.easyconfig import ActiveMNS, create_paths, process_easyconfig
from easybuild.framework.easyconfig.easyconfig import robot_find_easyconfig
from easybuild.framework.easyconfig.easyconfig import ActiveMNS, create_paths, process_easyconfig, module_is_available
from easybuild.framework.easyconfig.easyconfig import robot_find_easyconfig, get_toolchain_hierarchy
Copy link
Member

@boegel boegel Feb 15, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these actually still used here?

from easybuild.framework.easyconfig.format.format import DEPENDENCY_PARAMETERS
from easybuild.tools.build_log import EasyBuildError
from easybuild.tools.config import build_option
Expand Down Expand Up @@ -101,22 +101,6 @@ def skip_available(easyconfigs):
return retained_easyconfigs


def module_is_available(full_mod_name, modtool, avail_modules, hidden):
"""
Check whether specified module is available.

@param full_mod_name: full module name
@param modtool: ModulesTool instance that can be used to check whether (hidden) modules exist
@param avail_modules: list of available modules
@param hidden: hidden module
"""
resolved = full_mod_name in avail_modules
# hidden modules need special care, since they may not be included in list of available modules
if hidden:
resolved |= modtool.exist([full_mod_name])[0]
return resolved


def find_resolved_modules(easyconfigs, avail_modules, retain_all_deps=False):
"""
Find easyconfigs in 1st argument which can be fully resolved using modules specified in 2nd argument
Expand Down Expand Up @@ -168,94 +152,6 @@ def find_resolved_modules(easyconfigs, avail_modules, retain_all_deps=False):
return ordered_ecs, new_easyconfigs, avail_modules


def toolchain_hierarchy_cache(func):
"""Function decorator to cache (and retrieve cached) toolchain hierarchy queries."""
cache = {}

def cache_aware_func(toolchain):
"""Look up toolchain hierarchy in cache first, determine and cache it if not available yet."""
cache_key = (toolchain['name'], toolchain['version'])

# fetch from cache if available, cache it if it's not
if cache_key in cache:
_log.debug("Using cache to return hierarchy for toolchain %s: %s", str(toolchain), cache[cache_key])
return cache[cache_key]
else:
toolchain_hierarchy = func(toolchain)
cache[cache_key] = toolchain_hierarchy
return cache[cache_key]

return cache_aware_func


@toolchain_hierarchy_cache
def get_toolchain_hierarchy(parent_toolchain):
"""
Determine list of subtoolchains for specified parent toolchain.
Result starts with the most minimal subtoolchains first, ends with specified toolchain.

The dummy toolchain is considered the most minimal subtoolchain only if the add_dummy_to_minimal_toolchains
build option is enabled.

@param parent_toolchain: dictionary with name/version of parent toolchain
"""
# obtain list of all possible subtoolchains
_, all_tc_classes = search_toolchain('')
subtoolchains = dict((tc_class.NAME, getattr(tc_class, 'SUBTOOLCHAIN', None)) for tc_class in all_tc_classes)

current_tc_name, current_tc_version = parent_toolchain['name'], parent_toolchain['version']
subtoolchain_name, subtoolchain_version = subtoolchains[current_tc_name], None

# the parent toolchain is at the top of the hierarchy
toolchain_hierarchy = [parent_toolchain]

while subtoolchain_name:
# grab the easyconfig of the current toolchain and search the dependencies for a version of the subtoolchain
path = robot_find_easyconfig(current_tc_name, current_tc_version)
if path is None:
raise EasyBuildError("Could not find easyconfig for %(name)s toolchain version %(version)s",
current_tc_name, current_tc_version)

# parse the easyconfig
parsed_ec = process_easyconfig(path)[0]
# search the dependencies for the version of the subtoolchain
dep_tcs = [dep_toolchain['toolchain'] for dep_toolchain in parsed_ec['dependencies']
if dep_toolchain['toolchain']['name'] == subtoolchain_name]
unique_dep_tc_versions = set([dep_tc['version'] for dep_tc in dep_tcs])

if len(unique_dep_tc_versions) == 1:
subtoolchain_version = dep_tcs[0]['version']

elif len(unique_dep_tc_versions) == 0:
# only retain GCCcore as subtoolchain if version was found
if subtoolchain_name == GCCcore.NAME:
_log.info("No version found for %s; assuming legacy toolchain and skipping it as subtoolchain.",
subtoolchain_name)
subtoolchain_name = GCCcore.SUBTOOLCHAIN
subtoolchain_version = ''
# dummy toolchain: end of the line
elif subtoolchain_name == DUMMY_TOOLCHAIN_NAME:
subtoolchain_version = ''
else:
raise EasyBuildError("No version found for subtoolchain %s in dependencies of %s",
subtoolchain_name, current_tc_name)
else:
raise EasyBuildError("Multiple versions of %s found in dependencies of toolchain %s: %s",
subtoolchain_name, current_tc_name, unique_dep_tc_versions)

if subtoolchain_name == DUMMY_TOOLCHAIN_NAME and not build_option('add_dummy_to_minimal_toolchains'):
# we're done
break

# add to hierarchy and move to next
current_tc_name, current_tc_version = subtoolchain_name, subtoolchain_version
subtoolchain_name, subtoolchain_version = subtoolchains[current_tc_name], None
toolchain_hierarchy.insert(0, {'name': current_tc_name, 'version': current_tc_version})

_log.info("Found toolchain hierarchy for toolchain %s: %s", parent_toolchain, toolchain_hierarchy)
return toolchain_hierarchy


def refresh_dependencies(initial_dependencies, altered_dep):
"""
Refresh derived arguments in a dependency
Expand Down
1 change: 1 addition & 0 deletions easybuild/tools/robot.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ def resolve_dependencies(easyconfigs, retain_all_deps=False, minimal_toolchains=
@param use_existing_modules: boolean for whether to prioritise the reuse of existing modules (works in
combination with minimal_toolchains)
"""
minimal_toolchains = False
Copy link
Member

@boegel boegel Feb 15, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a FIXME here that this needs to be cleaned up (or just clean it up ;-))


robot = build_option('robot_path')
# retain all dependencies if specified by either the resp. build option or the dedicated named argument
Expand Down