Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
150 commits
Select commit Hold shift + click to select a range
ab933f1
Allow use of --copy-ec with --from-pr
Oct 19, 2020
9accd5d
Appease the hound
Oct 19, 2020
e2f270a
Fix broken tests
Oct 19, 2020
64e95f2
Fix broken tests
Oct 19, 2020
4e6980f
Fix broken tests (2nd attempt)
Oct 19, 2020
fceea89
Reinstate the copy of the easyconfigs from the PR
Oct 19, 2020
51f0b06
Add test for --from-pr and --copy-ec
Oct 19, 2020
d633ece
Fix linting and add additional tests
Oct 19, 2020
f8db57d
Address most comments in the review
Oct 19, 2020
b3615f4
Fix linting
Oct 19, 2020
2408626
Fix linting
Oct 19, 2020
34cd4a5
Rework the approach and add additional tests
Oct 20, 2020
f821af1
Rework the approach and add additional tests
Oct 20, 2020
33d8570
Fix comment
Oct 20, 2020
3980dc9
Be a little more careful with patches
Oct 20, 2020
f7aa646
Retain old behaviour when using try-*
Oct 20, 2020
11f5a48
Tidy up the new tests
Oct 20, 2020
0141243
Keep our patch list unique
Oct 20, 2020
e61e72d
Fix copy_ecs_to_target for case where I want to copy a single file to…
Oct 20, 2020
30145e4
Fix broken test
Oct 20, 2020
49b4f47
Fix broken test
Oct 20, 2020
1a8eecf
Fix broken test
Oct 20, 2020
380387b
Fix broken test
Oct 20, 2020
eedbfd1
Fix broken test
Oct 20, 2020
ae9e0ea
Only grab the current working directory once at the beginning.
Oct 20, 2020
c5560c4
Fix GitHub tools leaving you in temporary directory
Oct 20, 2020
26d9115
Fix GitHub tools leaving you in temporary directory
Oct 20, 2020
7944c0a
See if starting in the right directory makes a difference
Oct 20, 2020
6e5be8d
Final fix on broken test
Oct 20, 2020
1758308
use decorator to cache result of fetch_files_from_pr
boegel Oct 24, 2020
7bef831
add locate_files function to easybuild.tools.filetools
boegel Oct 24, 2020
aa470f1
check whether paths still exist before using cached result for fetch_…
boegel Oct 24, 2020
5c18a8e
use locate_files function in det_easyconfig_paths
boegel Oct 24, 2020
756f337
fix default for pr_target_account build option
boegel Oct 24, 2020
31261d2
add det_copy_ec_specs function to easybuild.framework.easyconfig.tools
boegel Oct 24, 2020
a761eab
use det_copy_ec_specs in main.py
boegel Oct 24, 2020
0ed52ec
clean up and enhance tests for combination of --copy-ec and --from-pr
boegel Oct 24, 2020
79f8edb
fix changed error pattern in test_robot_path_check due to using locat…
boegel Oct 24, 2020
2f6ac6e
fix duplicate default value for pr_target_account build option
boegel Oct 24, 2020
173ca95
determine --copy-ec target path *before* categorizing eb arguments by…
boegel Oct 24, 2020
bb96d1a
don't make assumptions about current working directory when checking …
boegel Oct 24, 2020
b6881ee
correctly add archive subdir to ignore_subdirs in det_easyconfig_path…
boegel Oct 24, 2020
1f7a0a7
use full path when checking patch file in --copy-ec --from-pr test
boegel Oct 24, 2020
aaa6737
take indices into account in locate_files
boegel Oct 25, 2020
c989ce9
enhance copy_files to avoid need for copy_ecs_to_target
boegel Oct 25, 2020
19402a6
clean up imports in easybuild/framework/easyconfig/tools.py
boegel Oct 25, 2020
e32d815
exit after copying when combining --copy-ec and --try-*
boegel Oct 25, 2020
9bca899
enhance copy_files: single file target, error on empty input list, ve…
boegel Oct 25, 2020
f4cf872
add locate_files function to filetools module
boegel Oct 25, 2020
20faaaf
move definition of fetch_easyblocks_from_pr and fetch_easyconfigs_fro…
boegel Oct 25, 2020
4028afe
cache result of fetch_files_from_pr (mainly to speed up tests involvi…
boegel Oct 25, 2020
c6091f7
also ignore vsc.* imports coming from from pkg_resources/__init__.py …
boegel Oct 29, 2020
ec1916d
bump version to 4.3.2dev
boegel Oct 30, 2020
25eef76
Merge pull request #3492 from boegel/develop
akesandgren Oct 30, 2020
521ab88
Merge pull request #3491 from boegel/fake_vsc_pkg_resources
akesandgren Oct 30, 2020
baad8f7
fix cache key in pr_files_cache + expose cache itself via _cache
boegel Nov 2, 2020
9a9b657
add test for pr_files_cache
boegel Nov 2, 2020
00d3f8a
fix typo in docstring, fix overindented line + drop print in fetch_fi…
boegel Nov 2, 2020
8922459
don't pass username in github_api_get_request when no GitHub token is…
boegel Nov 2, 2020
df6a864
skip test_fetch_files_from_pr_cache when no GitHub token is available
boegel Nov 2, 2020
638e94e
also inject -rpath options for all entries in $LIBRARY_PATH in RPATH …
boegel Nov 2, 2020
c35647f
retain library path for RPATH linking with same condition for paths s…
boegel Nov 2, 2020
0741191
Merge pull request #3495 from boegel/rpath_args_LIBRARY_PATH
Nov 2, 2020
c5f48c5
exclude test configuration with Lmod 7 and Python 3, except for Pytho…
boegel Nov 2, 2020
9c396bf
Merge pull request #3484 from boegel/fetch_files_from_pr_cache
migueldiascosta Nov 3, 2020
392a8ba
Merge pull request #3496 from boegel/trim_github_actions_workflows
Nov 3, 2020
1952311
add support for %(mod_name)s template value
boegel Nov 5, 2020
68a105c
fix test_template_constant_dict after introducing %(mod_name)s templa…
boegel Nov 5, 2020
d86ced2
rename %(mod_name)s template to %(module_name)s
boegel Nov 6, 2020
5046731
Merge pull request #3497 from boegel/mod_name_template
Nov 6, 2020
7b3801e
significantly speed up parsing of easyconfig files by only extracting…
boegel Nov 8, 2020
69ab582
avoid TypeError being raised by list_toolchains
boegel Nov 9, 2020
ab4aa6f
clean up implementation in list_toolchain functions
boegel Nov 9, 2020
ccce45b
Merge pull request #3499 from boegel/fix_typerror_list_toolchains
bartoldeman Nov 9, 2020
ad8a2b5
Added toolchain options to specify extra compiler arguments
mboisson Apr 18, 2017
af6d092
cosmetic changes
mboisson Apr 19, 2017
2ca2236
Appease the Hound
bartoldeman Nov 10, 2020
4756e49
Introduce extra_cxxflags/fcflags/f90flags.
bartoldeman Nov 10, 2020
cb4d07d
Fix logic and add test.
bartoldeman Nov 10, 2020
a378925
Merge branch 'develop' into extract_comments_on_demand
boegel Nov 10, 2020
9c041f1
Merge pull request #2193 from ComputeCanada/extra_compiler_flags
boegel Nov 10, 2020
27b8362
Merge pull request #3498 from boegel/extract_comments_on_demand
akesandgren Nov 10, 2020
831fe94
fix typo (maeks -> makes)
boegel Nov 11, 2020
444b980
Merge pull request #3494 from boegel/from_pr_no_github_token
migueldiascosta Nov 11, 2020
2e8949c
check if PR is already merged in --merge-pr
migueldiascosta Nov 16, 2020
90a6139
only raise already merged PR error in --merge-pr when not using dry-run
migueldiascosta Nov 16, 2020
8107a75
fix remarks
boegel Nov 17, 2020
bb18de4
tackle remarks
boegel Nov 17, 2020
731b01a
graciously handle wrong PR # in fetch_pr_data
migueldiascosta Nov 19, 2020
be5d589
Merge pull request #3483 from boegel/enhance_copy_files
akesandgren Nov 20, 2020
000e79e
Merge pull request #3485 from boegel/locate_files
akesandgren Nov 20, 2020
604e22f
Merge pull request #3503 from migueldiascosta/graceously_handle_wrong…
akesandgren Nov 24, 2020
efb67a4
gobff
SebastianAchilles Nov 25, 2020
03ce476
gobff
SebastianAchilles Nov 25, 2020
5789f37
Merge branch 'develop' into copy-ec-from-pr
boegel Nov 20, 2020
bc2d08e
fix broken tests
boegel Nov 26, 2020
c37f6e6
fix double return in cache_aware_func that was introduced in merge wi…
boegel Nov 26, 2020
d7236c4
also check if PR is closed in --merge-pr
migueldiascosta Nov 27, 2020
6eb5304
update test for --merge-pr to check that it errors out for closed/mer…
migueldiascosta Nov 27, 2020
66ce195
gobff: Update blis.py
SebastianAchilles Nov 27, 2020
9db3089
Merge pull request #3505 from SebastianAchilles/gobff
akesandgren Nov 27, 2020
43dfcc6
Merge pull request #3502 from migueldiascosta/merge_pr_check_already_…
boegel Nov 28, 2020
7bb0ecf
fix typo in comment
boegel Nov 28, 2020
fb34ca9
fix regression in apply_regex_substitutions introduced in #3450: also…
boegel Nov 28, 2020
f6ce140
better dry run output for apply_regex_substitutions + don't compile r…
boegel Nov 28, 2020
529a8b5
Merge pull request #3507 from boegel/apply_regex_substitutions_path_list
Micket Nov 28, 2020
bd4a885
don't include file/ldd/readelf commands run during RPATH sanity check…
boegel Nov 28, 2020
34627ca
Merge pull request #3508 from boegel/rpath_sanity_check_no_trace
Micket Nov 28, 2020
3ce315a
also check file contents in test for --copy-ec --from-pr
Nov 30, 2020
0fb8d5e
Merge pull request #3482 from boegel/copy-ec-from-pr
Nov 30, 2020
17986bb
clarify input format for --cuda-compute-capabilities (fixes #3500)
boegel Nov 30, 2020
9e4f49e
update installation procedure for EasyBuild in generated Singularity …
boegel Nov 30, 2020
7d8f251
Merge pull request #3509 from boegel/cuda_compute_capabilities_help
bartoldeman Nov 30, 2020
14e939d
Merge pull request #3510 from boegel/container_pip_install_easybuild
bartoldeman Nov 30, 2020
ff3d33f
add (experimental) support for specifying easyconfig files via "easys…
deniskristak Nov 30, 2020
068581b
tweak test_easystack_basic to get better error reporting if it fails
boegel Dec 1, 2020
54c8ee7
enable raising of errors in test_easystack_basic
boegel Dec 1, 2020
dea39f5
Merge pull request #3511 from boegel/fix_easystack_tests
migueldiascosta Dec 1, 2020
17e4c5e
fix error when --easystack is used without having PyYAML installed
boegel Dec 1, 2020
0fd5613
Merge pull request #3515 from boegel/fix_easystack_yaml_check
bartoldeman Dec 1, 2020
c0d12f8
also include *.yaml files from test/ dir in package
boegel Dec 4, 2020
10c8698
run tests outside of checked out repository dir (so we test installed…
boegel Dec 4, 2020
2aafb17
also test bootstrap script with Python 3.9 (supported since EasyBuild…
boegel Dec 4, 2020
3131501
Merge pull request #3517 from boegel/fix_package_test_yaml
bartoldeman Dec 4, 2020
7fe24b6
Merge branch 'develop' into GA_CI_fixes
boegel Dec 4, 2020
3fc575f
also take easystacks subdir into account in setup.py
boegel Dec 4, 2020
ae372e0
Merge pull request #3518 from boegel/GA_CI_fixes
migueldiascosta Dec 4, 2020
b45f519
Bump cryptography from 2.9.2 to 3.2
dependabot[bot] Dec 4, 2020
41c3dec
ignore CryptographyDeprecationWarning when using Python 2
boegel Dec 6, 2020
a8863d4
bump to cryptography 3.2.1 when using Python 2
boegel Dec 6, 2020
04bca8f
take into account slightly different error with Python 2.6 in test_ea…
boegel Dec 6, 2020
b08beb1
Merge pull request #3520 from boegel/fix_easystack_test_py26
verdurin Dec 6, 2020
d33b2b7
Merge branch 'develop' into dependabot/pip/cryptography-3.2
boegel Dec 6, 2020
dc91f0c
use --dry-run in test_easystack_basic (to avoid downloading of binuti…
boegel Dec 7, 2020
84911f8
sort easyconfig filenames in log message for parsed easystack to fix …
boegel Dec 7, 2020
e545186
Merge pull request #3521 from boegel/easystack_test_dry_run
Dec 7, 2020
3ab023c
Merge branch 'develop' into dependabot/pip/cryptography-3.2
boegel Dec 7, 2020
f07fbde
Merge pull request #3519 from easybuilders/dependabot/pip/cryptograph…
Dec 7, 2020
cf850e7
prepare release notes for EasyBuild v4.3.2 + bump version to 4.3.2
migueldiascosta Dec 9, 2020
c744a1d
Fix missing command output in test_step
Flamefire Dec 9, 2020
b5b391e
Add --skip-test-step to skip the test_step
Flamefire Dec 9, 2020
b29968f
Avoid stdout when running eb_main
Flamefire Dec 9, 2020
bcafacd
Update other tests for new test EC
Flamefire Dec 9, 2020
0429b6b
Merge pull request #3524 from Flamefire/skip_test_step
Dec 9, 2020
33b38e5
minor tweaks to EasyBuild v4.3.2 release notes
boegel Dec 9, 2020
d1fb2da
include #3524 in release notes for EasyBuild v4.3.2
boegel Dec 9, 2020
0639ea4
fix 'eb --help=rst' when running with Python 3
boegel Dec 9, 2020
5764650
also include #3525 in release notes for EasyBuild v4.3.2
boegel Dec 9, 2020
87fb230
Merge pull request #3525 from boegel/eb_help_rst_py3
Dec 9, 2020
51535c5
Merge pull request #3522 from migueldiascosta/eb432
boegel Dec 9, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions .github/workflows/unit_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,14 @@ jobs:
python: 3.8
- modules_tool: modules-4.1.4
python: 3.9
- modules_tool: Lmod-7.8.22
python: 3.5
- modules_tool: Lmod-7.8.22
python: 3.7
- modules_tool: Lmod-7.8.22
python: 3.8
- modules_tool: Lmod-7.8.22
python: 3.9
fail-fast: false
steps:
- uses: actions/checkout@v2
Expand Down Expand Up @@ -122,6 +130,9 @@ jobs:
EASYBUILD_MODULE_SYNTAX: ${{matrix.module_syntax}}
TEST_EASYBUILD_MODULE_SYNTAX: ${{matrix.module_syntax}}
run: |
# run tests *outside* of checked out easybuild-framework directory,
# to ensure we're testing installed version (see previous step)
cd $HOME
# initialize environment for modules tool
if [ -f $HOME/moduleshome ]; then export MODULESHOME=$(cat $HOME/moduleshome); fi
source $(cat $HOME/mod_init); type module
Expand Down Expand Up @@ -153,15 +164,12 @@ jobs:
# run test suite
python -O -m test.framework.suite 2>&1 | tee test_framework_suite.log
# try and make sure output of running tests is clean (no printed messages/warnings)
IGNORE_PATTERNS="no GitHub token available|skipping SvnRepository test|requires Lmod as modules tool|stty: 'standard input': Inappropriate ioctl for device|CryptographyDeprecationWarning: Python 3.5|from cryptography.*default_backend"
IGNORE_PATTERNS="no GitHub token available|skipping SvnRepository test|requires Lmod as modules tool|stty: 'standard input': Inappropriate ioctl for device|CryptographyDeprecationWarning: Python 3.5|from cryptography.*default_backend|CryptographyDeprecationWarning: Python 2"
# '|| true' is needed to avoid that Travis stops the job on non-zero exit of grep (i.e. when there are no matches)
PRINTED_MSG=$(egrep -v "${IGNORE_PATTERNS}" test_framework_suite.log | grep '\.\n*[A-Za-z]' || true)
test "x$PRINTED_MSG" = "x" || (echo "ERROR: Found printed messages in output of test suite\n${PRINTED_MSG}" && exit 1)

- name: test bootstrap script
# skip testing of bootstrap script with Python 3.9,
# until an EasyBuild release that is compatible with Python 3.9 is available
if: ${{ matrix.python != 3.9 }}
run: |
# (re)initialize environment for modules tool
if [ -f $HOME/moduleshome ]; then export MODULESHOME=$(cat $HOME/moduleshome); fi
Expand Down
2 changes: 1 addition & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ include optcomplete.bash
recursive-include etc *
recursive-include easybuild *py
recursive-include easybuild/scripts *
recursive-include test *py *eb
recursive-include test *py *eb *yaml
recursive-include test/framework/modules *
recursive-include test/framework/sandbox/sources *
include CONTRIBUTING.md
Expand Down
35 changes: 35 additions & 0 deletions RELEASE_NOTES
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,41 @@ For more detailed information, please see the git log.

These release notes can also be consulted at https://easybuild.readthedocs.io/en/latest/Release_notes.html.

v4.3.2 (December 10th 2020)
---------------------------

update/bugfix release

- add (experimental) support for specifying easyconfig files via an "easystack" file (#3479, #3511, #3515, #3517, #3520, #3521)
- see also https://easybuild.readthedocs.io/en/latest/Easystack-files.html
- add definition for new 'gobff' toolchain using BLIS and LibFLAME (#3505)
- various enhancements, including:
- add support for toolchain options like 'extra_cxxflags' to specify extra compiler options (#2193)
- fix combination of --copy-ec and --from-pr (#3482)
- enhance copy_files function: support single file target, error on empty input list, support verbose mode (#3483)
- cache result of fetch_files_from_pr function (mainly to speed up framework test suite) (#3484)
- add locate_files function to filetools module (#3485)
- add support for %(module_name)s template value (#3497)
- clarify input format for --cuda-compute-capabilities in 'eb --help' output (#3509)
- add support for skiping unit tests (test step) via --skip-test-step (#3524)
- various bug fixes, including:
- also ignore vsc.* imports coming from from pkg_resources/__init__.py (setuptools) in fake vsc namespace (#3491)
- don't pass username in github_api_get_request when no GitHub token is available (#3494)
- also inject -rpath options for all entries in $LIBRARY_PATH in RPATH wrappers (#3495)
- avoid TypeError being raised by list_toolchains (#3499)
- check if PR is already merged in --merge-pr (#3502)
- graciously handle wrong PR id in fetch_pr_data (#3503)
- fix regression in apply_regex_substitutions: also accept list of paths to patch (#3507)
- update installation procedure for EasyBuild in generated Singularity container recipes (#3510)
- fix GitHub Actions workflow for test suite: run outside of repo checkout + also test bootstrap script with Python 3.9 (#3518)
- bump cryptography from 2.9.2 to 3.2 for Python 2 in requirements.txt (#3519)
- fix 'eb --help=rst' when running with Python 3 (#3525)
- other changes:
- exclude test configurations with Lmod 7 and Python 3, except for Python 3.6 (#3496)
- significantly speed up parsing of easyconfig files by only extracting comments from an easyconfig file when they're actually needed (#3498)
- don't include file/ldd/readelf commands run during RPATH sanity check in --trace output (#3508)


v4.3.1 (October 29th 2020)
--------------------------

Expand Down
17 changes: 11 additions & 6 deletions easybuild/framework/easyblock.py
Original file line number Diff line number Diff line change
Expand Up @@ -2139,10 +2139,11 @@ def build_step(self):

def test_step(self):
"""Run unit tests provided by software (if any)."""
if self.cfg['runtest']:
unit_test_cmd = self.cfg['runtest']
if unit_test_cmd:

self.log.debug("Trying to execute %s as a command for running unit tests...")
(out, _) = run_cmd(self.cfg['runtest'], log_all=True, simple=False)
self.log.debug("Trying to execute %s as a command for running unit tests...", unit_test_cmd)
(out, _) = run_cmd(unit_test_cmd, log_all=True, simple=False)

return out

Expand Down Expand Up @@ -2460,7 +2461,7 @@ def sanity_check_rpath(self, rpath_dirs=None):
for path in [os.path.join(dirpath, x) for x in os.listdir(dirpath)]:
self.log.debug("Sanity checking RPATH for %s", path)

out, ec = run_cmd("file %s" % path, simple=False)
out, ec = run_cmd("file %s" % path, simple=False, trace=False)
if ec:
fails.append("Failed to run 'file %s': %s" % (path, out))

Expand All @@ -2470,7 +2471,7 @@ def sanity_check_rpath(self, rpath_dirs=None):
# ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
if "dynamically linked" in out:
# check whether all required libraries are found via 'ldd'
out, ec = run_cmd("ldd %s" % path, simple=False)
out, ec = run_cmd("ldd %s" % path, simple=False, trace=False)
if ec:
fail_msg = "Failed to run 'ldd %s': %s" % (path, out)
self.log.warning(fail_msg)
Expand All @@ -2483,7 +2484,7 @@ def sanity_check_rpath(self, rpath_dirs=None):
self.log.debug("Output of 'ldd %s' checked, looks OK", path)

# check whether RPATH section in 'readelf -d' output is there
out, ec = run_cmd("readelf -d %s" % path, simple=False)
out, ec = run_cmd("readelf -d %s" % path, simple=False, trace=False)
if ec:
fail_msg = "Failed to run 'readelf %s': %s" % (path, out)
self.log.warning(fail_msg)
Expand Down Expand Up @@ -3310,6 +3311,10 @@ def build_and_install_one(ecdict, init_env):
_log.debug("Skip set to %s" % skip)
app.cfg['skip'] = skip

if build_option('skip_test_step'):
_log.debug('Adding test_step to skipped steps')
app.cfg.update('skipsteps', TEST_STEP, allow_duplicate=False)

# build easyconfig
errormsg = '(no error)'
# timing info
Expand Down
7 changes: 6 additions & 1 deletion easybuild/framework/easyconfig/format/format.py
Original file line number Diff line number Diff line change
Expand Up @@ -612,12 +612,17 @@ def __init__(self):
raise EasyBuildError('Invalid version number %s (incorrect length)', self.VERSION)

self.rawtext = None # text version of the easyconfig
self.comments = {} # comments in easyconfig file
self._comments = {} # comments in easyconfig file
self.header = None # easyconfig header (e.g., format version, license, ...)
self.docstring = None # easyconfig docstring (e.g., author, maintainer, ...)

self.specs = {}

@property
def comments(self):
"""Return comments in easyconfig file"""
return self._comments

def set_specifications(self, specs):
"""Set specifications."""
self.log.debug('Set copy of specs %s' % specs)
Expand Down
15 changes: 13 additions & 2 deletions easybuild/framework/easyconfig/format/one.py
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,8 @@ def parse(self, txt):
"""
Pre-process txt to extract header, docstring and pyheader, with non-indented section markers enforced.
"""
super(FormatOneZero, self).parse(txt, strict_section_markers=True)
self.rawcontent = txt
super(FormatOneZero, self).parse(self.rawcontent, strict_section_markers=True)

def _reformat_line(self, param_name, param_val, outer=False, addlen=0):
"""
Expand Down Expand Up @@ -356,14 +357,24 @@ def dump(self, ecfg, default_values, templ_const, templ_val, toolchain_hierarchy

return '\n'.join(dump)

@property
def comments(self):
"""
Return comments (and extract them first if needed).
"""
if not self._comments:
self.extract_comments(self.rawcontent)

return self._comments

def extract_comments(self, rawtxt):
"""
Extract comments from raw content.

Discriminates between comment header, comments above a line (parameter definition), and inline comments.
Inline comments on items of iterable values are also extracted.
"""
self.comments = {
self._comments = {
'above': {}, # comments above a parameter definition
'header': [], # header comment lines
'inline': {}, # inline comments
Expand Down
2 changes: 0 additions & 2 deletions easybuild/framework/easyconfig/parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,8 +110,6 @@ def __init__(self, filename=None, format_version=None, rawcontent=None,
else:
raise EasyBuildError("Neither filename nor rawcontent provided to EasyConfigParser")

self._formatter.extract_comments(self.rawcontent)

def process(self, filename=None):
"""Create an instance"""
self._read(filename=filename)
Expand Down
7 changes: 6 additions & 1 deletion easybuild/framework/easyconfig/templates.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
# derived from easyconfig, but not from ._config directly
TEMPLATE_NAMES_EASYCONFIG = [
('arch', "System architecture (e.g. x86_64, aarch64, ppc64le, ...)"),
('module_name', "Module name"),
('nameletter', "First letter of software name"),
('toolchain_name', "Toolchain name"),
('toolchain_version', "Toolchain version"),
Expand All @@ -72,8 +73,8 @@
]
# values taken from the EasyBlock before each step
TEMPLATE_NAMES_EASYBLOCK_RUN_STEP = [
('installdir', "Installation directory"),
('builddir', "Build directory"),
('installdir', "Installation directory"),
]
# software names for which to define <pref>ver and <pref>shortver templates
TEMPLATE_SOFTWARE_VERSIONS = [
Expand Down Expand Up @@ -208,6 +209,10 @@ def template_constant_dict(config, ignore=None, skip_lower=None, toolchain=None)
softname = config['name']
if softname is not None:
template_values['nameletter'] = softname[0]

elif name[0] == 'module_name':
template_values['module_name'] = getattr(config, 'short_mod_name', None)

else:
raise EasyBuildError("Undefined name %s from TEMPLATE_NAMES_EASYCONFIG", name)

Expand Down
112 changes: 72 additions & 40 deletions easybuild/framework/easyconfig/tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,9 @@
from easybuild.tools.build_log import EasyBuildError, print_msg, print_warning
from easybuild.tools.config import build_option
from easybuild.tools.environment import restore_env
from easybuild.tools.filetools import find_easyconfigs, is_patch_file, read_file, resolve_path, which, write_file
from easybuild.tools.github import fetch_easyconfigs_from_pr, download_repo
from easybuild.tools.filetools import find_easyconfigs, is_patch_file, locate_files
from easybuild.tools.filetools import read_file, resolve_path, which, write_file
from easybuild.tools.github import fetch_easyconfigs_from_pr, fetch_files_from_pr, download_repo
from easybuild.tools.multidiff import multidiff
from easybuild.tools.py2vs3 import OrderedDict
from easybuild.tools.toolchain.toolchain import is_system_toolchain
Expand Down Expand Up @@ -348,44 +349,13 @@ def det_easyconfig_paths(orig_paths):
ec_files = [path for path in pr_files if path.endswith('.eb')]

if ec_files and robot_path:
# look for easyconfigs with relative paths in robot search path,
# unless they were found at the given relative paths

# determine which easyconfigs files need to be found, if any
ecs_to_find = []
for idx, ec_file in enumerate(ec_files):
if ec_file == os.path.basename(ec_file) and not os.path.exists(ec_file):
ecs_to_find.append((idx, ec_file))
_log.debug("List of easyconfig files to find: %s" % ecs_to_find)

# find missing easyconfigs by walking paths in robot search path
for path in robot_path:
_log.debug("Looking for missing easyconfig files (%d left) in %s..." % (len(ecs_to_find), path))
for (subpath, dirnames, filenames) in os.walk(path, topdown=True):
for idx, orig_path in ecs_to_find[:]:
if orig_path in filenames:
full_path = os.path.join(subpath, orig_path)
_log.info("Found %s in %s: %s" % (orig_path, path, full_path))
ec_files[idx] = full_path
# if file was found, stop looking for it (first hit wins)
ecs_to_find.remove((idx, orig_path))

# stop os.walk insanity as soon as we have all we need (os.walk loop)
if not ecs_to_find:
break

# ignore subdirs specified to be ignored by replacing items in dirnames list used by os.walk
dirnames[:] = [d for d in dirnames if d not in build_option('ignore_dirs')]

# ignore archived easyconfigs, unless specified otherwise
if not build_option('consider_archived_easyconfigs'):
dirnames[:] = [d for d in dirnames if d != EASYCONFIGS_ARCHIVE_DIR]

# stop os.walk insanity as soon as we have all we need (outer loop)
if not ecs_to_find:
break

return [os.path.abspath(ec_file) for ec_file in ec_files]
ignore_subdirs = build_option('ignore_dirs')
if not build_option('consider_archived_easyconfigs'):
ignore_subdirs.append(EASYCONFIGS_ARCHIVE_DIR)

ec_files = locate_files(ec_files, robot_path, ignore_subdirs=ignore_subdirs)

return ec_files


def parse_easyconfigs(paths, validate=True):
Expand Down Expand Up @@ -728,3 +698,65 @@ def avail_easyblocks():
easyblock_mod_name, easyblocks[easyblock_mod_name]['loc'], path)

return easyblocks


def det_copy_ec_specs(orig_paths, from_pr):
"""Determine list of paths + target directory for --copy-ec."""

target_path, paths = None, []

# if only one argument is specified, use current directory as target directory
if len(orig_paths) == 1:
target_path = os.getcwd()
paths = orig_paths[:]

# if multiple arguments are specified, assume that last argument is target location,
# and remove that from list of paths to copy
elif orig_paths:
target_path = orig_paths[-1]
paths = orig_paths[:-1]

# if --from-pr was used in combination with --copy-ec, some extra care must be taken
if from_pr:
# pull in the paths to all the changed files in the PR,
# which includes easyconfigs but also patch files (& maybe more);
# do this in a dedicated subdirectory of the working tmpdir,
# to avoid potential trouble with already existing files in the working tmpdir
# (note: we use a fixed subdirectory in the working tmpdir here rather than a unique random subdirectory,
# to ensure that the caching for fetch_files_from_pr works across calls for the same PR)
tmpdir = os.path.join(tempfile.gettempdir(), 'fetch_files_from_pr_%s' % from_pr)
pr_paths = fetch_files_from_pr(pr=from_pr, path=tmpdir)

# assume that files need to be copied to current working directory for now
target_path = os.getcwd()

if orig_paths:
last_path = orig_paths[-1]

# check files touched by PR and see if the target directory for --copy-ec
# corresponds to the name of one of these files;
# if so we should copy the specified file(s) to the current working directory,
# since interpreting the last argument as target location is very unlikely to be correct in this case
pr_filenames = [os.path.basename(p) for p in pr_paths]
if last_path in pr_filenames:
paths = orig_paths[:]
else:
target_path = last_path
# exclude last argument that is used as target location
paths = orig_paths[:-1]

# if list of files to copy is empty at this point,
# we simply copy *all* files touched by the PR
if not paths:
paths = pr_paths

# replace path for files touched by PR (no need to worry about others)
for idx, path in enumerate(paths):
filename = os.path.basename(path)
pr_matches = [x for x in pr_paths if os.path.basename(x) == filename]
if len(pr_matches) == 1:
paths[idx] = pr_matches[0]
elif pr_matches:
raise EasyBuildError("Found multiple paths for %s in PR: %s", filename, pr_matches)

return paths, target_path
Loading