Errors and warnings in unit tests.

[shrinny]

Following the installation instructions here, running the unit tests after installation results in a lot of errors and warnings. Not sure if these are expected or not.

Link to full output

The majority of the warnings are WARNING: 'eb' not found in $PATH, failed to determine installation prefix!, which is weird because I do have eb on my $PATH. It seems like the tests change PATH and PYTHONPATH environment variables. Moreover, some of the test failures also seem to be related to the changed environment variables as running some of the failing tests in a python terminal (without the testing framework) seems to work.

On the other hand, most of the errors seem to be complaining about some toolchains not being available e.g
'Toolchain GCC not found, available toolchains: GCCcore,system'. Not sure if I was meant to install other toolchains first before running the unit tests? Doesn't seem to be mentioned in the documentation in any case.

---

System Information

user@system:~$ uname -a
Linux berni 5.3.0-24-generic #26-Ubuntu SMP Thu Nov 14 01:33:18 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
user@system:~$ python -V
Python 3.7.5
user@system:~$ type module
module is a function
module ()
{
    eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
user@system:~$ type -f module
-bash: type: module: not found
user@system:~$ module --version

Modules based on Lua: Version 8.3.3  2020-03-02 15:56 -06:00
    by Robert McLay mclay@tacc.utexas.edu

user@system:~$ module av EasyBuild
-------------------------- /opt/modules/all --------------------------   
EasyBuild/4.1.1 (L)

  Where:
   L:  Module is loaded

Use "module spider" to find all possible modules and extensions. 
Use "module keyword key1 key2 ..." to search for all possible modules 
matching any of the "keys"

user@system:~$ which -a eb
/opt/software/EasyBuild/4.1.1/bin/eb
user@system:~$ eb --version
This is EasyBuild 4.1.1 (framework: 4.1.1, easyblocks: 4.1.1) on host system.
user@system:~$

[boegel]

It's a known issue that the tests are not fully isolated from the environment in which they're run (see #2754 for example), which may partially explain what you're seeing.

However, the problems you're reporting seem more serious, and it makes me wonder if something else is going on.

How exactly did you run the tests (which command did you use), and from which directory? Where was EasyBuild installed?

Since eb is available through $PATH, running which eb works as expected in the exact same environment/location where you're running the tests?

[shrinny]

Sorry for getting back to you so late, I was out travelling for a bit.

I ran the tests by doing python -m test.framework.suite from my home directory. which eb works from the same location

[ocaisa]

I too see these warning when I run the tests through Pycharm (and this relatively recent behaviour). I have eb loaded as a module when I start Pycharm.

[grotewold-lab]

Same problem here,

"AssertionError: "\nWARNING: 'eb' not found in $PATH, failed to determine installation prefix!\n\n\nWARNING: 'eb' not found in $PATH, failed to determine installation prefix!\n\n\nWARNING: 'eb' not found in $PATH, failed to determine installation prefix!\n\n\nWARNING: 'eb' not found in $PATH, failed to determine installation prefix!\n\n\nWARNING: 'eb' not found in $PATH, failed to determine installation prefix!\n\n\nWARNING: 'eb' not found in $PATH, failed to determine installation prefix!\n\n" is not false"

python --version
Python 3.7.0
module -v
Version 8.2.7
module av EasyBuild
EasyBuild/4.1.1
Running in the same directory where EB was installed
which eb
$HOME/EB/software/EasyBuild/4.1.1/bin/eb

Running in a Red Hat Enterprise Linux Server 7.7 x86-64

[kelseymh]

I see the same errors myself, with the EasyBuild repositories checked out in my local area. Using the special module created by the installation procedure (note that I shortened the repository directory names),

#%Module
set root    "/scratch/user/kelsey/EasyBuild"
conflict    EasyBuild
prepend-path    PATH            "$root/framework"
prepend-path    PYTHONPATH      "$root/framework"
prepend-path    PYTHONPATH      "$root/easyblocks"
prepend-path    PYTHONPATH      "$root/easyconfigs"

The 'eb' command resolves properly at the command line,

$ which eb
/scratch/user/kelsey/EasyBuild/framework/eb

but fails during the test suite, as reported by @shrinny .

[boegel]

@kelseymh It's possible that the problem is caused by some tests getting rid of preloaded modules (which would effectively remove the location to eb from your $PATH).

We already have some protection against this though in the tests, but maybe we're not leveraging that everywhere we should be (or maybe not correctly):

easybuild-framework/test/framework/utilities.py

Lines 172 to 177 in 1064122

	# save values of $PATH & $PYTHONPATH, so they can be restored later
	# this is important in case EasyBuild was installed as a module, since that module may be unloaded,
	# for example due to changes to $MODULEPATH in case EasyBuild was installed in a module hierarchy
	# cfr. https://github.com/easybuilders/easybuild-framework/issues/1685
	self.env_path = os.environ.get('PATH')
	self.env_pythonpath = os.environ.get('PYTHONPATH')

[ocaisa]

I looked into this a bit last week, at some point the MODULEPATH get's cleaned from the environment and I suspect that triggers Lmod to unload the module that defines eb but I couldn't find exactly where/why this happens. I added a print statement outside the point that prints this warning and the interesting thing was that print statement was printed twice, once with a full PATH variable (so the eb command was found) and once without (hence the error). I couldn't figure out exactly where this was being triggered in the test suite.