bootstrap install error for RHEL6

[steevmi1]

I'm trying to install using the bootstrap method on a RHEL6 system with the stock python (2.6.6), and get the following error:

[[INFO]] +++ STAGE 2: installing EasyBuild in /opt/software/RHEL6-x86_64/easybuild with EasyBuild from stage 1...

Traceback (most recent call last):
  File "bootstrap_eb.py", line 1014, in <module>
    main()
  File "bootstrap_eb.py", line 817, in main
    stage2(tmpdir, templates, install_path, distribute_egg_dir, sourcepath)
  File "bootstrap_eb.py", line 687, in stage2
    easybuild_main()
  File "/tmp/tmpDDGLJh/eb_stage1/lib/python2.6/site-packages/easybuild_framework-3.3.1-py2.6.egg/easybuild/main.py", line 189, in main
    eb_go = eboptions.parse_options(args=args)
  File "/tmp/tmpDDGLJh/eb_stage1/lib/python2.6/site-packages/easybuild_framework-3.3.1-py2.6.egg/easybuild/tools/options.py", line 1061, in parse_options
    raise EasyBuildError("Failed to parse configuration options: %s" % err)
easybuild.tools.build_log.EasyBuildError: "Failed to parse configuration options: 'NoneType' object has no attribute 'startswith'"

-bash-4.1$ python -V
Python 2.6.6

[boegel]

@steevmi1 Hmm, this is a curious error, I suspect it is something in your environment that is making the bootstrap procedure fail.

Do you have any $EASYBUILD_* environment variables defined (check with env | grep ^EASYBUILD)?

Can you check whether you get some more information by defining $EASYBUILD_BOOTSTRAP_DEBUG (just run export EASYBUILD_BOOTSTRAP_DEBUG=1 before bootstrapping EasyBuild)?

You may also need to define $DEBUG_EASYBUILD_OPTIONS to get any additional useful information, since the problem occurs very early on, i.e. when EasyBuild tries to figure out how it is configured.

[OndrejMarsalek]

I have the same problem on Fedora 20. Here is the debug output and my environment:

https://gist.github.com/OndrejMarsalek/454e4866bf187f90b6400e4dc3d30cb0

[OndrejMarsalek]

When I run with DEBUG_EASYBUILD_OPTIONS and interactively (but not with redirection to a file), I get additional output, which seems to be key:

== 2017-08-08 15:26:28,736 filetools.py:309 WARNING Could not find command 'eb' (with permissions to read/execute it) in $PATH (['/tmp/tmpheXKCh/bin', '/tmp/tmpheXKCh/bin', '/home/users/marsalek/opt/lmod/lmod/libexec', '/usr/local/bin', '/usr/bin', '/usr/local/sbin', '/usr/sbin', '/home/users/marsalek/.local/bin', '/home/users/marsalek/bin'])
== 2017-08-08 15:26:28,741 build_log.py:156 ERROR EasyBuild crashed with an error (at ?:124 in __init__): Failed to parse configuration options: 'NoneType' object has no attribute 'startswith' (at easybuild/tools/options.py:1061 in parse_options)

[boegel]

As mentioned by @zao on Slack: is /tmp mounted noexec?

Can you try using a different location for temporary files? You can control that via $EASYBUILD_TMPDIR.

[arnold-jr]

Hi,

I'm having the same issue and did some debugging.

In function, easybuild/tools/filetools.py(177)resolve_path():

177  -> def resolve_path(path):
178       """
179       Return fully resolved path for given path.
180   
181       :param path: path that (maybe) contains symlinks
182       """
183       try:
184           resolved_path = os.path.realpath(path)
185       except OSError as err:
186           raise EasyBuildError("Resolving path %s failed: %s", path, err)
187   
188       return resolved_path

path is None, so the resulting error is

AttributeError: AttributeError("'NoneType' object has no attribute 'startswith'",)

path is being assigned the result of the call to
easybuild/tools/filetools.py(286)which():

286   def which(cmd, retain_all=False):
287       """
288       Return (first) path in $PATH for specified command, or None if command is not found
289   
290       :param retain_all: returns *all* locations to the specified command in $PATH, not just the first one"""
291       if retain_all:
292           res = []
293       else:
294           res = None
295   
296       paths = os.environ.get('PATH', '').split(os.pathsep)
297       for path in paths:
298           cmd_path = os.path.join(path, cmd)
299           # only accept path is command is there, and both readable and executable
300           if os.access(cmd_path, os.R_OK | os.X_OK) and os.path.isfile(cmd_path):
301               _log.info("Command %s found at %s" % (cmd, cmd_path))
302               if retain_all:
303                   res.append(cmd_path)
304               else:
305                   res = cmd_path
306                   break
307   
308       if not res:
309           _log.warning("Could not find command '%s' (with permissions to read/execute it) in $PATH (%s)" % (cmd, paths))
310       return res
311   

that occurs in
easybuild/framework/easyconfig/tools.py(254)get_paths_for():

249           path_list = [robot_path]
250       # consider Python search path, e.g. setuptools install path for easyconfigs
251 B     path_list.extend(sys.path)
252   
253       # figure out installation prefix, e.g. distutils install path for easyconfigs
254  ->     eb_path = resolve_path(which('eb'))
255       if eb_path is None:
256           _log.warning("'eb' not found in $PATH, failed to determine installation prefix")
257       else:
258           # eb should reside in <install_prefix>/bin/eb
259           install_prefix = os.path.dirname(os.path.dirname(eb_path))

I'm unclear as to why the path to eb should already be set, but I don't really know anything about the installation process. I tried setting EASYBUILD_TMPDIR but encountered the same error.

Hope this helps. Thanks!

[boegel]

@arnold-jr Great job debugging!

EasyBuild itself uses the location of where eb is available to figure out where the easyconfig files included in it are located.

It's clearly not doing a great job in properly reporting a situation in which eb is not available, so that should be improved.

Question is why is eb not available through $PATH, i.e. why is which returning None...

Do you have eb available in one of the locations mentioned in the Could not find command 'eb' warning? And is it readable/executable?

[arnold-jr]

@boegel Thanks!

Here's the warning:

== 2017-08-08 13:55:43,895 filetools.py:309 WARNING Could not find command 'eb' (with permissions to read/execute it) in $PATH (['/tmp/tmphKX4Mh/bin', '/tmp/tmphKX4Mh/bin', '/usr/local/bin', '/bin', '/usr/bin', '/usr/local/sbin', '/usr/sbin', '/var/cfengine/bin', '/home/fido/.local/bin', '/home/fido/bin'])

eb is present in /tmp/tmphKX4Mh/eb_stage1/bin but this is not in $PATH.

[boegel]

@arnold-jr OK, thanks!

With some help from @vanzod, I've traced the problem down to the location of the eb installed during stage 1 of the bootstrap procedure not being added to $PATH.

This is a hard requirement now since the change in #2248, which was included in v3.3.1 but not in v3.3.0. When no eb is available in $PATH, which returns None, and resolve_path doesn't deal well with None (cfr. your debugging session).

I'll look into a fix for the bootstrap script, but we should probably also avoid making this such a hard requirement (just to fix the 'regression' that this introduced).

I retest the bootstrap script right after every release, but didn't bump into this problem because I had another eb in my $PATH somewhere...

[boegel]

Thanks for the detailed debugging information @arnold-jr, and for helping with figuring this out @vanzod.

Bootstrapping EasyBuild v3.3.1 is broken in the current context:

• there is no eb command available through $PATH already (the actual location doesn't matter)
• the setuptools version available on the system is considered to be too old (< 0.6c11), and hence stage 0 of the bootstrap procedure is not skipped

An update to the bootstrap script to fix this problem is available in #2281. Once that is merged, it should suffice to re-download the bootstrap script as outlined in http://easybuild.readthedocs.io/en/latest/Installation.html#bootstrapping-procedure.

An accompanying change on top of #2248 will be made as well to fix the regression that caused this problem to occur.

[arnold-jr]

@boegel and @vanzod Thanks for all your help!