Adding support for the use of `--from-pr` in EasyStack files

[casparvl]

Goal
With #4021 we can parse EasyConfig-specific options in an EasyStack file. However, nothing is done with those parsed options yet.

One of the options of primary importance would be to be able to use --from-pr, so that EasyStack files can be also include software that has just been PR-ed, but is not part of an EasyBuild release yet.

Required adaptations
In this issue, I want to discuss the approach and issues surrounding --from-pr support in EasyStack files.

In #3513 (comment) we have identified a couple of places in the code that would definitely need adaptation for --from-pr support in EasyStack files.

• det_easyconfig_paths: gets the original paths for the EasyConfigs specified in the EasyStack file. At this stage, no dependency resolution is done yet. However, this function should resolve the list of EasyConfig filenames to paths to EasyConfig files. As such, it needs to be aware of --from-pr, so that EasyConfig files that are part of PRs are resolved to the tempdir where the PR is checked out.
• resolve_dependencies: receives a list of EasyConfig objects as argument, and then resolves the dependencies for those. Since this function is responsible for locating the paths to EasyConfig files for these dependencies, it has to be aware of --from-pr. It finds EasyConfigs from dependencies by calling robot_find_easyconfig, which in turn relies on build_option('robot_path'), which in turn is set during initial configuration of eb by set_up_configuration. Technically, this means the robot_path build option should be updated in order for robot_find_easyconfig to be able to find the EasyConfig in a tempdir that stores a PR.

Challenges
It is not completely clear what the desired behaviour should be. Let us consider the following EasyStack file:

# my_easystack.yaml
easyconfigs:
  - A.eb
    options: {
      'from_pr': 1235,
    }
  - B.eb

Where both A and B have a dependency on C, and PR 1234 includes an updated C.eb. Clearly, we can only install one version of C, so
eb --easystack my_easystack.yaml --robot
should pick one C.eb, install it, and then install A and B. But: which C.eb should it pick? Technically, our definiton above said "We want to use PR 1234 for A.eb (and it's dependency C)", but it also said "Install B.eb (and it's dependency C) from EasyConfigs found on the default robot search path". Thus, this essentially refers to two different versions of C. That, clearly, is not an option.

The reason this question is important is that we should wonder: how should resolve_dependencies find EasyConfigs? Should it realize that C is a dependency of A, and that A had a PR set, and therefore should it search the tempdir in which that PR was checked out for C.eb? What if we had listed B.eb first? Then, it would have already resolved C.eb to its local version.

One option would be that we simply add all from_pr's specified for specific EasyConfigs to the global build_option('from_pr') early on. That way, the behaviour is predictable (and the implementation is actually easier): C.eb will always resolve to the version from PR 1234.

[casparvl]

Note that actually the same problem applies for more, if not all options. Specifying an EasyConfig-specific --robot-path would suffer the same issue: what if the 'C.eb' on the --robot-path specified for A.eb is different from the C.eb on the default --robot-path (which should be used according to B.eb? And what if a different --optarch is defined for A.eb, what should be used for C.eb?

Another way to approach this would be to say: any of these EasyConfig-specific options apply to A.eb and only to A.eb. That's quite annoying for a --from-pr though, since that would mean you have to explicitely include C.eb in this EasyStack file, with the 'from_pr': 1235 option specified as well. That might be rather counter-intuitive, and quite different from what people might expect when running EasyBuild without EasyStack files...

[casparvl]

I've made a (very early) attempt here: #4052

The behaviour in terms of the challenge I described above is 'first come, first serve', meaning that it will resolve C.eb from PR 1235, simply because that is encountered first. This means that

eb --easystack my_easystack.yaml --robot with

# my_easystack.yaml
easyconfigs:
  - A.eb
    options: {
      'from_pr': 1235,
    }
  - B.eb

Would essentially behave the same as:

eb A.eb --from-pr 1235 --robot
eb B.eb --robot

Clearly, the first eb invocation here will resolve dependency C.eb to the one from PR 1235, and install software C based on that. To me this feels like a reasonably intuitive appproach.

[casparvl]

Summary from a discussion with @boegel :

• Move loop from det_easyconfigs_path to main.py
• At line 355 in main here:
• From here, use an if, ifelse, else, where the if checks for installing an updated framework, the ifelse checks for using easystack,
• In easybuild/tools/options.py: init_build_options creates a singleton. Will give an error now. We need to whipe that if we want to re-call set_up_configuration

Dummy code loop for main:

if easystack:
  for x in orig_paths:
     validate_command_opts(args, easystack.ec_opts[x]) # Validates if options in easystack.ec_opts exist in args
  for x in orig_paths:
    wipe_build_options()  # Does BuildOptions.__class__._instances.clear()
    wipe_cache()  # Whipes the easyconfigs cache _easyconfigs_cache.clear()
    new_args = merge_command_line_opts(args, easystack.ec_opts[x], validate = False) # overwrite any command lines opts with options specific for this EasyConfig
    set_up_configuration(new_args)
    rest_of_main([x])  # Basically run all the functions in the rest of main
else:
  rest_of_main(orig_paths)

[casparvl]

Fixed in #4057 , closing this issue