EasyBuild v4.0

[boegel]

This issue keeps track of what changed are planned for EasyBuild v4.0, in which backwards-incompatible changes will be made compared to EasyBuild v3.x.

ETA for EasyBuild v4.0: summer 2019 (subject to change!)

---

Codebase

• drop support for using EasyBuild on top of Python 2.6 (?) (not dropping Py2.6 support for now)

  • after consulting with community via user survey?
  • only if Python 2.6 support really gets in the way (mainly w.r.t. also supporting Python 3)
  • (turns out this is not needed, we can keep supporting Python 2.6 without too much hassle)

• drop deprecated functionality (see log.deprecated statements)

  • (no action needed here, log.deprecated("...", '4.0') statements will now simply result an error

• support running EasyBuild on top of (both Python 2.7 &) Python 3 (because 2+3=4) [DONE]

  • see support Python 3.x easybuild-framework#133
  • blocker: vsc-base should be compatible with Python 3 (cfr. start looking into making vsc-base Py3 compatible hpcugent/vsc-base#231, )
    • fixed by ingesting vsc-base & vsc-install into framework, see ingest vsc-base & vsc-install easybuild-framework#2708
  • see PRs marked with python3 label: https://github.com/easybuilders/easybuild-framework/labels/python3

• avoid setuptools dependency [DONE]

  • only needed for pkg_resources.declare_namespace used to spread easybuild namespace across multiple directories
  • also relevant for --include-* support, but can be worked around? (OK)
  • central idea is to basically just replace the use of pkg_resources.declare_namespace by pkgutil.extend_path, which seems to be the one true way (tm), cfr. https://packaging.python.org/guides/packaging-namespace-packages/#pkgutil-style-namespace-packages ("This is the recommended approach for the highest level of compatibility.")
  • see drop requirement for setuptools as runtime dependency easybuild-framework#2836 + only import from pkg_resources (provided by setuptools) to check GC3Pie version easybuild-framework#2837

---

Packaging & installation

• try to avoid requiring setuptools to install EasyBuild (try to avoid requiring setuptools to install EasyBuild easybuild-framework#2941) (DONE)
• update bootstrap script to be able to install EasyBuild 4.0 (see update bootstrap script to support installing EasyBuild v4.0 easybuild-framework#3017)

---

Documentation

• overview of (breaking) changes
  • see https://easybuild.readthedocs.io/en/develop/EasyBuild4-overview-of-changes.html
• release notes

---

Features

• more flexibility w.r.t. dependency versions?
  • unclear how at this point...
• (fat) easyconfigs in YAML syntax (.yeb)?
  • unlikely, unless someone has time to work on it...
• drop exts_classmap? (will look into deprecating this in EasyBuild 4.x)
  • see exts_classmap should be removed, and replaced by an ext_option option easybuild-framework#506
• replace dummy toolchain with system toolchain
  • deprecate dummy toolchain to trigger deprecation warning on use, to be removed in EasyBuild 5.0
  • see Please rename the dummy toolchain easybuild-framework#1181
  • done, see deprecate 'dummy' toolchain, replace with 'system' toolchain easybuild-framework#2877
  • also add alias for system toolchain to EasyBuild 3.x: support using 'system' as alias for 'dummy' toolchain easybuild-framework#2960
• [ ] deprecate runtest in favor of testopts, see deprecate runtest in favor of testopts (was: Add pretestopts and testops) easybuild-framework#2868
• stable support for --containerize (needs more time)

---

Easyblocks

• PythonPackage

  • switch to installing Python packages with pip by default (rather than setup.py install, i.e. setuptools) (not switching the default, but will keep requiring use_pip = True in contributed easyconfigs)
  • cfr. https://docs.python.org/2/installing/index.html
  • needs to be re-evaluated...
  • enable download_dep_fail by default, to ensure that no dependencies are automatically downloaded by whichever Python installation tool is being used (already done in PythonBundle)

• custom easyblock for OpenMPI (done, see add custom easyblock for OpenMPI easybuild-easyblocks#1789)

---

Easyconfigs

• archiving of old easyconfig files
  • anything using ictce toolchain (already deprecated in EasyBuild 3.8.0)
    • archive easyconfigs using deprecated ictce or intel (< 2016a) toolchain easybuild-easyconfigs#8558
  • anything using goolf/1.4.10 (already deprecated in EasyBuild 3.8.0)
    • archive easyconfigs using deprecated goolf(c) toolchain easybuild-easyconfigs#8557
  • also oldest versions of intel and foss toolchains? (already deprecated in EasyBuild 3.8.0)
  • < intel/2016a: archive easyconfigs using deprecated ictce or intel (< 2016a) toolchain easybuild-easyconfigs#8558
  • < foss/2016a: archive easyconfigs using a gompi/foss toolchain older than {gompi,foss}/2016a easybuild-easyconfigs#8585 (EasyBuild v3.9.4)
  • archive easyconfig for old software versions:
  • archive ancient versions of GC3Pie/GCC/OpenMPI/ORCA easybuild-easyconfigs#8586 (EasyBuild v3.9.4)
  • try to avoid losing support for particular software packages while doing this...
• "basic" Python at GCCcore + SciPy bundle?
  • done, see {lang}[GCCcore/8.2.0] Python v3.7.2 easybuild-easyconfigs#7898 (Python 3.7.2), {lang}[GCCcore/8.2.0] Python v2.7.15 easybuild-easyconfigs#7821 (Python 2.7.15), {lang}[foss/2019a] SciPy-bundle v2019.03 w/ Python 2.7.15 + 3.7.2 easybuild-easyconfigs#7921 (SciPy-bundle)
• adopt ComputeCanada's approach to installing Python packages for multiple Python versions
  • done, see add support for multi_deps easyconfig parameter (syntactic sugar for iterative list of builddependencies) easybuild-framework#2813
• avoid warnings w.r.t. naming of local variables (all done!)
  • cfr. recommended naming scheme for local variables in warn if non-single-letter local variables not named 'local_*' or '_*' are used + enhance --fix-deprecated-easyconfigs to rename them easybuild-framework#2938
    • docs update @ add documentation on use of local variables in easyconfig files #527

[cclauss]

You folks do know that https://snarky.ca/stop-using-python-2-6 was written 3 1/2 year ago?

[boegel]

We're fully aware that Python 2.6 is obsolete, and has been for a long time, yet we opt to keep supporting it for now.
The main reason is that a small (& shrinking) part of the EasyBuild user base (~16% early 2018, ~6% early 2019 according to our user survey) is still using an OS that has Python 2.6 as the default system Python (for example CentOS 6.x). Although upgrading that to Python 2.7 is definitely doable, many people opt out of that for the sake of maintaining system stability.

I re-evaluated whether or not to keep support for Python 2.6 while working on the port to Python 3.x (in the 4.x branches), and it turned out to be very little effort to do so (since we're definitely going to keep support for Python 2.7 for a while, despite it going EOL end of 2019, for very similar reasons).

The main benefit would be being able to remove small parts of code that are required to maintain Python 2.6 support, which we haven't touched in years (so they have very little impact on development), and being able to clean up the mess (a bit) in the requirements.txt for the EasyBuild framework component (cfr. your PR easybuilders/easybuild-framework#2874).

In my view, neither of those reasons are compelling enough to drop Python 2.6 support and annoy a minority of our user base along the way, especially since it has little impact on other users or ongoing development.

[cclauss]

@hugovk Would you be willing to show us how to get the Python versions being used to pip install:

• easybuilders/easybuild
• easybuilders/easybuild-framework
• easybuilders/easybuild-easyconfigs
• easybuilders/easybuild-easyblocks

Each of these repos has a 4.x branch that will add Python 3 support. The current users should all be on legacy Python and Python 2.4 is still supported. It would be interesting to know which versions of Python are being used to install the EasyBuild tools. Thx.

[boegel]

@cclauss Are you looking for this? https://pypistats.org/packages/easybuild

The Python 2.4 is a mistake in the setup.py in https://github.com/easybuilders/easybuild, I'll get that fixed, since it's actually not supported anymore (Python 2.6 is current minimal required version).