Rbundle build extremely slow even with --skip (655 exts ~27min)

[BenjaminHCCarr]

I am building a 655 extension R bundle but it is taking ~27 minutes to run --skip --rebuild even though the only thing needed is to rerun sanity checks:

ran effectively:

eb R-bundle-community-3.6.3-foss-2019a-R-3.6.3.eb # fails after 655 modules
eb R-bundle-community-3.6.3-foss-2019a-R-3.6.3.eb --module-only
eb R-bundle-community-3.6.3-foss-2019a-R-3.6.3.eb --skip --rebuild

the --skip --rebuild stage taking 27 minutes

grep -i class easybuild-R-bundle-community-3.6.3-20200922.163759.kaFbB.log | grep -i instance
== 2020-09-22 16:39:48,950 easyconfig.py:1790 INFO Successfully obtained EB_Rmpi class instance from easybuild.easyblocks.rmpi
== 2020-09-22 17:02:09,211 easyconfig.py:1790 INFO Successfully obtained EB_Rserve class instance from easybuild.easyblocks.rserve

[BenjaminHCCarr]

Finally got a full extension build through:

<snip 652 packages>
== skipping extension argparse
== skipping extension intergraph
== skipping extension ggnetwork
== restore after iterating...
== postprocessing [skipped]
== sanity checking...
== cleaning up...
== creating module...
== comparing module file with backup /opt/scp/unsupported/scpops/SCPAPP-2383_R-3.6.3-community-bundle/modules/all/R-bundle-community/3.6.3-foss-2019a-R-3.6.3.bak_20200922171610_19876; no differences found
== permissions...
== packaging...
== COMPLETED: Installation ended successfully (took 32 min 23 sec)
== Results of the build can be found in the log file(s) /opt/scp/unsupported/scpops/SCPAPP-2383_R-3.6.3-community-bundle/software/R-bundle-community/3.6.3-foss-2019a-R-3.6.3/easybuild/easybuild-R-bundle-community-3.6.3-20200922.174820.log
== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/eb-jnCchQ/easybuild-zWyXFD.log* have been removed.
== Temporary directory /tmp/eb-jnCchQ has been removed.

real    32m37.794s
user    29m50.290s
sys     1m50.925s

[boegel]

@BenjaminHCCarr This issue should be largely fixed with the changes in #3498.

There's more room for improvement, but with less performance benefit (and requiring significantly more work to implement)...

[boegel]

Update on this: there's some work-in-progress on support for installing extensions in parallel, especially R packages: see #3667 and easybuilders/easybuild-easyblocks#2408 .

It's already functional, but it needs a little bit more love before we can include this in an EasyBuild release (perhaps we should make it an experimental feature to encourage testing).