Cron test script

[kgerheiser]

setup_cron.sh sets variables and is the entry point of cron. It then downloads hpc-stack and runs build-and-test.sh with the options given by setup-cron.sh.

Then, build-and-test.sh builds the develop branch hpc-stack (along with the debug version of ESMF), and then it checks out the ufs-weather-model and runs the regression test.

A hash is saved after each build so that it is only re-built when there are changes.

A nicely formatted email is sent out with the results of the test, for example:

==============================================================================
Mon Dec 14 03:33:19 UTC 2020
hpc-stack hash: bbbea9dec4768febe3feb7a71a0b1268a5cc80b7
ufs hash: 9bd6b59b3c5a230558430f238e32ca60d2844e57

hpc-stack build: SUCCESS
hpc-stack log: /home/Kyle.Gerheiser/.hpc-stack/logs/hpc-stack_12-14-2020-00:53.log
UFS regression test: PASS
ufs log: /home/Kyle.Gerheiser/.hpc-stack/logs/ufs_12-14-2020-02:29.log

==============================================================================

Fixes #81

[edwardhartnett]

Awesome!

[kgerheiser]

Ready for review now.

I tested it out on Hera over the weekend. I fixed some odds and ends, added the UFS tests, and added a nicely formatted email with the results.

I'll install it across our machines this week.

It will be installed in a public nightly-develop folder every day (if there are changes in hpc-stack) alongside the versioned release so that anyone can try out the latest and greatest.

[aerorahul]

some suggestions for modularizing

[kgerheiser]

How about this?

I factored out the testing into its own script, and created another script test-applications.sh which calls the individual test scripts.

[aerorahul]

I like this much better.
One suggestion is to also turn on set -x. It produces a lot of output, but is worth it for debugging.
Do however set +x around any module commands.

[kgerheiser]

I removed the if-block and fixed the typo.

I added a special return code build-hpc-stack.sh if the stack isn't built because the hash hasn't changed so that the top-level script knows not to run the tests. Is there a better way to handle that case?

I enabled -x in build-hpc-stack.sh and redirected it to a log.

I don't want to get an email everyday with all that output. It's available in the logs.

[kgerheiser]

Done

[aerorahul]

looks great.
Thanks for setting this up.

[aerorahul]

I noticed you are using -l so that $MODULESHOME/init/bash is sourced because that is the case on HPC's.
When a sudo executes this script, there will be no login shell.

My recommendation is to remove -l and do a source $MODULESHOME/init/sh wherever modules are being loaded/used/etc.

[kgerheiser]

I'll try it out, but I don't think $MODULESHOME will even be set. You'll have to source the absolute directory like /apps/contrib/lmod/lmod/init/bash which is a pain to set for each machine.