Provide a TMPDIR for the container

[jennan]

Currently /tmp is in the tmpfs and It's quite small, fill up quick and causes issues (@CallumWalley can you detail on these issues?).

One possibility is to create a TMPDIR in the nobackup folder of the project but:

• we need to retrieve project number, won't work if run from login (which we never promised to support, so it is likely not an issue...)
  • adapt jupyter to pass an environment variable for this
  • or retrieve it from JOB_ID, e.g. scontrol show jobid="$SLURM_JOB_ID" -o | sed -r "s/.*Account=(\S+).*/\1/"
• make sure that there is enough space? and gracefully handling failure
• cleanup
  • but that is risky in my opinion (rm -rf in nobackup)
  • unless we cleanup from within the container (and only bind this folder)... to be on the safe side
  • or leave it to the normal nobackup policy (but that would generate unnecessary emails)

[jennan]

Next release of Jupyter will expose JUPYTER_JOB_ACCOUNT which should help to create the TMPDIR in the right space.

[MattBixley]

tmpdir fills up for
install.packages()
creating large objects/functions eg using raster files

error that returns is a failure to write message