Improve memory-allocation during single-model training

[uermel]

octopi seems to show memory spikes during training which make it necessary to over-allocate ressources for training jobs. Specifically, during the validation step (every 10 epochs), memory seems to briefly spike from 12 GB to >40 GB for a 6-class training run with 6 train and 3 validation tomograms.