Locking mechanism to avoid competing builds to run

[mboisson]

At the very beginning of a build, EasyBuild should create the installation directory and touch a file lock in it, and delete it at the end. This is to avoid having two builds that are happening at the same time for the same installed path.

[boegel]

Currently the installation directory is only created relatively late in the installation process (as a part of the install step).
But we could touch a temporary file somewhere else in the install tree, or name it .<installdir>.lock?

[mboisson]

In our case, it could be, since we are using a single build node, but as soon as you start using multiple nodes to build, you want it to be on a shared filesystem, and you cannot assume the build directory to be on the shared filesystem. Where would you put this file ?

[damianam]

If you are using multiple nodes to build software and installing in non-shared directories (let's say, via RPMs), then probably the best would be to create the lock in the directory where the easyconfigs are coming from. Of course if that doesn't come from a shared directory you are pretty much on the same point.

In general, I think it makes sense to do .<installdir>.lock. Optionally one could make it configurable with --lockpath or something like that, but I think that'd be a corner case that I am not sure would affect anyone.

[akesandgren]

Yes please.... and if there is a lock one need to be able to define the behavior, wait or die.
Mostly when I push build jobs to the cluster I sometime accidentally run eb --job multiple times with targets that share no-yet-built dependencies. I do not want it to fail in that case but wait until the lock is gone and then detect that it is already built and exit gracefully with a SUCCESS state.

[mboisson]

Hum... waiting does not make sense to me. It could potentially wait indefinitely... There is no easy way to guarantee that the lock is deleted in case of failure.

[akesandgren]

Since a normal batch job has a timelimit, so that's not a problem. And if interactive you'll probably notice sooner or later. Especially if it prints a warning, and the default should be to not wait.

[mboisson]

Mmm, ok. Presumably, if we are speaking about sleeping, we need to define a rate at which the presence of a file would be checked. Should this be configurable, or a constant (say once per minute) ?

[mboisson]

I just opened a PR for this. Now, I know you will ask for tests for this, but I have no damn idea how to write a test for something that requires two parallel builds of the same thing at the same time to test the feature.

[mboisson]

This has been closed by the merged PR.