test timeouts for netCDF 4.9.0

[Ghepardo]

(I tried raising this by e-mailing easybuild-request@lists.ugent.be, but I received no responses.)

I am getting a consistent build error when attempting to build netCDF-4.9.0-gompi-2022a.eb. This is a dependency for the R 4.2.1 build. I have tried this using both 10 CPUs and 20 CPUs on the build system, with the same result, which is a failure in the MPI tests. I attach copies of the build logs. Can anyone shed any light on what's going wrong here?
easybuild-FMwVd0.log
easybuild-netCDF-4.9.0-20220802.094732.cKaNQ.log
easybuild-netCDF-4.9.0-20220802.094732.cKaNQ_test_report.md

[Micket]

161 - nc_test4_run_par_test (Timeout)

so it times out.
My first guess is what this was perhaps build from a node with an interactive job, doing something like

srun -n 20 bash
eb netCDF--etc...eb 

If so, be aware that this will not work and will typically fail with the same reason, the srun that's running the job interactively blocks the MPI tests from launching (until it's complete, which, it never will be, because it's waiting for the mpi job to run)
This happens on basically everything that runs MPI as part of their tests.

[Ghepardo]

Thanks Mikael. You are absolutely right that I was running this on a node where I'd opened a session using srun. However, I have now retried this by connecting to the same node using a simple SSH session, such that all 32 of the node's CPUs were available to me, and I got the same error - a timeout on test 161. I attach copies of the new logs. Should I have done anything special in setting up my SSH session? Or is there an srun option that would allow the use of MPI within an interactive session running under it? Regards Ed

[Micket]

However, I have now retried this by connecting to the same node using a simple SSH session, such that all 32 of the node's CPUs were available to me,

Just to be doubly sure, can you write a batch script that simply submits this job to the queue? i.e.

#!/usr/bin/env bash
#SBATCH -n 20

module load EasyBuild
eb netCDF-4.9.0-gompi-2022a.eb -r

[Micket]

Regarding the other questions:

Should I have done anything special in setting up my SSH session?

if this happens to be a node where you are currently running a job, and ssh'ing into it with PAM's slurm_job_adopt active, then it makes no difference. SLURM still considers it busy.

Or is there an srun option that would allow the use of MPI within an interactive session running under it?

there are surely flags, but, it's not very convenient to try to pass those down.
I've hacked around this a few occasions when i just wanted to play around and test stuff, by simply unsetting SLURM* environment variables in the shell, which will let mpirun ignore SLURM and just run (I don't know which ones are strictly required, i just unset all of them)

unset SLURMD_NODENAME
unset SLURM_CLUSTER_NAME
unset SLURM_CONF
unset SLURM_CPUS_ON_NODE
unset SLURM_DISTRIBUTION
unset SLURM_GTIDS
unset SLURM_JOBID
unset SLURM_JOB_ACCOUNT
unset SLURM_JOB_CPUS_PER_NODE
unset SLURM_JOB_GID
unset SLURM_JOB_ID
unset SLURM_JOB_NAME
unset SLURM_JOB_NODELIST
unset SLURM_JOB_NUM_NODES
unset SLURM_JOB_PARTITION
unset SLURM_JOB_QOS
unset SLURM_JOB_RESERVATION
unset SLURM_JOB_UID
unset SLURM_JOB_USER
unset SLURM_LAUNCH_NODE_IPADDR
unset SLURM_LOCALID
unset SLURM_NNODES
unset SLURM_NODEID
unset SLURM_NODELIST
unset SLURM_NPROCS
unset SLURM_NTASKS
unset SLURM_PRIO_PROCESS
unset SLURM_PROCID
unset SLURM_PTY_PORT
unset SLURM_PTY_WIN_COL
unset SLURM_PTY_WIN_ROW
unset SLURM_SRUN_COMM_HOST
unset SLURM_SRUN_COMM_PORT
unset SLURM_STEPID
unset SLURM_STEP_GPUS
unset SLURM_STEP_ID
unset SLURM_STEP_LAUNCHER_PORT
unset SLURM_STEP_NODELIST
unset SLURM_STEP_NUM_NODES
unset SLURM_STEP_NUM_TASKS
unset SLURM_STEP_TASKS_PER_NODE
unset SLURM_SUBMIT_DIR
unset SLURM_SUBMIT_HOST
unset SLURM_TASKS_PER_NODE
unset SLURM_TASK_PID
unset SLURM_TOPOLOGY_ADDR
unset SLURM_TOPOLOGY_ADDR_PATTERN
unset SLURM_UMASK
unset SLURM_WORKING_CLUSTER

[Ghepardo]

Thanks Mikael. I'm afraid nothing seems to make any difference: I have tried your sbatch script, and I have even tried turning the Slurm daemon off on our build machine, but it all gives the same result: a failure at test 161. The user that is attempting this build has no SLURM* environment variables set: 'env | grep -i slurm' returns nothing.

[Micket]

It's best if you answer comments on github instead of directly on the email; your client adds a bunch of extra stuff that you that ends up here, making things very hard to read (i edited those out of your comments).

I think you need to dig deeper here;

eb --dump-env netCDF-4.9.0-gompi-2022a.eb
source netCDF-4.9.0-gompi-2022a.env
cd /path/to/build/dir/netCDF-4.9.0-gompi-2022a/...

and try to run the test commands manually, look at what is happening with the calls (is there any load at all?), perhaps strace'ing the test.

If think there is just something wrong with this one test (I don't know if there are other mpi tests that pass.. maybe?) then you can also just skip the test step.

[Ghepardo]

Noted (and sorry) re response method.

I've created that .env file and sourced it, and I've found a directory /home/apps/eb/build/netCDF/4.9.0/gompi-2022a. This contains no files and 2 subdirectories, easybuild_obj and netcdf-c-4.9.0. I'm not clear what you're asking me to do from here (I have no idea how to run the test commands manually).

[Micket]

I've created that .env file and sourced it, and I've found a directory /home/apps/eb/build/netCDF/4.9.0/gompi-2022a. This contains no files and 2 subdirectories, easybuild_obj and netcdf-c-4.9.0.

yes. the build and the source directory respectively.

I'm not clear what you're asking me to do from here (I have no idea how to run the test commands manually).

well i'm just suggesting the steps you can do in order to dig down and debug this problem yourself as to why this one test fails.
You can find each configure/build/test commands shat EB does by e.g. grepping after run.py in the log file. Otherwise, there is no general answer, just generic problem solving.
I have never used netCDF, and, wouldn't surprise me if no other maintainers have either, like with 99% of the software we build. We are unlikely to find the cause given noone else seem to have managed to reproduce any issue.

[Ghepardo]

Hmm, I'm pretty lost here. The log file shows:
== 2022-08-09 12:52:18,019 run.py:233 INFO running cmd: OMPI_MCA_rmaps_base_oversubscribe=1 make test

...before the start of the list of the 227 tests. I can see there's a Makefile, and it contains 2 "special rules" for "target test". One of these specifies a ".../ctest" command with arguments, which also appears in the log file. Are you saying I should try to run that command manually? If I do, won't I simply get the same result that Easybuild does?

I am not sure about getting strace output. If I managed this, wouldn't it be voluminous for those 227 tests? And searching for a timeout error in it would be like looking for a needle in a haystack?

I think you are saying mine is the only Easybuild site that has reported this error, and that other sites have built this with no problem. If that's the case, that suggests the problem is with our site, and I do understand that you can't spend your time trying to help find such a problem. But it does present us with an issue, given that this is a dependency for R, which is a very important package for us. We may have to look for alternative ways of obtaining R.

I do also understand that the Easybuild maintainers can't be responsible for maintaining easyconfigs submitted by the community. It is a shame that Easybuild does not seem to have any mechanism for ensuring that the submitter provides some level of support.

[Micket]

...before the start of the list of the 227 tests. I can see there's a Makefile, and it contains 2 "special rules" for "target test". One of these specifies a ".../ctest" command with arguments, which also appears in the log file. Are you saying I should try to run that command manually? If I do, won't I simply get the same result that Easybuild does?

Well that's how i start tackling most software problems, start be reproducing it. Then continue to dig deeper, and deeper, and deeper. Like checking if there were any other tests that ran MPI and somehow worked, seeing what actually happens on the system while the failing test is running (full load? waiting for MPI init?). Is there some stderr/stdout from the failing test ctest -v? Maybe turn on some OpenMPI debug environment variables to see if they reveal anything.

I am not sure about getting strace output. If I managed this, wouldn't it be voluminous for those 227 tests? And searching for a timeout error in it would be like looking for a needle in a haystack?

No, just the exact command (whatever ctest actually runs, ctest -v -R failing_test_name, and i wouldn't start of with strace. I'd just see if it was even doing anything, i.e. is it just stuck at MPI init, or is it stuck in a loop at full load? then just proceed from there.
Basically just standard debugging.

I think you are saying mine is the only Easybuild site that has reported this error, and that other sites have built this with no problem.

So far yes (as far as I've seen). Whoever merges PRs makes sure there is at least 1 successfull build report from EB, though most of the common ones probably sees +10 different places before it's shipped.
But this is a newer toolchain, not as many outside of the maintainers might have built this yet

I do also understand that the Easybuild maintainers can't be responsible for maintaining easyconfigs submitted by the community. It is a shame that Easybuild does not seem to have any mechanism for ensuring that the submitter provides some level of support.

Then noone would volunteer to submit anything, so we certainly won't ever hold anyone to that (how could we? noone is getting payed here). It's all just a collective best effort where everyone submits patches, fixes, improvements.

[Ghepardo]

OK thanks. My first attempt to run "make test" manually did in fact produce slightly more information than appeared in the Easybuild log file: it suggested I reran with some additional arguments, "--rerun-failed --output-on-failure". Once I'd worked out where to put these (it was in the Makefile), I reran the "make test".

Unfortunately, this produced no further useful information: it seems to be a timeout in the "Parallel Performance Test for NASA".

I did attach strace to one of the PIDs participating in this - one with no further children. All I got was the following repeated over and over again:

fcntl(16, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
fcntl(16, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
pwrite64(16, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 40, 28208) = 40

Only the penultimate number (28208 in the extract above) varied between repetitions. I wouldn't know where to begin in making any use of that. This brings me back to my point about maintainers: it might mean something to someone who (a) is a developer and (b) knows this product. I am neither of those things myself.

If you've got any further suggestions I'd be interested to hear them, but I sense we've probably got as far as we can with this. I shall have to seek alternatives to Easybuild. But I'd be reluctant to close this issue, because it's not been resolved.

[Ghepardo]

Sorry, I didn't address a couple of the points you made:

Like checking if there were any other tests that ran MPI and somehow worked, seeing what actually happens on the system while the failing test is running (full load? waiting for MPI init?).

• I'm afraid I wouldn't know how to identify those tests that ran MPI and those that didn't, and, if I did, I wouldn't know what to do with the information.
• The system did not seem to be very loaded while the failing test was running - certainly far less than 100% CPU usage. I'm afraid I wouldn't know how to identify whether a system was waiting for MPI init.

[Ghepardo]

I have noticed that this build will succeed for me if I comment out the following line in the easyconfig:

runtest = 'test'

I have no way of telling what this line does: "runtest" does not appear in the Makefile, but it does appear in literally hundreds of other files in the build directory. This is a new line in this easyconfig, compared to the previous version.

@branfosj I see you committed this easyconfig. Is it possible to tell who submitted it to you, so that we may pick their brains on "runtest"?