PyTorch-1.10.0-foss-2021a-CUDA-11.3.1.eb build failure on A100 GPU within Singularity container

[sassy-crick]

I am having the problem to build the recent Pytorch versions using either PyTorch-1.10.0-foss-2021a-CUDA-11.3.1.eb or PyTorch-1.9.0-fosscuda-2020b.eb. The builds are done by requesting an interactive session from SLRUM like this srun -p 4gpu --nodes=1 --ntasks-per-node=36 --mem=480G --gres=gpu:1 --time=7-0:0 --pty /bin/bash, the Singularity container is started with including --nv , any SLURM related environment variables are removed from the container, and the actual build is done like this eb -d --cuda-compute-capabilities=8.0 PyTorch-1.10.0-foss-2021a-CUDA-11.3.1.eb.

Thanks to the help of Kenneth the error messages are:

======================================================================
ERROR: test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
DDP communication hook can only be registered once. This test validates whether
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 418, in wrapper
    self._join_processes(fn)
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 637, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 687, in _check_return_codes
    raise RuntimeError(
RuntimeError: Process 0 terminated or timed out after 610.0295519828796 seconds

======================================================================
ERROR: test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
This unit test makes sure that register_comm_hook properly checks the format
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 418, in wrapper
    self._join_processes(fn)
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 637, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 687, in _check_return_codes
    raise RuntimeError(
RuntimeError: Process 0 terminated or timed out after 610.0408778190613 seconds

======================================================================
ERROR: test_round_robin (__main__.ProcessGroupGlooTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 418, in wrapper
    self._join_processes(fn)
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 637, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 687, in _check_return_codes
    raise RuntimeError(
RuntimeError: Process 0 terminated or timed out after 605.0606961250305 seconds

======================================================================
ERROR: test_round_robin_create_destroy (__main__.ProcessGroupGlooTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 418, in wrapper
    self._join_processes(fn)
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 637, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 682, in _check_return_codes
    raise RuntimeError(error)
RuntimeError: Process 0 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 536, in run_test
    getattr(self, test_name)()
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 420, in wrapper
    fn()
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 2850, in wrapper
    return func(*args, **kwargs)
  File "/dev/shm/easybuild/eb-5azo6i3u/tmpxe0ioidc/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 2850, in wrapper
    return func(*args, **kwargs)
  File "/dev/shm/easybuild/PyTorch/1.10.0/foss-2021a-CUDA-11.3.1/pytorch/test/distributed/test_c10d_gloo.py", line 1438, in test_round_robin_create_destroy
    pg = create(num=num_process_groups, prefix=i)
  File "/dev/shm/easybuild/PyTorch/1.10.0/foss-2021a-CUDA-11.3.1/pytorch/test/distributed/test_c10d_gloo.py", line 1424, in create
    [
  File "/dev/shm/easybuild/PyTorch/1.10.0/foss-2021a-CUDA-11.3.1/pytorch/test/distributed/test_c10d_gloo.py", line 1425, in <listcomp>
    c10d.ProcessGroupGloo(
RuntimeError: Wait timeout

----------------------------------------------------------------------
Ran 85 tests in 2073.886s

FAILED (errors=4)
distributed/test_c10d_gloo failed!

and

======================================================================
FAIL: test_thread_shutdown (__main__.TestAutograd)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/dev/shm/easybuild/PyTorch/1.10.0/foss-2021a-CUDA-11.3.1/pytorch/test/test_autograd.py", line 4305, in test_thread_shutdown
    self.assertRegex(s, "PYTORCH_API_USAGE torch.autograd.thread_shutdown")
AssertionError: Regex didn't match: 'PYTORCH_API_USAGE torch.autograd.thread_shutdown' not found in 'PYTORCH_API_USAGE torch.python.import\nPYTORCH_API_USAGE c10d.python.import\nPYTORCH_API_USAGE tensor.create\n'

----------------------------------------------------------------------
Ran 476 tests in 19.531s

FAILED (failures=1, skipped=65, expected failures=1)
test_autograd failed!

At the end of the logfile I get:

distributed/test_c10d_gloo failed!
test_autograd failed!

The actual build works and it is failing at the testing stage:

== building...
== ... (took 7 hours 30 mins 26 secs)
== testing...
== ... (took 3 hours 50 mins 23 secs)
== FAILED: Installation ended unsuccessfully (build directory: /dev/shm/easybuild/PyTorch/1.10.0/foss-2021a-CUDA-11.3.1): build failed (first 300 chars): cmd "export 
PYTHONPATH=/dev/shm/easybuild/eb-5y1vttp7/tmpl5jqz6uw/lib/python3.9/site-packages:$PYTHONPATH &&  cd test && PYTHONUNBUFFERED=1 
/apps/easybuild/software/Python/3.9.5-GCCcore-10.3.0/bin/python run_test.py --continue-through-error  --verbose -x distributed/elastic/utils/distributed_test di (took 11 hours 20 mins 59 
secs)
== Results of the build can be found in the log file(s) /dev/shm/easybuild/eb-5y1vttp7/easybuild-PyTorch-1.10.0-20220104.113117.sbDKU.log

ERROR: Build of /apps/easybuild/software/EasyBuild/4.5.1/easybuild/easyconfigs/p/PyTorch/PyTorch-1.10.0-foss-2021a-CUDA-11.3.1.eb failed (err: 'build failed (first 300 chars): cmd "export PYTHONPATH=/dev/shm/easybuild/eb-5y1vttp7/tmpl5jqz6uw/lib/python3.9/site-packages:$PYTHONPATH &&  cd test && PYTHONUNBUFFERED=1 /apps/easybuild/software/Python/3.9.5-GCCcore-10.3.0/bin/python run_test.py --continue-through-error  --verbose -x distributed/elastic/utils/distributed_test di')

Kenneth suggested: This looks like it could be related, not sure: https://discuss.pytorch.org/t/gloo-nccl-connection-issues-build-from-source/137264/3

I am stuck. I am happy to provide more information if requested.
Thanks!

[branfosj]

@sassy-crick I'd suggest the following to debug this:

• Install PyTorch with --ignore-test-failure. The tests will still run, but failures will not stop the build completing
• Unpack the source
• Load the PyTorch module and any build dependencies from PyTorch-1.10.0-foss-2021a-CUDA-11.3.1.eb
• Change into the test directory
• Run python run_test.py --verbose -i [testname]
  • where [testname] is test_autograd or distributed/test_c10d_gloo
  • From there it is then attempting to see if you can see a reason why it goes wrong. It is probably worth testing it with 1, 2, and 3 GPUs in the Slurm job.

[sassy-crick]

I will do that and report back. The build takes quite a while as you know. Thanks for your suggestions!

[casparvl]

You could also try to set NVIDIA_TF32_OVERRIDE=0 and see if that makes any difference. I had a lot of failing tests in the original PR because the PyTorch tests used too small error margins when comparing test results to a reference when TensorFloat32 datatypes are used. See #14233

I don't really expect there to be more tests that fail because of this (I see no reason why I wouldn't have hit that issue), but who knows... It's easy to try and disable it - if it works afterwards, at least you've identified the problem :)

[sassy-crick]

I think I saw that somewhere as well. I stick with what @branfosj has suggested and then, once I have identified the failing tests, will add yours to the mix. It is currently at the testing stage, so bear with me!

[sassy-crick]

Update on this:
Running export NVIDIA_TF32_OVERRIDE=0 ;python run_test.py --verbose -i test_autograd with or without the NVIDIA_TF32_OVERRIDE=0 results in this error:

======================================================================
FAIL: test_reentrant_parent_error_on_cpu_cuda (__main__.TestAutogradDeviceTypeCUDA)
----------------------------------------------------------------------
RuntimeError: grad can be implicitly created only for scalar outputs

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 1396, in wrapper
    method(*args, **kwargs)
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 1396, in wrapper
    method(*args, **kwargs)
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_device_type.py", line 368, in instantiated_test
    result = test(self, **param_kwargs)
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_device_type.py", line 881, in only_fn
    return fn(slf, *args, **kwargs)
  File "/users/hpcbuild/build/pytorch-testing/pytorch/test/test_autograd.py", line 7981, in test_reentrant_parent_error_on_cpu
    self._test_reentrant_parent_error_on_cpu(device)
  File "/users/hpcbuild/build/pytorch-testing/pytorch/test/test_autograd.py", line 7967, in _test_reentrant_parent_error_on_cpu
    torch.autograd.backward([t5.sum(), t7.sum()])
AssertionError: "Simulate error" does not match "grad can be implicitly created only for scalar outputs"

----------------------------------------------------------------------

For the other test, again with or without the NVIDIA_TF32 option I get when I am running export NVIDIA_TF32_OVERRIDE=0 ;python run_test.py --verbose -i distributed/test_c10d_gloo this output:

======================================================================
ERROR: test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
This unit test makes sure that register_comm_hook properly checks the format
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 418, in wrapper
    self._join_processes(fn)
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 637, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 687, in _check_return_codes
    raise RuntimeError(
RuntimeError: Process 0 terminated or timed out after 605.1035096645355 seconds

======================================================================
ERROR: test_round_robin (__main__.ProcessGroupGlooTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 418, in wrapper
    self._join_processes(fn)
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 637, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 687, in _check_return_codes
    raise RuntimeError(
RuntimeError: Process 0 terminated or timed out after 605.0758104324341 seconds

======================================================================
ERROR: test_round_robin_create_destroy (__main__.ProcessGroupGlooTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 418, in wrapper
    self._join_processes(fn)
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 637, in _join_processes
    self._check_return_codes(elapsed_time)
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 682, in _check_return_codes
    raise RuntimeError(error)
RuntimeError: Process 1 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 536, in run_test
    getattr(self, test_name)()
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 420, in wrapper
    fn()
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 2850, in wrapper
    return func(*args, **kwargs)
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 2850, in wrapper
    return func(*args, **kwargs)
  File "/users/hpcbuild/build/pytorch-testing/pytorch/test/distributed/test_c10d_gloo.py", line 1438, in test_round_robin_create_destroy
    pg = create(num=num_process_groups, prefix=i)
  File "/users/hpcbuild/build/pytorch-testing/pytorch/test/distributed/test_c10d_gloo.py", line 1424, in create
    [
  File "/users/hpcbuild/build/pytorch-testing/pytorch/test/distributed/test_c10d_gloo.py", line 1425, in <listcomp>
    c10d.ProcessGroupGloo(
RuntimeError: Timeout waiting for key: 1/0/0/rank_0
Process 2 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 536, in run_test
    getattr(self, test_name)()
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 420, in wrapper
    fn()
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 2850, in wrapper
    return func(*args, **kwargs)
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 2850, in wrapper
    return func(*args, **kwargs)
  File "/users/hpcbuild/build/pytorch-testing/pytorch/test/distributed/test_c10d_gloo.py", line 1438, in test_round_robin_create_destroy
    pg = create(num=num_process_groups, prefix=i)
  File "/users/hpcbuild/build/pytorch-testing/pytorch/test/distributed/test_c10d_gloo.py", line 1424, in create
    [
  File "/users/hpcbuild/build/pytorch-testing/pytorch/test/distributed/test_c10d_gloo.py", line 1425, in <listcomp>
    c10d.ProcessGroupGloo(
RuntimeError: Timeout waiting for key: 1/0/0/rank_0

Process 3 exited with error code 10 and exception:
Traceback (most recent call last):
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 536, in run_test
    getattr(self, test_name)()
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 420, in wrapper
    fn()
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 2850, in wrapper
    return func(*args, **kwargs)
  File "/apps/easybuild/software/PyTorch/1.10.0-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 2850, in wrapper
    return func(*args, **kwargs)
  File "/users/hpcbuild/build/pytorch-testing/pytorch/test/distributed/test_c10d_gloo.py", line 1438, in test_round_robin_create_destroy
    pg = create(num=num_process_groups, prefix=i)
  File "/users/hpcbuild/build/pytorch-testing/pytorch/test/distributed/test_c10d_gloo.py", line 1424, in create
    [
  File "/users/hpcbuild/build/pytorch-testing/pytorch/test/distributed/test_c10d_gloo.py", line 1425, in <listcomp>
    c10d.ProcessGroupGloo(
RuntimeError: Timeout waiting for key: 1/0/0/rank_0



----------------------------------------------------------------------
Ran 85 tests in 1847.781s

FAILED (errors=3, skipped=3)
Traceback (most recent call last):
  File "/users/hpcbuild/build/pytorch-testing/pytorch/test/run_test.py", line 1030, in <module>
    main()
  File "/users/hpcbuild/build/pytorch-testing/pytorch/test/run_test.py", line 1008, in main
    raise RuntimeError(err_message)
RuntimeError: distributed/test_c10d_gloo failed!

The server has 4 A100 GPUs and all are available to the program.
Anything else you want me to try?

[boegel]

That distributed/test_c10d_gloo test just seems flaky to me... See also #15137

Maybe @Flamefire has some feedback on this?

[Flamefire]

I opened #15904 with various patches especially relevant to POWER. Maybe they also fix this failure?

[sassy-crick]

I don't have access to the A100 any more. In light that some time has passed, if @boegel wants to close that I am happy with it.
Thanks for all your help!

[Micket]

PyTorch always fails :) I think we have given up on this old version by now.