The more NCCL communicators, the more GPU memory?

[ForFishes]

Please tell me about the relationship between GPU memory usage and the number of NCCL communicators.

1. Does it mean that the more NCCL communicators are created, the more GPU memory will be used? Is it a linear relationship?

It is found here that when an NCCL communicator is established and allreduce communication is performed, the memory occupies about 1.6GB; when multiple NCCL communicators (>=3) are established, the memory occupies about 6GB. If a lot of nccl connections are established, will it be OOM?

2. Is there any relationship between the memory usage of NCCL and the number of GPUs?

Is there any difference in the amount of memory used when hundreds of GPUs establish NCCL connections and dozens of GPUs establish NCCL connections?

[sjeaugey]

1. More communicators = more memory used, except if communicators are created with ncclCommSplit and the flag to share resources is set in the communicator config.
2. Intra-node it depends on many factors.

• How much bandwidth we want to reach: we'll use more memory to reach H100 peak NVLink bandwidth (350GB/s) than a PCI Gen3 peak bandwidth (12GB/s). The memory usage is linearly tied to the number of channels. Reducing the number of channels will significantly reduce the memory usage.
• The algorithms we want to use. Collnet/Direct uses a lot of memory because of alltoall direct connections, while Ring uses a limited amount of memory as it only has 2 connections regardless of the scale.
• The number of GPUs, although beyond 2 nodes it should no longer increase.
• Whether you perform send/receive operations or only say allreduce.

[ForFishes]

Thanks for your reply.