Implicit distributed backend selection #516
Conversation
booxter
commented
Apr 30, 2025
booxter
commented
- chore: bump pytorch to 2.6.0+
- feat: Rely on implicit detection of distributed backend
mergify
bot
added
dependencies
|
E2E (NVIDIA L40S x4) workflow launched on this PR: View run |
|
e2e workflow succeeded on this PR: View run, congrats! |
|
@tiran any particular concerns with this bump of minimal pytorch to 2.6.0+ for training library? (It's already 2.6.0+ in ilab so I'd not expect any, but better double-check...) |
|
As confirmed by Doug H, this won't change versions used downstream. ilab already pulls 2.6.0+ for all flavors. |
|
This pull request has merge conflicts that must be resolved before it can be |
This is in line with ilab repo. There are some features in later pytorch releases that we may want to have access to. Signed-off-by: Ihar Hrachyshka <[email protected]>
From the official docs, ``` Since 2.6, if backend is not provided, c10d will use a backend registered for the device type indicated by the device_id kwarg (if provided). ``` and: ``` If neither backend nor device_id is provided, c10d will detect the accelerator on the run-time machine and use a backend registered for that detected accelerator (or cpu). ``` While the library is still cuda centric, this is one tiny step towards a more agnostic implementation. Signed-off-by: Ihar Hrachyshka <[email protected]>
| init = functools.partial(torch.distributed.init_process_group, "nccl") | ||
| if timeout is not None: | ||
| init(timeout=timeout) | ||
| torch.distributed.init_process_group(timeout=timeout) |
There was a problem hiding this comment.
ooohhh very nice. I wonder if this helps us get closer to removing our hard-dep on CUDA only training
3 participants
