Add timeout to init_process_group in entrypoint

[apoorvkh]

Question: what's the longest a distributed operation should reasonably take?
How long would it take to "all-gather" a large amount of memory (like 80 GB)?

Let's set a smaller default timeout... maybe 180 seconds?
And then we can pass an argument to override this.

torchrunx/src/torchrunx/agent.py

Lines 83 to 85 in cd1a895

	dist.init_process_group(
	backend=backend, world_size=worker_args.world_size, rank=worker_args.rank, store=store
	)

[apoorvkh]

pytorch/pytorch#13056 (comment)

[apoorvkh]

I guess they added monitored_barrier but it only works for GLOO.

https://pytorch.org/docs/stable/distributed.html#torch.distributed.monitored_barrier

I still think we can reduce this default, but allow users to override if desired.

[pmcurtin]

pytorch/pytorch#13056 (comment)

Oh yeah, that'd be great. I was thinking barriers might be an issue. I think this is a PyTorch issue really.

I still think we can reduce this default, but allow users to override if desired.

Yeah that sounds reasonable.

[apoorvkh]

Let's actually keep the same default but allow users to override :)