submitit fix

[pmcurtin]

No description provided.

[pmcurtin]

Seemed to work for me, just try out on your side and then we can merge!

[apoorvkh]

Okay great, I think this works when the script is successful!

However, it does not seem to work when the script raises an error :(

E.g. simply set this in your submitit test script.

def main():
    raise RuntimeError

Would you adjust for this case? Thanks!

But also, I think we designed our scripts with the assumption that child processes die with their parents and we might need to make more changes if that is not the case?

[apoorvkh]

I think we might also be interested in making daemon processes (which die with the parent process):

https://stackoverflow.com/a/34353290

[pmcurtin]

Yeah we could daemonize the printing process, not sure if it can easily be done for the processes spawned by PyTorch, however.

[apoorvkh]

Looks like this works! I also tested it on two nodes with 1 GPU each.