Worker killed error

[apoorvkh]

No description provided.

[apoorvkh]

Notes:

• We should annotate all the places where failures occur.
  • If the user's program raises an exception, our worker processes catch it and return that as a WorkerException.
  • If any worker process fails (due to raised exception or signal/exit), ctx.wait(5) captures this as a failure. We return that as a WorkerKilledError.
  • I think launcher_agent_group.sync_agent_statuses (wrapping dist.all_gather) raises a RuntimeError if launcher or any agent fails to respond. This should crash all agents.
  • The Launcher raises the WorkerException or WorkerKilledError, and raises an AgentKilledError if it catches a RuntimeError from sync_agent_statuses.
• I think having both WorkerException and WorkerKilledError is confusing. Maybe ExceptionInWorker (not this but similar)? And maybe we can rename to AgentFailedError and WorkerFailedError.
• Can we more closely wrap the RuntimeError as an AgentFailedError for clarity?

[apoorvkh]

unrelated, but let's also make a note that dist_mp.start_processes copies environment variables into the worker

[apoorvkh]

I tried to address above all comments, please review @pmcurtin ! We can document further in #73

[pmcurtin]

Awesome!