[train] Exit actor and log appropriately when poll_workers is in terminal state

[TimothySeah]

Summary

@justinvyu observed the following logs:

(TrainController pid=109066) [State Transition] RUNNING -> ABORTED.
(TrainController pid=109066) [FailurePolicy] Decision: FailureDecision.RAISE, Error source: controller, Error count / maximum errors allowed: 1/0, Error: Training failed due to controller error:
(TrainController pid=109066) Worker group is not active. Call WorkerGroup.create() to create a new worker group.
(TrainController pid=109066) [State Transition] ABORTED -> SHUTTING_DOWN.

This indicates the following sequence of events:

1. The controller is RUNNING
2. The controller calls _poll_workers and therefore asyncio.sleeps
3. The user Ctrl - C's
4. The asyncio event loop switches to the abort asyncio task, which shuts down the worker group and prints RUNNING -> ABORTED
5. The abort asyncio.task calls ray.actor.exit_actor, but https://docs.ray.io/en/latest/ray-core/api/doc/ray.actor.exit_actor.html says For asyncio actors, there may be a short delay before the actor exits if the API is called from a background task
6. The asyncio event loop switches back to the _poll_workers task, which fails an assert (the Worker group is not active log above) and goes through the SHUTTING_DOWN + ERRORED path.

This PR does the following:

1. Add traceback to all ControllerErrors and log it when making a failure decision so we can see where Worker group is not active. Call WorkerGroup.create() to create a new worker group. is coming from. I also sanity checked that this does not cause UserExceptionWithTraceback to double print the traceback because this only applies to ControllerError
2. _poll_workers has the only asyncio.sleep in the Ray Train controller. After waking up, it exits from the foreground asyncio task if its state is terminal, which can happen due to the issue mentioned in 5).

Testing

Unit tests - I was unable to reproduce this in a workspace since the exit_actor behavior is inconsistent.

I created a fake failure to showcase what the logs look like before and after adding a traceback:

Before

(TrainController pid=90972) [FailurePolicy] RAISE
(TrainController pid=90972)   Source: controller
(TrainController pid=90972)   Error count: 1 (max allowed: 0)
(TrainController pid=90972) 
(TrainController pid=90972) Training failed due to controller error:
(TrainController pid=90972) fake error

After

(TrainController pid=15478) [FailurePolicy] RAISE
(TrainController pid=15478)   Source: controller
(TrainController pid=15478)   Error count: 1 (max allowed: 0)
(TrainController pid=15478) 
(TrainController pid=15478) Traceback (most recent call last):
(TrainController pid=15478)   File "/Users/tseah/ray/python/ray/train/v2/_internal/execution/controller/controller.py", line 421, in _step
(TrainController pid=15478)     raise ValueError("fake error")
(TrainController pid=15478) ray.train.ControllerError: Training failed due to controller error:
(TrainController pid=15478) fake error

Here is what worker group failures look like without a traceback:

python/ray/train/v2/tests/test_failure_policy.py::test_max_failures[1] ✓                                              22% ██▎       2025-11-17 13:27:49,354	INFO default.py:44 -- [FailurePolicy] RETRY
  Source: worker group
  Error count: 1 (max allowed: 10)

ray.train.WorkerGroupError: Training failed due to worker errors:
Worker group failed

[justinvyu]

Thanks!