You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Preserve Progress Through Rollout Interruptions #3949
NeMo Gym coordinates long-running rollouts across multiple stateful components. Support for bounded retries, lifecycle cleanup, failure classification, and recovery currently varies by component and workload. Interruptions can require unfinished work to restart, while some failure paths do not yet produce a consistent terminal outcome.
Goal
Users can run evaluation and training rollouts with confidence that work will finish or fail clearly, valid progress will not be unnecessarily lost, and infrastructure problems will not be mistaken for model behavior. After an interruption, Gym recovers supported work and clearly identifies anything that must restart.
Activity
[-]Ensure Reliable and Recoverable Rollout Execution[/-][+]Preserve Progress Through Rollout Interruptions[/+]