Skip to content

Preserve Progress Through Rollout Interruptions #3949

Description

@cwing-nvidia

Observation

NeMo Gym coordinates long-running rollouts across multiple stateful components. Support for bounded retries, lifecycle cleanup, failure classification, and recovery currently varies by component and workload. Interruptions can require unfinished work to restart, while some failure paths do not yet produce a consistent terminal outcome.

Goal

Users can run evaluation and training rollouts with confidence that work will finish or fail clearly, valid progress will not be unnecessarily lost, and infrastructure problems will not be mistaken for model behavior. After an interruption, Gym recovers supported work and clearly identifies anything that must restart.

Activity

  1. changed the title [-]Ensure Reliable and Recoverable Rollout Execution[/-] [+]Preserve Progress Through Rollout Interruptions[/+] on Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions