Skip to content

Make rollout failures bounded, visible, and recoverable #2750

Open
4 of 7 issues completed
Open

Description

@ananthsub

Program goal

NeMo Gym must make rollout failures bounded, visible, and recoverable without silently changing evaluation or training semantics. The framework reports policy-neutral facts; callers decide whether to retry, replace, mask, omit, or stop.

This issue is the program tracker. It owns the target architecture, priority waves, workstream ownership, and dependency DAG. Child issues own implementation.

Core invariants

  • Shared transport and provider retry loops terminate by default.
  • A possibly delivered mutation is not replayed without an explicit safe-replay contract.
  • A completed result, a no-result failure, and unresolved work remain distinguishable.
  • Infrastructure failure is never fabricated as a successful reward-zero sample.
  • Completed masked rewards cannot influence training statistics as policy evidence.
  • Resume validates run identity and preserves attempt history.
  • Acquired remote sessions have explicit bounded release and an expiry backstop where required.
  • Partial startup, cancellation, and shutdown release locally owned resources within configured bounds.

Scope boundaries

This program does not own:

Observability and downstream training issues consume this program's facts, but do not block Gym contract completion.

Priority waves

Wave 0 — urgent independent correctness

These can land immediately and in parallel:

Wave 1 — core framework contracts

These can proceed largely in parallel:

Changes that expose new terminal transport errors must integrate with #2135's row-associated failure boundary before the program closes. They do not need to serialize all development behind #2135.

Wave 2 — migrations and adoption

Wave 3 — operational follow-ups

Dependency DAG

Solid arrows are hard semantic dependencies. Dashed arrows are later adaptation or instrumentation and do not block the upstream fix.

flowchart TD
    Vocabulary["#3179 Failure vocabulary"]
    Provider["#2845 Provider bounds"]
    Leak["#3157 Gymnasium leak"]
    Cohort["#3181 Cohort timeout"]
    RLMask["RL#4061 Mask correctness"]
    Outcomes["#2135 Durable outcomes"]
    Transport["#3159 Transport safety"]
    Verify["#2608 Verify fields"]
    Session["#2609 Session protocol"]
    Lifecycle["#3160 Local lifecycle"]
    Migration["#3180 Producer migration"]
    Adoption["#3037 Phased adoption"]
    StopCLI["#144 Stop CLI"]
    Visibility["#2859 Live visibility"]
    Telemetry["#3182 Telemetry"]

    Vocabulary --> Verify
    Vocabulary --> Outcomes
    Verify --> Migration
    Outcomes --> Migration
    Session --> Adoption
    Lifecycle --> StopCLI
    Verify -.-> RLMask
    Outcomes -.-> Cohort
    Leak -.-> Session
    Provider -.-> Transport
    Outcomes -.-> Visibility
    Lifecycle -.-> Visibility
    Transport -.-> Telemetry
    Outcomes -.-> Telemetry
    Session -.-> Telemetry
    Lifecycle -.-> Telemetry
Loading

True merge dependencies

Not hard gates:

Workstream ownership

#2249 was closed because its temporary compatibility responsibilities are now explicit parts of #2135 and #3180 rather than a separate framework layer.

Already landed

Program completion criteria

  • Default shared transport and internal provider retry loops terminate within configured bounds.
  • Possibly delivered mutations are not silently replayed.
  • Supported collector paths distinguish completed results from no-result failures without fabricated rewards or responses.
  • Failures and attempts persist incrementally, and resume rejects incompatible input or configuration identity.
  • Completed masked results remain visible but do not silently affect policy statistics.
  • Session allocation is deduplicated and explicit cleanup is attempted within a bound; expensive abandoned sessions expire.
  • Partial startup, cancellation, and local shutdown release owned local resources within configured deadlines.
  • CLI evaluation and direct callers receive documented policy-neutral facts.

Live inspection, comprehensive telemetry, every environment migration, dashboards, and downstream training policy are tracked by this program but are not Gym program-completion gates.

Design documents

The documents describe the architectural destination. This tracker defines which portions are immediate safety work, true dependencies, and later integrations.

Active implementation candidates

Activity

  1. self-assigned this
    on Aug 25, 2026
  2. tdene commented on Aug 25, 2026

    @tdene
    Contributor

    I've edited #2552 according to this plan; let me know what you think.

  3. sharonyu-115 commented on Aug 27, 2026

    @sharonyu-115

    Relevant to #2609. @waple0820 could you review the design here and provide your feedbacks. Thanks!

  4. waple0820 commented on Aug 27, 2026

    @waple0820
    Contributor

    @sharonyu-115 Thanks for the ping.

    We ran into #2609 from browser-based RL in production — one cloud browser session per rollout — so our angle is the training side and the live environment: what these failures end up doing to the training data and to real resources, not just how they get recorded.

    The part of this design that helps us most is the split between no valid result and a complete result that should not be used, and that the failure record deliberately carries no mask_sample because "there is no sample to mask". That distinction is sharper than what we wrote in #2608. We had both cases collapsed into one flag, and in hindsight a rollout that never finished and a rollout that finished while its environment died really are not the same thing to a trainer — the first has to be redone, the second can stay and hold its position in the group. Same for the three fields being orthogonal: mask_sample=False alongside a failure_kind/failure_reason expresses "degraded but the score is real", and our version simply could not say that.

    We updated #2611 yesterday to match: dropped the duplicate failure_reason declaration, kept mask_sample only, left failure_kind to #2728. We also added the half that reads it, since the flag was a declaration nothing consumed — rollout_collection now reports masked_pct and reward_unmasked next to the existing reward, and the gap between the two reward series is the score a run lost to its own infrastructure. So the contract half is caught up. What follows is about the resource half.

    Acquire and release

    Every failure path here ends by producing a record, but none of them tells the environment it can let go. What SimpleResourcesServer exposes today is /seed_session, /verify, /aggregate_metrics, /reverify_mode — there is an acquire, and no release.

    The catalogue already names this: "Cancelling the local await does not reverse those effects." But the lifecycle section of the proposal picks up the processes RunHelper owns, not the resources one rollout holds on a remote environment. RolloutFailureRecord even reserves "Cleanup or remote-completion information when known" — nothing can fill that field today, because the release never happens.

    And it multiplies. _DEFAULT_MAX_ROLLOUT_ATTEMPTS = 3, and a resumed attempt seeds again, so one task can leave three live sessions behind, two of them garbage. We ate this once: steps 61–80, 1172 of 1280 rollouts died in session reset and every one of them entered GRPO as a legitimate reward=0.

    One thing worth stressing: a caller-driven release is necessary but not sufficient. When the collector process itself is killed, that release request never goes out. So the environment side also needs an idle timeout of its own. #2612 is how we did it.

    Replaying seed_session

    POST defaulting to before_delivery — not repeating a possibly delivered request — is the right rule. But /seed_session is a POST that allocates a resource, and walking it through: the environment has already built the session, the response is lost on the way back, the caller may not replay it under the rule — so the session exists and the caller has no id for it.

    Under the current contract there is no recovery from that step. With no id, neither a release nor a sweeper has anything to act on.

    Our suggestion is to put /seed_session under deduplicated: the caller mints the session id and sends it with the request. Replay stays safe, and a lost response is still releasable. #2613 points the same way.


    One question: where does #2609 land in the sequencing — reading it, "Add lifecycle control, identity, and observability" looks like the phase it belongs to?

  5. cafzal commented on Sep 5, 2026

    @cafzal

    Additional evidence from a completed recovery exercise: one logical rollout first ended in an agent-side graph-state failure and produced an empty executable output. A fresh attempt for the same input completed verification but was a valid wrong answer. Both had scalar reward 0, while only the first was an invalid execution. Preserving the original attempt and appending the recovery was necessary to avoid silently changing the denominator or relabeling history.

    The implementation-specific glue required selection, retry keys, source-manifest hashing, append-only import, and a derived union. That cost is not benchmark-specific; long-running coding, browser, sandbox, and tool-use tasks need the same restart-from-input contract.

    For the restart-first slice, I suggest fault tests that prove:

    1. Selection by stable logical rollout ID creates attempt N+1 without modifying attempt N.
    2. Re-running the same recovery command is idempotent.
    3. Input/config/source-artifact digest drift fails closed or creates an explicit new experiment identity.
    4. Delayed output from N cannot supersede accepted output from N+1.
    5. Coverage reconciles original, recovered, unresolved, omitted, and unknown dispositions to the materialized input count.
    6. Possibly delivered operations require an explicit replay-safety declaration before fresh dispatch.

    This evidence supports the stateless restart milestone, not transparent continuation or deterministic replay of external effects.

  6. 5 remaining items

  7. ananthsub commented on Sep 9, 2026

    @ananthsub
    ContributorAuthor

    Tracker re-review is complete.

    Immediate parallel lanes:

    Hard merge gates:

    Deferred integration combines collector reconciliation, provider policy, session adoption, NeMo-RL masking, sentinel migration, and cohort completion. #3182 then standardizes cross-cutting telemetry under #2831, while stable events can be instrumented earlier.

    The older #2361/#2363/#2365/#2366/#2373 stack and #2044 are closed as superseded, with reusable tests and implementation ideas recorded in the owning issues. Active candidates #2728, #2611, #2527, #2612, #2613, #2384, #2385, and #2964 now point to their current owners and dependencies.

  8. waple0820 commented on Sep 19, 2026

    @waple0820
    Contributor

    @ananthsub #2608 is closed now that #2611 landed, which leaves #2609 as the next thing on the critical path — it gates broad adoption in #3037.

    One correction on this issue's text: the note that #2612 "requires substantial rework against #2609's concurrency, identity, and bounded-cleanup contract" was true when written, but the rework happened on Sep 10. It was rebuilt against the contract rather than patched.

    What's in it now: caller-created _ng_session_id with a close capability stored as a digest; seed deduplication that replays the stored response on the same payload, 409s on a different one, and keeps a bounded tombstone for a released id; the four terminal close outcomes, with an unknown id and an already-released one answering identically; _close_session_endpoint owning identity, conflict checks and idempotency while release_session(session_id, reason) owns only release; concurrent closers joining one release task; and expiry that active handlers suppress, plus an absolute lifetime they do not.

    Three places where the spec left room are called out in the PR, all easy to overturn — the hook name, the absence of a not_found status, and a seed for a released id returning 409 rather than replaying.

    Rebased onto main after the #2611 merge, 448fd79. Ready whenever you want to look at it.

  9. yuhezhang-ai commented on Oct 30, 2026

    @yuhezhang-ai
    Contributor

    Status refresh, Oct 30 (updated Nov 2)

    Checked against main at 0c7ac5119. Since Oct 30, #3302 and #4953 have merged. The remaining #2135 work now lands as one stack, starting with #5072 (see #2135 below).

    Where each item stands

    Item Status Done Left
    #3179 failure vocabulary Done
    #2845 provider-response retries Mostly done #4786, #3571 and #3415 bound provider retries, with jitter and Retry-After Clients with internal=True still retry 429 and 5xx without a limit (no caller in the repo uses it now), and browsecomp_advanced_harness copies that loop. #2527 is superseded by #4786.
    #3157 Gymnasium leak Done
    #3181 GenRM cohorts Done Follow-up #5330 (open) masks only the two rollouts of a failed comparison instead of the whole group; it needs NVIDIA-NeMo/RL#4968 first.
    NVIDIA-NeMo/RL#4061 masked failures in GRPO Partly done NVIDIA-NeMo/RL#4160 adds masked_reward_policy: include | exclude, default include NVIDIA-NeMo/RL#4968 (in review) adds an opt-in exclude_infra that reads failure_kind. Under the default, masked rewards still move group statistics.
    #2135 outcomes, persistence, resume Mostly done Incremental persistence, cancel and drain, coverage and retry reporting, reverification identity (#5221); resume identity, attempt history, run_outcomes() and CLI exit codes (#3302) One stack, in review, merging in this order: #5072 (the failure record across servers) → #5343 (recovery from saved results with gym eval resubmit --recover, a re-judge of saved answers after judge failures, and the gym eval inspect health gate) → #5443 (run_examples records, run-folder shards and gym eval status read the rollout store; carries #5318, #5359 and #5360) → #3881 (health reports read the attempts selected for scoring) and #3879 (narrowed to letting gym eval reverify read resumed runs, since #5343 now does the re-judge). Still to check: whether #5201 (draft) is still needed next to #5343's re-judge.
    #3159 transport bounds and replay Partly done An explicit cap now stops every retry path (#3415, #3914); bounded connection retries in openai_model and the vLLM route-around (#5199, #4365); a cut-off body isn't resent (#5197) Defaults: ServerClient retries dropped connections without a limit and has no connect timeout. A POST is still resent after an ambiguous disconnect unless the caller opts out. #5181 is open.
    #2608 completed verify fields Done
    #2609 session protocol Done #3544, #3786, #4679 The remaining gaps belong to #3037: close returns no outcome, and a failed legacy /run doesn't close its session (#5085, open).
    #3160 startup, shutdown, cancellation Partly done #3735 bounds component readiness and unwinds a failed startup The head server's readiness wait has no limit, servers share the CLI's process group, shutdown has no single deadline, and the middleware turns a cancellation into HTTP 500. #5141 (approved, needs a rebase) adds process groups and one grace period. legacy_agent episode deadlines (#5200) were folded into #5072.
    #3180 producer migration In progress Inventory in #4883; #4882 fixes in #5223, #5315 (in review) and #5342 (draft) See "Next for #3180" below.
    #3037 session adoption and TTL Partly done About a dozen of the ~40 resources servers that seed sessions also close them (#4679, #4692) The rest, including the SWE sandbox servers, osworld and workplace_assistant, seed without closing, and there is no shared expiry. Two open tracks for SWE sessions need one choice: #4277 and #4302–#4304, or #4576, #5266, #5267 and #5331.
    #144 stop CLI Not started Waits on #3160's process ownership (#5141).
    #2859 live rollout visibility Not started Per-rollout traces exist (#4751, #5052); stable rollout identity landed with #3302 Waits on #3160's progress state.
    #3182 error-recovery telemetry Partly done Outcome, reward and masked, cancellation and connection-pool metrics (#3638, #4448, #4609, #4028, #3632) Retry and exhaustion metrics, session cleanup, cohort timeouts and stalled work; the outcome counter doesn't cover every collection path yet.

    Implementation candidates in the body: #2527 is superseded by #4786, and #2612 was closed without merging.

    Completion criteria

    New work the plan should include

    Next for #3180

    The #4883 inventory found 171 of 237 components with at least one path where a failure can be scored as a model result. 38 of the affected resources servers ship training data.

    1. Settle the questions listed in tracking: audit reward-zero and fake-success failure producers #4883. The main one is whose failure it is when a run fails after the model started working. feat(grpo): add exclude_infra masked-reward policy for infra-masked rollouts RL#4968 already takes the proposed default for training: a failure that isn't clearly the policy's doesn't count.
    2. Shared fixes, which cover many components at once: the default for failed agent runs in agent_outcome_policy (after Q1; failed agent runs reach the grader in 36 components), one rule for unreadable judge replies (after Q2; 12 components), and an opt-in to count masked failures as 0 for benchmarks whose official scoring does (Q6).
    3. Coding (SWE) training environments, now. They stay on the legacy_agent relay after chore(configs): move simple_agent configs off the legacy_agent relay #4953, so waiting doesn't make their fix smaller, and they share their grading code in resources_servers/swebench. After the last grading retry, mask with a failure_kind instead of returning 0.
    4. Judge- and rule-based environments, after the feat(eval): report causal failures across service boundaries #5072 stack lands (feat(eval): persist failures and resume unfinished evaluations #3302 and chore(configs): move simple_agent configs off the legacy_agent relay #4953 have). Then the fix for each path is a few lines: raise a failure with a reason instead of returning 0, and Gym records it and retries it.
    5. A conformance check that forces failures, so new environments don't add gaps.
    6. The rest of fix: complete failure metadata on existing masked results #4882: conversational_tool_use_simulation (one mask covers both judge verdicts and generation errors, so it needs splitting first), the anyswe, anyterminal and osworld agents (after the open PRs on them land), and token capture.

    If you're fixing how a component reports or scores a failure, please link this issue or #3180, so the fixes stay in one place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions