Repository navigation
Description
Program goal
NeMo Gym must make rollout failures bounded, visible, and recoverable without silently changing evaluation or training semantics. The framework reports policy-neutral facts; callers decide whether to retry, replace, mask, omit, or stop.
This issue is the program tracker. It owns the target architecture, priority waves, workstream ownership, and dependency DAG. Child issues own implementation.
Core invariants
- Shared transport and provider retry loops terminate by default.
- A possibly delivered mutation is not replayed without an explicit safe-replay contract.
- A completed result, a no-result failure, and unresolved work remain distinguishable.
- Infrastructure failure is never fabricated as a successful reward-zero sample.
- Completed masked rewards cannot influence training statistics as policy evidence.
- Resume validates run identity and preserves attempt history.
- Acquired remote sessions have explicit bounded release and an expiry backstop where required.
- Partial startup, cancellation, and shutdown release locally owned resources within configured bounds.
Scope boundaries
This program does not own:
- model-call capture infrastructure, owned by [epic] Rollout observability & reliability fast-follows #2831;
- training policy inside NeMo-RL, tracked by fix(grpo): keep masked Gym failures out of prompt-group statistics RL#4061;
- rollout task-memory admission, owned by Bound rollout task admission and persist completed results incrementally #3006;
- multi-worker resources state placement, owned by Make resources-server multi-worker serving easier and state-safe #3039;
- checkpoint semantics, owned by feat(checkpoint): add resources state participant #2945/feat(checkpoint): adapt initial stateful environments #2946.
Observability and downstream training issues consume this program's facts, but do not block Gym contract completion.
Priority waves
Wave 0 — urgent independent correctness
These can land immediately and in parallel:
- fix(model): bound provider-response retries #2845 — bound internal provider-response retries. Do not wait for feat(core): bound HTTP retries and make request replay explicit #3159.
- fix(gymnasium): release subclass state when an episode ends #3157 — fix the concrete Gymnasium subclass leak.
- fix(genrm): bound incomplete cohorts and preserve cohort accounting #3181 — bound GenRM cohort waits and prevent unmarked partial rewards.
- feat(core): define a shared failure-kind registry #3179 — establish the small shared failure vocabulary.
- fix(grpo): keep masked Gym failures out of prompt-group statistics RL#4061 — fix existing direct GRPO
valid_maskparity; top-level Gym ingestion follows feat(core): define completed verifier-result failure semantics #2608.
Wave 1 — core framework contracts
These can proceed largely in parallel:
- feat(core): define typed rollout outcomes and recovery #2135 — rewardless no-result failures, durable attempts, resume identity, and exact reconciliation.
- feat(core): bound HTTP retries and make request replay explicit #3159 — finite transport retries, socket-connect bounds, and conservative replay of mutating requests.
- feat(core): define completed verifier-result failure semantics #2608 — completed-result
mask_sample,failure_kind, andfailure_reasonsemantics. - feat(core): define resources-session identity and close protocol #2609 — caller-known session identity, seed deduplication, explicit idempotent close, and bounded agent cleanup.
- feat(core): own startup, shutdown, and cancellation #3160 — transactional startup, finite readiness and shutdown, descendant ownership, and cancellation propagation.
Changes that expose new terminal transport errors must integrate with #2135's row-associated failure boundary before the program closes. They do not need to serialize all development behind #2135.
Wave 2 — migrations and adoption
- refactor(core): replace reward-zero failure sentinels with typed outcomes #3180 — migrate reward-zero/fake-response producers after feat(core): define typed rollout outcomes and recovery #2135 and feat(core): define completed verifier-result failure semantics #2608 stabilize.
- feat(resources): add session TTL and lifecycle adoption #3037 — adopt feat(core): define resources-session identity and close protocol #2609 in risk order and add TTL only where abandonment retains resources.
- fix(grpo): keep masked Gym failures out of prompt-group statistics RL#4061 — prefer feat(core): define completed verifier-result failure semantics #2608's top-level fields while retaining a bounded compatibility fallback.
Wave 3 — operational follow-ups
- feat(cli): add bounded graceful server shutdown #144 — user-facing stop after feat(core): own startup, shutdown, and cancellation #3160 supplies durable process ownership.
- feat: live visibility into in-progress rollouts #2859 — live rollout inspection under [epic] Rollout observability & reliability fast-follows #2831 after stable identity and progress APIs exist.
- feat(observability): standardize error-recovery telemetry #3182 — error-recovery telemetry under [epic] Rollout observability & reliability fast-follows #2831, instrumented incrementally as producer contracts land.
Dependency DAG
Solid arrows are hard semantic dependencies. Dashed arrows are later adaptation or instrumentation and do not block the upstream fix.
flowchart TD
Vocabulary["#3179 Failure vocabulary"]
Provider["#2845 Provider bounds"]
Leak["#3157 Gymnasium leak"]
Cohort["#3181 Cohort timeout"]
RLMask["RL#4061 Mask correctness"]
Outcomes["#2135 Durable outcomes"]
Transport["#3159 Transport safety"]
Verify["#2608 Verify fields"]
Session["#2609 Session protocol"]
Lifecycle["#3160 Local lifecycle"]
Migration["#3180 Producer migration"]
Adoption["#3037 Phased adoption"]
StopCLI["#144 Stop CLI"]
Visibility["#2859 Live visibility"]
Telemetry["#3182 Telemetry"]
Vocabulary --> Verify
Vocabulary --> Outcomes
Verify --> Migration
Outcomes --> Migration
Session --> Adoption
Lifecycle --> StopCLI
Verify -.-> RLMask
Outcomes -.-> Cohort
Leak -.-> Session
Provider -.-> Transport
Outcomes -.-> Visibility
Lifecycle -.-> Visibility
Transport -.-> Telemetry
Outcomes -.-> Telemetry
Session -.-> Telemetry
Lifecycle -.-> Telemetry
True merge dependencies
- feat(core): define a shared failure-kind registry #3179's stable names precede final
failure_kindproducer migrations in feat(core): define completed verifier-result failure semantics #2608 and feat(core): define typed rollout outcomes and recovery #2135. Schema work may proceed in parallel, and unknown names warn rather than discard evidence. - feat(core): define typed rollout outcomes and recovery #2135 and feat(core): define completed verifier-result failure semantics #2608 stabilize before refactor(core): replace reward-zero failure sentinels with typed outcomes #3180 changes canonical producer semantics.
- feat(core): define resources-session identity and close protocol #2609's identity and release hook precede broad adoption in feat(resources): add session TTL and lifecycle adoption #3037.
- feat(core): own startup, shutdown, and cancellation #3160's process-ownership API precedes feat(cli): add bounded graceful server shutdown #144.
Not hard gates:
- fix(model): bound provider-response retries #2845 can make provider retries finite before feat(core): bound HTTP retries and make request replay explicit #3159 defines any shared accounting.
- fix(genrm): bound incomplete cohorts and preserve cohort accounting #3181 can bound cohort waits before feat(core): define typed rollout outcomes and recovery #2135 integration.
- RL#4061 can fix current
valid_maskcall sites before feat(core): define completed verifier-result failure semantics #2608 lands. - feat: live visibility into in-progress rollouts #2859 and feat(observability): standardize error-recovery telemetry #3182 do not block correctness work or program completion.
Workstream ownership
- feat(core): define a shared failure-kind registry #3179 — shared low-cardinality failure vocabulary.
- feat(core): define typed rollout outcomes and recovery #2135 — no-result outcomes, persistence, resume, and reconciliation.
- refactor(core): replace reward-zero failure sentinels with typed outcomes #3180 — producer migration.
- fix(genrm): bound incomplete cohorts and preserve cohort accounting #3181 — GenRM cohort completion.
- feat(core): bound HTTP retries and make request replay explicit #3159 — transport bounds and replay safety.
- fix(model): bound provider-response retries #2845 — provider-response retry policy.
- feat(core): define resources-session identity and close protocol #2609 — session identity and explicit close.
- feat(resources): add session TTL and lifecycle adoption #3037 — phased adoption and TTL backstop.
- fix(gymnasium): release subclass state when an episode ends #3157 — immediate Gymnasium cleanup bug.
- feat(resources): add session TTL and lifecycle adoption #3037 — phased adoption and TTL backstop.
- feat(core): define completed verifier-result failure semantics #2608 — completed verifier-result semantics.
- fix(grpo): keep masked Gym failures out of prompt-group statistics RL#4061 — downstream training correctness.
- feat(core): own startup, shutdown, and cancellation #3160 — local startup, shutdown, and cancellation ownership.
- [epic] Rollout observability & reliability fast-follows #2831 owns operational visibility and telemetry:
- feat: live visibility into in-progress rollouts #2859 — live rollout inspection.
- feat(observability): standardize error-recovery telemetry #3182 — error-recovery telemetry.
#2249 was closed because its temporary compatibility responsibilities are now explicit parts of #2135 and #3180 rather than a separate framework layer.
Already landed
- Align aggregate metrics with persisted rollouts #1799 aligns aggregate metrics with persisted success rows.
- Record agent /run failures as row-associated failures #2017 adds opt-in row-associated
/runfailures, incremental sidecar persistence, continued independent work, and bounded rollout attempts. - feat: judge endpoint resiliency #2383 adds endpoint rebinding and optional bounded connection retries.
- feat: declare failure_reason on BaseVerifyResponse #2552 adds
failure_reasontoBaseVerifyResponse. - fix: clear failure sidecar on fresh rollout runs #2723 clears stale failure sidecars on fresh runs.
- fix: preserve HTTP errors across process boundaries #2726 preserves HTTP errors across process boundaries.
- fix(workplace-assistant): release session state after verification #2886 fixes Workplace Assistant verification-time cleanup.
- [Super 3.5 evals] Terminus2 Model call timeout parity; Servers cancel requests when client cancels requests #3064 propagates client disconnects into active handlers.
- add option to not increment endpoint retry limit #3079 bounds external provider status retries by default.
- fix: AUT-2228 route partial reruns to the current attempt #3163 assigns partial reruns to the current attempt.
Program completion criteria
- Default shared transport and internal provider retry loops terminate within configured bounds.
- Possibly delivered mutations are not silently replayed.
- Supported collector paths distinguish completed results from no-result failures without fabricated rewards or responses.
- Failures and attempts persist incrementally, and resume rejects incompatible input or configuration identity.
- Completed masked results remain visible but do not silently affect policy statistics.
- Session allocation is deduplicated and explicit cleanup is attempted within a bound; expensive abandoned sessions expire.
- Partial startup, cancellation, and local shutdown release owned local resources within configured deadlines.
- CLI evaluation and direct callers receive documented policy-neutral facts.
Live inspection, comprehensive telemetry, every environment migration, dashboards, and downstream training policy are tracked by this program but are not Gym program-completion gates.
Design documents
- Current-state summary
- Proposed design
- Resources-session API and adoption inventory
- Rollout failure catalogue
The documents describe the architectural destination. This tracker defines which portions are immediate safety work, true dependencies, and later integrations.
Active implementation candidates
- feat(core): define the shared failure-kind vocabulary #3213 replaces closed draft PR feat: add shared failure-kind registry #2728 for feat(core): define a shared failure-kind registry #3179.
- feat(core): report infrastructure failure from verify, not just a reward #2611 is the feat(core): define completed verifier-result failure semantics #2608 candidate and should retain the additive schema while leaving run-level reconciliation to feat(core): define typed rollout outcomes and recovery #2135.
- fix: bound model request retries #2527 is the fix(model): bound provider-response retries #2845 candidate and can land finite provider policy without waiting for a shared feat(core): bound HTTP retries and make request replay explicit #3159 budget.
- feat(core): give stateful resources servers a teardown hook and an optional idle sweeper #2612 contains useful session tests but requires substantial rework against feat(core): define resources-session identity and close protocol #2609's concurrency, identity, and bounded-cleanup contract.
- feat: judge cohort hardening #2385 is the fix(genrm): bound incomplete cohorts and preserve cohort accounting #3181 candidate; incomplete cohorts must fail or be explicitly masked rather than emit ordinary partial rewards.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDev Todo
Activity
tdene commentedon Aug 25, 2026
I've edited #2552 according to this plan; let me know what you think.
feat: declare failure_reason on BaseVerifyResponse (#2552)
sharonyu-115 commentedon Aug 27, 2026
Relevant to #2609. @waple0820 could you review the design here and provide your feedbacks. Thanks!
waple0820 commentedon Aug 27, 2026
@sharonyu-115 Thanks for the ping.
We ran into #2609 from browser-based RL in production — one cloud browser session per rollout — so our angle is the training side and the live environment: what these failures end up doing to the training data and to real resources, not just how they get recorded.
The part of this design that helps us most is the split between no valid result and a complete result that should not be used, and that the failure record deliberately carries no
mask_samplebecause "there is no sample to mask". That distinction is sharper than what we wrote in #2608. We had both cases collapsed into one flag, and in hindsight a rollout that never finished and a rollout that finished while its environment died really are not the same thing to a trainer — the first has to be redone, the second can stay and hold its position in the group. Same for the three fields being orthogonal:mask_sample=Falsealongside afailure_kind/failure_reasonexpresses "degraded but the score is real", and our version simply could not say that.We updated #2611 yesterday to match: dropped the duplicate
failure_reasondeclaration, keptmask_sampleonly, leftfailure_kindto #2728. We also added the half that reads it, since the flag was a declaration nothing consumed —rollout_collectionnow reportsmasked_pctandreward_unmaskednext to the existingreward, and the gap between the two reward series is the score a run lost to its own infrastructure. So the contract half is caught up. What follows is about the resource half.Acquire and release
Every failure path here ends by producing a record, but none of them tells the environment it can let go. What
SimpleResourcesServerexposes today is/seed_session,/verify,/aggregate_metrics,/reverify_mode— there is an acquire, and no release.The catalogue already names this: "Cancelling the local await does not reverse those effects." But the lifecycle section of the proposal picks up the processes
RunHelperowns, not the resources one rollout holds on a remote environment.RolloutFailureRecordeven reserves "Cleanup or remote-completion information when known" — nothing can fill that field today, because the release never happens.And it multiplies.
_DEFAULT_MAX_ROLLOUT_ATTEMPTS = 3, and a resumed attempt seeds again, so one task can leave three live sessions behind, two of them garbage. We ate this once: steps 61–80, 1172 of 1280 rollouts died in session reset and every one of them entered GRPO as a legitimatereward=0.One thing worth stressing: a caller-driven release is necessary but not sufficient. When the collector process itself is killed, that release request never goes out. So the environment side also needs an idle timeout of its own. #2612 is how we did it.
Replaying seed_session
POST defaulting to
before_delivery— not repeating a possibly delivered request — is the right rule. But/seed_sessionis a POST that allocates a resource, and walking it through: the environment has already built the session, the response is lost on the way back, the caller may not replay it under the rule — so the session exists and the caller has no id for it.Under the current contract there is no recovery from that step. With no id, neither a release nor a sweeper has anything to act on.
Our suggestion is to put
/seed_sessionunderdeduplicated: the caller mints the session id and sends it with the request. Replay stays safe, and a lost response is still releasable. #2613 points the same way.One question: where does #2609 land in the sequencing — reading it, "Add lifecycle control, identity, and observability" looks like the phase it belongs to?
Record agent /run failures as row-associated failures (#2017)
cafzal commentedon Sep 5, 2026
Additional evidence from a completed recovery exercise: one logical rollout first ended in an agent-side graph-state failure and produced an empty executable output. A fresh attempt for the same input completed verification but was a valid wrong answer. Both had scalar reward
0, while only the first was an invalid execution. Preserving the original attempt and appending the recovery was necessary to avoid silently changing the denominator or relabeling history.The implementation-specific glue required selection, retry keys, source-manifest hashing, append-only import, and a derived union. That cost is not benchmark-specific; long-running coding, browser, sandbox, and tool-use tasks need the same restart-from-input contract.
For the restart-first slice, I suggest fault tests that prove:
N+1without modifying attemptN.Ncannot supersede accepted output fromN+1.This evidence supports the stateless restart milestone, not transparent continuation or deterministic replay of external effects.
feat(core): reconcile every materialized rollout to one disposition
5 remaining items
ananthsub commentedon Sep 9, 2026
Tracker re-review is complete.
Immediate parallel lanes:
Hard merge gates:
failure_kindDeferred integration combines collector reconciliation, provider policy, session adoption, NeMo-RL masking, sentinel migration, and cohort completion. #3182 then standardizes cross-cutting telemetry under #2831, while stable events can be instrumented earlier.
The older #2361/#2363/#2365/#2366/#2373 stack and #2044 are closed as superseded, with reusable tests and implementation ideas recorded in the owning issues. Active candidates #2728, #2611, #2527, #2612, #2613, #2384, #2385, and #2964 now point to their current owners and dependencies.
fix(finance_sec_search): read the judge's verdict, not the first [[N]…
feat(core): report infrastructure failure from verify, not just a rew…
waple0820 commentedon Sep 19, 2026
@ananthsub #2608 is closed now that #2611 landed, which leaves #2609 as the next thing on the critical path — it gates broad adoption in #3037.
One correction on this issue's text: the note that #2612 "requires substantial rework against #2609's concurrency, identity, and bounded-cleanup contract" was true when written, but the rework happened on Sep 10. It was rebuilt against the contract rather than patched.
What's in it now: caller-created
_ng_session_idwith a close capability stored as a digest; seed deduplication that replays the stored response on the same payload, 409s on a different one, and keeps a bounded tombstone for a released id; the four terminal close outcomes, with an unknown id and an already-released one answering identically;_close_session_endpointowning identity, conflict checks and idempotency whilerelease_session(session_id, reason)owns only release; concurrent closers joining one release task; and expiry that active handlers suppress, plus an absolute lifetime they do not.Three places where the spec left room are called out in the PR, all easy to overturn — the hook name, the absence of a
not_foundstatus, and a seed for a released id returning 409 rather than replaying.Rebased onto main after the #2611 merge, 448fd79. Ready whenever you want to look at it.
yuhezhang-ai commentedon Oct 30, 2026
Status refresh, Oct 30 (updated Nov 2)
Checked against main at
0c7ac5119. Since Oct 30, #3302 and #4953 have merged. The remaining #2135 work now lands as one stack, starting with #5072 (see #2135 below).Where each item stands
Retry-Afterinternal=Truestill retry 429 and 5xx without a limit (no caller in the repo uses it now), andbrowsecomp_advanced_harnesscopies that loop. #2527 is superseded by #4786.masked_reward_policy: include | exclude, defaultincludeexclude_infrathat readsfailure_kind. Under the default, masked rewards still move group statistics.run_outcomes()and CLI exit codes (#3302)gym eval resubmit --recover, a re-judge of saved answers after judge failures, and thegym eval inspecthealth gate) → #5443 (run_examplesrecords, run-folder shards andgym eval statusread the rollout store; carries #5318, #5359 and #5360) → #3881 (health reports read the attempts selected for scoring) and #3879 (narrowed to lettinggym eval reverifyread resumed runs, since #5343 now does the re-judge). Still to check: whether #5201 (draft) is still needed next to #5343's re-judge.openai_modeland the vLLM route-around (#5199, #4365); a cut-off body isn't resent (#5197)ServerClientretries dropped connections without a limit and has no connect timeout. A POST is still resent after an ambiguous disconnect unless the caller opts out. #5181 is open./rundoesn't close its session (#5085, open).legacy_agentepisode deadlines (#5200) were folded into #5072.Implementation candidates in the body: #2527 is superseded by #4786, and #2612 was closed without merging.
Completion criteria
/runends collection unlessroute_failures_to_sidecaris on, and 171 of 237 components still have a failure path scored as a model result (tracking: audit reward-zero and fake-success failure producers #4883).run_outcomes()and CLI exit codes landed with feat(eval): persist failures and resume unfinished evaluations #3302; a failed/runstill ends collection by default (see the third criterion).New work the plan should include
simple_agentconfigs onto environment servers: feat(scripts): switch simple_agent relays to single_agent_turn_legacy #4952 and chore(configs): move simple_agent configs off the legacy_agent relay #4953 (merged), then refactor(configs): retire tool_simulation_agent and non_executing_simple_agent into simple_agent #4954, fix(env): own resources bindings and resolve inherited renames #5108 and feat(deepagents): run agent sessions in process or inside a task sandbox #5003. On an environment server, a failed/verifybecomes a failure row instead of ending the run. That changes how the refactor(core): replace reward-zero failure sentinels with typed outcomes #3180 fixes should look.Next for #3180
The #4883 inventory found 171 of 237 components with at least one path where a failure can be scored as a model result. 38 of the affected resources servers ship training data.
agent_outcome_policy(after Q1; failed agent runs reach the grader in 36 components), one rule for unreadable judge replies (after Q2; 12 components), and an opt-in to count masked failures as 0 for benchmarks whose official scoring does (Q6).legacy_agentrelay after chore(configs): move simple_agent configs off the legacy_agent relay #4953, so waiting doesn't make their fix smaller, and they share their grading code inresources_servers/swebench. After the last grading retry, mask with afailure_kindinstead of returning 0.If you're fixing how a component reports or scores a failure, please link this issue or #3180, so the fixes stay in one place.
fix(core): budget a trimmed failure report against omitted=false