Skip to content

feat(agent): let a caller stop a running rollout and still grade it - #5145

Merged
yuhezhang-ai merged 20 commits into
NVIDIA-NeMo:mainfrom
gwarmstrong:georgea/gym-rollout-stop
Oct 30, 2026
Merged

yuhezhang-ai merged 20 commits into
NVIDIA-NeMo:mainfrom
gwarmstrong:georgea/gym-rollout-stop

Conversation

@gwarmstrong

@gwarmstrong gwarmstrong commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Lets a caller end a running rollout early and still get it graded. Today an agent's only time budget is its own agent_timeout_sec, fixed when the rollout starts, and cancelling a rollout in flight (#4029) discards it. Some budgets can only be judged from outside the agent while it runs: time that leaves out waiting on the model endpoint, token or cost caps, a training-step deadline for stragglers, an operator stopping a runaway rollout. Design and discussion: #5091.

  • Route. POST /rollouts/{rollout_id}/stop on the agent server, with an optional {"reason": ...}. rollout_id is the id in /ng-rollout/<id>. It answers {"rollout_id", "stopped"}; stopped is false when the run already ended or a stop was already requested, so repeating it is safe. An unknown or closed rollout answers 404, and an agent that has not opted in answers 501. With several server workers the request needs no session cookie: the worker running the rollout claims its /rollouts/<rollout_id> path, and the others forward the stop to it. A stop applies to the rollout's current agent session: an environment that opens a new session for each turn, such as Harbor's multi-step tasks, still schedules the later turns, so a caller enforcing a whole-episode budget must also keep the environment from starting further turns. stopped: true means the stop was accepted; the run can still finish on its own before it takes effect.
  • How the run ends. The stop moves the supervisor's deadline to now (a new --expire-file, which process_supervisor.sh polls alongside its stop file, written as a file within the stop's timeout, so it does not wait for exec capacity the running harness holds). The harness then ends through the existing timeout path: TERM, grace, cleanup. If the marker write fails after reaching the sandbox, the route answers 502, and the stop is still kept once the receipt says timed_out and the marker is confirmed in place. Finalization waits for a marker write still in flight before that check and bounds the whole check by its timeout; without a confirmation the stop stays unattributed. A stop that arrives while the session is still seeding takes effect when the harness would launch, and the harness never starts. Runners see a timeout, not a cancellation: no stop.request marker is written. Stopping a running harness needs a sandbox provider whose uploads never wait for capacity that running commands hold, declared by the new optional capability SupportsSandboxIndependentUploads: Local, and OpenSandbox with operations.background_exec: true. On other providers (Docker, Apptainer and Enroot share CLI slots, OpenShell worker threads, OpenSandbox in foreground mode its HTTP connection pool; Daytona, E2B and ECS Fargate are not shown to be independent), SandboxSession.expire() raises ExpiryUnsupportedError for a running harness, and the route answers 501 with the reason.
  • How it is graded. The agent returns the response it returns at a timeout, status: incomplete, which single_agent_turn verifies. The response metadata gains ng_rollout_stop, a JSON object with the caller's reason, which tells a stop apart from a timeout or a cancellation. A stop is not a failure, so the response carries no ng_agent_outcome failure kind and feat(agent): report native causes for shared grading policy #4834's files are unchanged; a stopped and graded rollout is an ordinary result.
  • Caller sequence. /stop returns right away. The caller keeps awaiting its original /run request, which returns the graded result once collection and verification finish. Calling aclose() on a run_examples handle first would cancel those requests.
  • Agents. Shared code in nemo_gym (route, multi-worker routing, seed-time stop, SandboxSession.expire(), report_rollout_stop()); an agent opts in by returning its SandboxSession from _rollout_sandbox_session() and passing its response through report_rollout_stop(). Codex is wired here. Each other SandboxSession agent (Claude Code, Hermes, Kilo, OpenClaw, OpenCode, Pi, Stirrup) needs about five lines, as follow-ups.

Size: +257/−5 non-test lines in nemo_gym, +23/−1 in codex_agent, plus a paragraph in add-or-update-a-harness.mdx.

Closes #5091.

Validation

  • Unit: codex_agent 468 passed (7 skipped), single_agent_turn 123, the supervisor, sandbox-session and agent-session tests 144, and session routing 89. New cases cover an unknown rollout, a finished run, a repeated stop, a stop during seeding, a stop after close, an agent that has not opted in, a failed stop write that can be retried, Codex stopped mid-run (incomplete, partial output kept, marker set) and before launch (never started), and single_agent_turn verifying a stopped run whose result keeps the marker. Added for the review: a stop through another server worker, during that worker's seed, and after close; a stop for an unknown rollout leaving no routing claim; expiry while the harness holds every Local exec slot (one slot, and all eight default slots); an expiry write that never returns failing within the timeout and succeeding on retry; Codex stopped after its worker launched but before it wrote events; a transcript that exists but fails to download after a stop, and a failed transcript check, keeping the error path; a marker write that lands but loses its acknowledgment keeping the stop's reason, and one that never lands leaving a natural timeout unattributed; the bash supervisor's expire file before launch and while running, and a stop file winning over it; a marker write that fails during finalization; the marker check while another workload holds every Local exec slot; a stop request that disconnects before its write starts, which still delivers the marker or stays retryable; a write still pending when finalization stops waiting, which leaves the stop unattributed; a running harness on a provider without independent uploads, refused with 501 and no marker written, while a stop before launch works on any provider; and which providers declare the capability.
  • Live on OpenSandbox at 2b71260: with background_exec: true, a stop of a running harness returned in 0.07 s and the harness ended 0.8 s later through the timeout path (receipt timed_out, exit 143, cleanup confirmed). In foreground mode the stop was refused at once and the harness kept running. Earlier, on Gym's Docker provider (before running-harness stops were limited to providers with independent uploads): a stop mid-run ended the harness in 0.4 to 0.5 s through the timeout path, and a stop before launch kept it from starting; no containers were left.
  • With feat(eval): persist failures and resume unfinished evaluations #3302 (its head f60544d plus this change, in a test that is not part of this PR because it needs feat(eval): persist failures and resume unfinished evaluations #3302): a stopped and graded rollout is saved as a normal result; interrupting the collector and resuming does not rerun it or spend the failure budget; a verifier failure after a stop follows feat(eval): persist failures and resume unfinished evaluations #3302's recovery rules (a timeout is retried once, a malformed verifier is terminal). Details in Let callers stop a running rollout and still grade it #5091.
  • Long-horizon: an earlier version of this change (which labeled the stop with agent_limit instead of the marker) ran under a rollout_collection_driver that enforced a 1-hour budget of counted agent time, with a 3-hour agent_timeout_sec as the safety cap. Three Codex rollouts on Terminal-Bench-Science were each stopped once, 0.6 to 0.85 s after the budget, and then verified normally.

Rollouts

N/A: agent-server control route.

Compatibility and benchmark impact

New optional route; nothing changes for callers that don't use it. Timeouts and cancellation behave as before. Agents that don't opt in answer 501. Stopping a running harness answers 501 on providers without SupportsSandboxIndependentUploads; a stop before launch works on every provider. Reserving stop capacity in Docker, Apptainer, Enroot, OpenShell and OpenSandbox's foreground mode, so they can stop a running harness too, is a follow-up.

POST /rollouts/{rollout_id}/stop ends a rollout's harness exactly as at
its time limit, so the episode is still verified and graded as a
timeout. It is for callers that enforce their own budget (agent time,
tokens, cost) and for operators. Cancelling the /run request instead
discards the rollout.

The supervisor gains an expire request file that moves its deadline to
now; one present before launch keeps the harness from starting. No
stop.request marker is written, so harness runners see their deadline
path. SandboxSession.expire writes the file, or marks the next launch
when the harness has not started. A stop during seeding applies when
the harness would launch.

Agents opt in by returning the rollout's SandboxSession from
_rollout_sandbox_session and passing the response through
report_rollout_stop, which keeps the timeout's incomplete status and
adds the agent_limit cause. Other agents answer 501. Codex opts in.

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
… failure kind

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 27, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@gwarmstrong

Copy link
Copy Markdown
Contributor Author

/ok to test 21744ea

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:agent Agent harnesses and Responses API agent behavior feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Oct 27, 2026

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH CARE — three RISK findings: rollout-only stops can miss the owning worker, expiry delivery can queue behind the harness, and an immediate post-launch stop can become an ungraded artifact failure.

Completed review of all 12 changed files and their directly affected routing, provider, artifact, and grading paths. The findings below include correction invariants and proposed regression checks. Static inspection only; no tests were executed.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread nemo_gym/base_responses_api_agent.py Resolved
Comment thread nemo_gym/agent_utils/sandbox_session.py Resolved
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
@gwarmstrong

Copy link
Copy Markdown
Contributor Author

/ok to test ed95769

@yuhezhang-ai yuhezhang-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the follow-up fixes. I rechecked ed9576949 before posting: the cookieless cross-worker stop and the stop-before-events startup reproductions now pass, so I am dropping those original findings. The new Linux Local-provider tests also cover the actual marker/supervisor path that was missing from the earlier tests; I could not execute those Linux-only cases on this macOS host.

One issue remains in the startup fix: a genuine transcript download failure after a stop is now converted into an empty graded response. I would address that before merging. The other three inline comments are nonblocking clarifications or follow-ups.

I do not see a reason to wait solely for #3302 or #5072. The intended separation still makes sense: an intentional stop can be a normal graded result, while genuine agent, artifact-collection, and verifier failures retain their normal failure handling. The artifact-download comment matters to that boundary. I have not independently rerun the combined #3302 branch.

Validation on the current revision: 23 focused checks passed, including the routing/startup reproductions and fault-injection probes described inline; two Linux-only tests skipped. This was a focused check of the updates after the earlier full review, not a repeat of the full suite or a live model/container run.

Comment thread nemo_gym/agent_utils/sandbox_session.py Resolved
yuhezhang-ai
yuhezhang-ai previously approved these changes Oct 30, 2026
…-stop

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>

# Conflicts:
#	nemo_gym/base_responses_api_agent.py
#	tests/unit_tests/test_agent_sessions.py
@gwarmstrong

Copy link
Copy Markdown
Contributor Author

/ok to test 417f7f1

…-stop

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
@gwarmstrong

Copy link
Copy Markdown
Contributor Author

/ok to test caacdbd

@gwarmstrong

Copy link
Copy Markdown
Contributor Author

@yuhezhang-ai one more, sorry. #5214 broke swe_pivot's tests on main, so CI failed here at 417f7f1 for reasons outside this PR (main's own merge-queue runs failed the same way), and #5388 fixed it. I merged main again in caacdbd: no conflicts this time. Since your approval of 7f6535a, the merges brought in main's #5211, #5212 and #5214 (imports in base_responses_api_agent.py and one test file; git carried this PR's session-routing changes through the server_runtime move) and #5388. The PR's tests pass (643). Could you re-approve once CI is green? I'll queue it right away.

@yuhezhang-ai yuhezhang-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked caacdbdc2 and the intervening main merges. The conflict resolutions preserve the reviewed implementation; session routing moved intact into server_runtime, with only its import path changed. The final merge bringing in #5388 reproduces cleanly. No new code-review findings.

Independent validation on this exact head: 75 focused tests passed, covering stop delivery, saturation, supervisor behavior, cross-worker routing, and grading; lint and formatting passed for all 17 changed Python files. No fresh model runs were needed for these import/move resolutions. GitHub's routing end-to-end and Codex conformance checks have passed; the full CI run is still in progress. Reapproving the code review; merge remains gated on the remaining required checks.

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41142997696

This branch was previously deployed

1 inactive deployment
public — caacdbdc Deployed Oct 30, 2026 by copy-pr-bot[bot] via release / finalize / notify #8861
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let callers stop a running rollout and still grade it

4 participants