Skip to content

Let callers stop a running rollout and still grade it #5091

Closed

Description

@gwarmstrong

Problem

An agent's only time budget today is its own wall-clock timeout (agent_timeout_sec, which a sample can set), fixed when the rollout starts. When the harness hits it, the supervisor ends the harness and the episode still goes on to verification. A caller can also cancel rollouts that are in flight (#4029), but a cancelled rollout is discarded, not graded.

Some budgets can only be judged from outside the agent while the rollout runs. For those there is no way to end the rollout and keep a graded result.

Use cases

  • Budgets that leave out infrastructure time. A long-horizon task with an 8-hour budget that loses an hour waiting on its model endpoint (queueing for capacity, rate-limit backoff, a cold start, a restart) got 7 hours of work, and its score reflects the infrastructure. feat(browsecomp): per-rollout wall-clock limit and run timing summary #4174 makes this visible for BrowseComp: its timing summary flags rollouts that queued on the model servers. A stop hook lets the caller hold the budget fixed in the time that counts instead.
  • Token or cost caps across a whole rollout, enforced by the caller that sees the usage.
  • Stragglers in RL rollout collection. At a step deadline, end the slowest rollouts and keep them as graded partial episodes instead of discarding them, for the deadline consumers feat(rollout): let run_examples callers cancel rollouts still in flight #4029 was built for.
  • An operator stopping a runaway rollout without losing it.

Proposal

One endpoint on agent servers whose harness runs under SandboxSession: POST /rollouts/{rollout_id}/stop, with an optional body {"reason": "..."}. rollout_id is the id in /ng-rollout/<id>.

  • It ends the harness through the same supervisor path as the agent's own timeout. It writes the session's existing stop-request marker first, so harnesses that read it (Hermes, Stirrup) can tell a requested stop from a wall-time one.
  • The agent then returns as it does at a timeout, and the episode is verified and graded. The response carries a typed cause, so environments and agent_outcome_policy can tell a requested stop from a timeout.
  • It returns 200 with whether this call stopped the harness. It is idempotent, does nothing once the rollout has finished, and returns 404 for an unknown rollout.
  • Agents opt in through a small shared helper: Codex first, the others as their owners want.

Nothing changes for callers that don't use it. The budget logic stays with the caller (for example a rollout_collection_driver), so Gym does not have to define budgets.

Why this shape

  • It builds on what SandboxSession already has: the stop-request marker and stop_harness().
  • It does not depend on how the model is served: vLLM directly, a hosted API, or a router in between.
  • It can stop a harness at any point, including during a long tool call.

Alternatives considered

  • An API that moves the agent's deadline. Gym keeps the clock and the caller adds time. That puts more state in the agent, and the caller still has to know the budget.

Prototyping this against current main and can send a PR once the shape is agreed.

Activity

  1. yuhezhang-ai commented on Oct 27, 2026

    @yuhezhang-ai
    Contributor

    Thanks George—this looks useful, and keeping budget accounting with the caller makes sense.

    A few related changes to keep in mind:

    • #4834 is merged and provides agent outcome classification plus the policy for grading, retrying, or failing a response. The requested-stop cause should fit that existing path and remain distinguishable from timeout and cancellation.
    • #3302, which I’m working on, is still open. It saves completed results and failure records, controls retries, and resumes unfinished evaluations. For your stop-and-grade path, a successfully graded partial attempt should remain a saved result on resume; the intentional stop alone should not cause it to rerun or consume the failure budget.
    • #5072 is a draft covering failure details passed between services. It is relevant if stopping, collecting output, or verification fails. Those actual failures should retain their causes through the shared reporting mechanism. An intentional stop does not itself need to become a failure.

    I don’t think you need to wait for those open PRs to prototype this on main. If the implementation needs changes to shared outcome or failure fields, let’s coordinate those before merging. A useful integration check with #3302 would cover a stopped-and-graded result surviving save/resume without rerunning, plus a grading failure after stopping following the existing recovery rules.

    One question about the infrastructure-time example: how would this interact with the existing agent_timeout_sec? An eight-hour hard timeout would still fire before the caller’s eight hours of counted work if there was infrastructure waiting. Would callers configure a larger safety timeout while enforcing their actual budget through /stop?

    One other API detail worth making explicit: after /stop returns, does the caller keep awaiting the original rollout request to receive the graded result? For the training-deadline use case, stopping the agent still leaves cleanup and verification to finish. It would help to show the intended caller sequence, including how it avoids cancelling the result through the existing close/drain API.

  2. gwarmstrong commented on Oct 27, 2026

    @gwarmstrong
    ContributorAuthor

    Thanks, this is very helpful. Proposed adjustments first, then the two questions.

    Outcome. A requested stop won't be a failure kind. The stopped run returns status: incomplete, which #4834 already verifies, with a Gym-owned marker ng_rollout_stop: {"reason": ...} in the response metadata. That keeps it distinct from agent_timeout and cancelled without touching the failure-kind registry. (incomplete_details.reason would be the natural place, but the OpenAI SDK types limit it to max_output_tokens and content_filter.) Does that marker name and placement work for you, or would you prefer it elsewhere in the shared outcome fields?

    #3302. Since the stop isn't a failure, a stopped and graded rollout should save as a normal result. I'll run the check you suggested against #3302: a stopped, graded result survives save/resume without rerunning or using the failure budget, and a grading failure after a stop follows the existing recovery rules. I'll post the results here.

    #5072. Agreed: failures while stopping, collecting or verifying keep their causes, and the stop itself isn't one. Those will go through the shared reporting once #5072 lands.

    agent_timeout_sec. Yes. The caller sets agent_timeout_sec as a wall-clock safety cap above its budget plus whatever waiting it is willing to tolerate, and enforces the counted budget with /stop. In a test with a 1-hour counted budget and a 3-hour cap, all three rollouts were stopped within a second of the budget and then verified normally.

    Caller sequence. /stop ends the agent's run, not the rollout request. Grading happens inside the same /run request that started the rollout: the environment server calls the agent, then verifies. After /stop, the agent returns its response as it would at a timeout, the environment server collects and verifies as usual, and the original /run returns the graded result. So the caller keeps awaiting that request instead of cancelling it:

    future = ...  # the in-flight /run request, or its run_examples future
    await client.post(f"{agent}/rollouts/{rollout_id}/stop", json={"reason": "budget"})
    result = await future  # graded; status incomplete, ng_rollout_stop set

    For a training deadline: post /stop for each straggler, keep consuming results until they arrive, and only then call aclose_and_drain(). Calling aclose() first would cancel the requests that are about to return those graded results.

  3. gwarmstrong commented on Oct 27, 2026

    @gwarmstrong
    ContributorAuthor

    Following up with the #3302 check. I applied the stop change to #3302's head (f60544d) and added an integration test that goes through the real pieces: the agent's report_rollout_stop marks the response, single_agent_turn verifies it, and #3302's collector saves and resumes the result.

    • A stopped and graded rollout is a saved result. single_agent_turn returns it as a result (status: incomplete, the verifier's reward, ng_rollout_stop in the response metadata), and the collector saves it as a successful, measured result with the marker intact.
    • Resume reuses it. Interrupting the collector right after that result was saved and resuming does not dispatch it again, and it spends nothing from the failure budget. A later resume with nothing left to run dispatches nothing.
    • Verification failing after a stop follows the existing rules. A transient verifier failure (a timeout) after a stop is recorded as a nonterminal, delivered failure, counts once against the budget, and the next attempt is graded. A malformed-verifier failure is terminal and not retried. Both keep stage: verification and their own reason; the stopped response stays attached as the partial-response diagnostic, marker included.
    • Nothing in feat(eval): persist failures and resume unfinished evaluations #3302 treated incomplete or the marker specially. feat(eval): persist failures and resume unfinished evaluations #3302's recovery and records suites, single_agent_turn's and the session tests pass with the change (473 tests).

    The branch now matches what I described above: a stopped run carries no failure kind, #4834's files are untouched, and the caller's reason is recorded as ng_rollout_stop: {"reason": ...}, a JSON string like ng_agent_outcome. I'll open the PR once the marker works for you. The integration test depends on #3302, so it can land with whichever of the two merges second.

  4. gwarmstrong commented on Oct 27, 2026

    @gwarmstrong
    ContributorAuthor

    @yuhezhang-ai the PR is up: #5145. It uses the ng_rollout_stop marker as described above; happy to move it if you'd rather it live elsewhere.

  5. ananthsub commented on Oct 28, 2026

    @ananthsub
    Contributor

    @gwarmstrong @yuhezhang-ai is there infrastructure from the partial rollout checkpointing that can be re-used for this? #3024 and stack here: #3882

  6. gwarmstrong commented on Oct 28, 2026

    @gwarmstrong
    ContributorAuthor

    Thanks @ananthsub. Yes, and #5145 already builds on two parts of it:

    Two other pieces looked close but don't fit the "end it now and grade it" case:

    • retire also stops an attempt's work, but it discards the attempt and refuses its later requests until forget. A stop has to keep the run and send it to verification.
    • Boundaries and generation cuts park an in-process turn loop before a model call or after a tool round. A stop has to end a CLI harness (Codex, Claude Code, ...) that may be in the middle of a long tool call, so it ends the process through the supervisor's timeout path instead.

    One more piece we could reuse if you'd like: the control plane's auth. Today POST /rollouts/{rollout_id}/stop sits next to the agent's other routes with no token. It could move under /ng-control/v1/ and require a bearer token like the checkpoint routes, so only the controller can stop rollouts. I'd keep that token separate from checkpoint.enabled, so a caller can stop rollouts without turning on checkpointing; a rollout_collection_driver reads it from the same global config the servers use. Happy to make that change in #5145 if that's the direction you prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions