Skip to content

feat(environment): stop the agent once a live world ends the episode - #5838

Open
yfw wants to merge 3 commits into
mainfrom
yifuw/stop-agent-at-episode-end
Open

yfw wants to merge 3 commits into
mainfrom
yifuw/stop-agent-at-episode-end

Conversation

@yfw

@yfw yfw commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Codex and OpenCode end their turn only when the model replies without a tool call. In a live world that ends the task on its own, such as a desktop task that ends on DONE, FAIL or its action limit, the harness keeps acting after the episode is over. This wastes model calls and time.

Fix

  • Episode status route. A Resources Server can answer an optional, typed POST /episode_status (ResourcesEpisodeStatusRequest, ResourcesEpisodeStatusResponse). The route exists only when a server overrides episode_status, so every other server answers 404. It is never an MCP tool.
  • Environment Server option. With episode_status_poll_seconds set, the single-agent Environment Server polls that route during the agent's turn. When the world reports done, it calls POST /rollouts/{rollout_id}/stop (feat(agent): let a caller stop a running rollout and still grade it #5145) with the reason. A 404 ends the polling. The legacy server inherits the option. When the option is unset, nothing changes.
  • OpenCode stop. OpenCode supports the rollout stop, as Codex and the other sandboxed harness agents on main do. A stop that arrives before OpenCode starts gives an empty incomplete run, not a failure. Its observations use the same schema (2.0) and agent instance as a session that ran.
  • Cut step. When a timeout or a stop cuts the last step, OpenCode reports it as turn_interrupted instead of turns_unavailable, with the cut session's invocation_id. The gap keeps the capture diagnostics from Record native capture diagnostics for missing OpenCode steps #5237. Earlier steps with no step finish are still turns_unavailable. A tool that the stop cut stays incomplete, as main records it since feat(opencode): capture qualified tool timing #5507, and main's TE-5 check already accepts an incomplete tool without output (feat(copilot): add a sandboxed Copilot CLI agent server #5082). The harness checker accepts one more thing a cut leaves behind:
    • TE-9: one HTTP 200 call with no step, when the stop came after the model's reply returned but before the harness saved any of it. It must be the row's only call without a step, and the last call of the invocation that a turn_interrupted gap names.

No issue: we found this while running OpenCode on a desktop benchmark.

Evidence

All runs used 10 OSWorld tasks (5 tasks, 2 repeats each), OpenCode 1.17.11, the same model and settings, and a step limit of 25. "Prototype" is an earlier version of this change without the cut-step part. "This PR" is the first commit of this PR with the OSWorld server's episode_status answer added. These runs are on the PR's earlier base, before the merge with main; no run was made after the merge.

without stop prototype this PR
Rollouts that ended at the stop n/a 9 of 10 9 of 10 (5 after DONE, 4 at the action limit)
Scored 3 of 10 4 of 10 4 of 10
Model calls 273 166 126
Wall time 31 min 19 min 18 min
TE-3 steps.gap failures 0 9 rows 0
TE-5 tools.output failures 0 4 rows 0
TE-9 steps.attempt_accounting failures 0 0 1 row on the first commit, 0 with the second (see below)
  • In this run, the 9 stopped rows have turn_interrupted and none has turns_unavailable.
  • The 4 rows stopped at the action limit each have one cut tool call without output. On the earlier base, OpenCode marked it timeout, and this PR accepted that in TE-5. On main, the same tool stays incomplete, which main's TE-5 check accepts, so the merge removed this PR's own TE-5 exception.
  • The one rollout that was not stopped ended on its own when its last reply hit the output limit. No rollout timed out.
  • The runs with the stop left no sandbox behind.

The TE-9 row is covered by the second commit. The run used the first commit. In one stopped row, the model's last reply had already returned, but the stop ended OpenCode before it saved any part of that reply. The cut message had no parts and no turn, so its completed call joined no step (assistant_message_call_unmatched), and TE-9 failed. The call itself was still captured. With the second commit, the gap names the invocation and the checker accepts that one call. I checked the recorded row again with the gap as the new code writes it, and every check passes.

Not in this PR

A desktop benchmark needs more before Codex or OpenCode can use this stop:

  • Its Resources Server must answer POST /episode_status. The OSWorld server on main does not yet, and it also lacks the typed seed and close and the desktop tools that a session harness calls. That is a separate OSWorld change.
  • Its configuration sets episode_status_poll_seconds (5 s in the runs above).
  • The harness's sandbox provider must support uploads that do not wait for running commands: Local, or OpenSandbox with operations.background_exec: true. On other providers, the stop of a running harness answers HTTP 501.
  • Codex has supported the stop since feat(agent): let a caller stop a running rollout and still grade it #5145, but its adapter does not report a cut step as turn_interrupted. A stopped Codex row was not tested against the harness checks.

Validation

At the new head (merge of main at 82f574a83):

  • pytest environment_servers/single_agent_turn/tests tests/unit_tests/server_runtime tests/unit_tests/rollouts tests/unit_tests/agents tests/unit_tests/trajectory tests/unit_tests/test_public_api.py tests/unit_tests/test_fern_docs_links.py: 6296 passed, 12 skipped.
  • pytest environment_servers/single_agent_turn_legacy/tests: 48 passed. It runs on its own, because both server suites have a tests/test_app.py.
  • pytest responses_api_agents/opencode_agent/tests (its own venv, from the repository root): 925 passed, 23 skipped.
  • pre-commit run --files on the PR's files: passed.
  • Before the merge, I reverted each part in turn: the OpenCode session hook, the stop marker, both pre-launch paths, the route registration, the reserved MCP name, the poller, the interrupted flag, the cut-step logic, its diagnostics, each condition of the TE-9 exception, and the gap's invocation_id. A new test failed each time.

Compatibility

New, opt-in API and option. Rows without a stop are unchanged. The only change for existing rows is the cut step: a run that timed out on its own now gets turn_interrupted for its last unfinished step instead of turns_unavailable.

🤖 Generated with Claude Code

Codex and OpenCode end their turn only when the model replies without a
tool call. In a live world that ends the task on its own, such as a
desktop task that ends on DONE or on its action limit, the harness keeps
acting after the episode is over.

- Resources Servers can answer an optional, typed POST /episode_status.
  The route exists only when a server overrides episode_status, so other
  servers answer 404. It is never an MCP tool.
- The single-agent Environment Server option episode_status_poll_seconds
  polls that route during the agent's turn. When the world reports done,
  it calls POST /rollouts/{rollout_id}/stop on the Agent Server with the
  reason. A 404 ends the polling. The legacy server inherits the option.
- OpenCode supports the rollout stop, as Codex does, including a stop
  that arrives before OpenCode starts.
- When a timeout or a stop cuts the last step, OpenCode reports it as
  turn_interrupted instead of missing evidence. The harness checker
  accepts a tool call with status timeout, error_type
  invocation_timeout and no output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 9, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yfw yfw added area:env-infra Shared environment framework, lifecycle, registry, scaffolding, and validation feature New capabilities, enhancements, or enablement work labels Nov 9, 2026
@yfw

yfw commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 5bbd7c0

…ed it

A rollout stop can end OpenCode after the model's last reply returned but
before OpenCode saved any part of it. The cut message then has no turn,
so its completed call joins no step, and the TE-9 step join failed the
row (1 of 9 stopped rows in an OSWorld run).

- OpenCode emits one turn_interrupted gap for each cut session and sets
  its invocation_id.
- steps.attempt_accounting accepts one policy call with no step only
  when it is the row's only such call, returned HTTP 200, and is the last
  call of an invocation that a turn_interrupted gap names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@yfw

yfw commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 7f67046

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Resolve the conflicts with main at 82f574a:
- single_agent_turn: keep main's checkpoint text and imports; the episode
  status poller uses main's LOGGER.
- OpenCode: keep main's step-limit marker, qualified tool timing and call
  ID normalization next to the rollout stop and the cut-step report. The
  run stopped before launch now reports schema 2.0 observations with the
  session's agent instance, like a run that started.
- Harness checks: take main's TE-5 rule. Since #5507 OpenCode keeps a tool
  that a stop cut as incomplete, and main accepts an incomplete tool
  without output (#5082), so this PR's timeout exception, its test and
  its doc sentence are removed. The TE-9 exception stays.
- Docs: name every agent on main that supports the rollout stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@yfw

yfw commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 910d858

@yfw
yfw marked this pull request as ready for review November 13, 2026 02:20

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH CARE

I found three RISK issues: the stop request can queue behind the agent turn it must stop, a timeout can hide missing evidence from an already-completed child session, and TE-9 can excuse a call that isn't the invocation's final attempt.

I inspected all 20 changed files and traced the affected session, sandbox, trajectory-joining, and conformance paths. This was a static review; I couldn't execute tests or independently confirm the validation reported in the PR.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

continue
reason = f"episode over: {status.reason}" if status.reason else "episode over"
try:
stop_response = await self.server_client.post(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Send the watcher requests through the reserved control pool.

Failure: With global_aiohttp_connector_limit_per_host=1, the outstanding agent /v1/responses request holds the only data connection to that host. Resources can return done: true, but this stop request queues behind the turn it must stop. The harness therefore continues until it finishes or reaches its own deadline. The same problem occurs at higher concurrency when all per-host slots are occupied. ServerClient.request() defaults to the data pool; the shared client already has a separate control pool specifically to avoid this starvation.

Correction: Could you use that pool for both watcher calls? The status poll also needs independence when the total data-connection limit is exhausted. In _stop_agent_when_episode_ends(), replace each self.server_client.post(...) with self.server_client.request(method="POST", _control=True, ...), retaining all existing arguments. For example, the stop becomes:

stop_response = await self.server_client.request(
    method="POST",
    _control=True,
    server_name=self.config.agent_server.name,
    url_path=f"/rollouts/{rollout_id}/stop",
    json=RolloutStopRequest(reason=reason[:200]).model_dump(),
)

Apply the same transport change to status_response, preserving its Resources cookies and empty body. The invariant is that watcher traffic must not wait for connection capacity held by the running turn. This suggested code hasn't been executed; please verify it before adopting it.

Acceptance (proposed): Use the real HTTP client with a per-host limit of one and an agent response handler that waits for its stop handler. Have Resources return done: true; assert that the stop arrives and the episode reaches verification before a short test deadline. Also retain the existing cases where 404 ends polling, transient status failures retry, and an unset option sends neither watcher request.

unavailable = set()

def missing(session_id: str, message_id: str) -> None:
if last_assistant.get(session_id) == message_id:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Preserve missing evidence from sessions that finished before the timeout.

Failure: interrupted describes the whole harness run, but last_assistant includes every retained session. Consider a child assistant with finish="stop", a valid completion timestamp, step-start and text, but no step-finish; then let a later root step time out. _parse_opencode_session() reports that child as completed, yet this branch converts its missing finish into turn_interrupted. The root timeout cannot have cut the already-completed child. If the child turn joins its captured call normally, steps.gap now accepts its missing evidence, weakening conformance results for interrupted multi-session runs.

Correction: Could you qualify missing() with the session/message's native terminal evidence, not only its position and the global timeout flag? At minimum, a message with explicit finish="stop" and a valid time.completed must retain turns_unavailable when its finish part is missing. Only a genuinely unfinished final policy step should gain turn_interrupted. Don't reject interruption merely because any completion timestamp exists: a tool-calls reply can complete while its invocation still runs tools. The complete terminal qualification needs to preserve that distinction, so I haven't suggested a blanket timestamp-based patch.

Acceptance and preservation (proposed, not executed): Add the completed-child/root-cut database above and assert that the child produces turns_unavailable, while the root produces turn_interrupted with the root invocation ID. After joining their captured calls, steps.gap must fail because of the child's lost evidence. Retain the assertions in the existing cut-step tests: a running-tool cut keeps its function call, an empty cut creates no turn, and an earlier unfinished step remains unavailable. Keep valid completed children gap-free, summary/synthetic messages excluded, and continuation gap IDs remapped to their activation.

if invocation.get("invocation_id") == owners[stepless[0]][0]
for reference in _objects(invocation.get("model_calls"))
for index in _resolve(reference, self.calls)
if index in policy - http_errors

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Compare the cut call against every later invocation attempt.

Failure: Starting from stop_cut_record(), append another invocation-owned call after cut, with HTTP 500 and no step. This filter removes that later attempt from owned, so owned[-1:] still equals cut and steps.attempt_accounting passes. But the harness proceeded to another request: cut is no longer the invocation's last attempt, as the exception requires. Later compaction-helper calls are also ignored. This lets the new exception falsely certify a missing step outside its terminal-stop window.

Correction: Keep the helper/HTTP-error exclusions for ordinary step accounting, but remove them when deciding whether the candidate is terminal. The smallest correction is:

                 for reference in _objects(invocation.get("model_calls"))
                 for index in _resolve(reference, self.calls)
-                if index in policy - http_errors

The existing one-stepless-policy-call, HTTP-200, and invocation-owner requirements remain unchanged; the terminal comparison then includes every resolved attempt owned by that invocation. This is a source-reviewed suggestion, not an executed patch; please verify it before adopting it.

Acceptance and preservation (proposed): Add cases with a later owned HTTP-500 attempt and a later explicit compaction helper; both must make the earlier stepless HTTP-200 call fail steps.attempt_accounting. An HTTP error or helper before a genuinely terminal cut must not prevent that cut from being excused. Retain the terminal-cut pass and the existing no-gap, two-stepless, not-last, and wrong-owner failures. Also retain ordinary retry/helper accounting: HTTP errors may remain unbound, helpers stay outside policy steps, and duplicate step references still fail. I couldn't execute these checks.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Nov 16, 2026

This branch was previously deployed

1 inactive (outdated) deployment
public — 7f67046a Deployed Nov 9, 2026 by copy-pr-bot[bot] via release / finalize / notify #10982
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:env-infra Shared environment framework, lifecycle, registry, scaffolding, and validation feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants