Repository navigation
Conversation
Codex and OpenCode end their turn only when the model replies without a
tool call. In a live world that ends the task on its own, such as a
desktop task that ends on DONE or on its action limit, the harness keeps
acting after the episode is over.
- Resources Servers can answer an optional, typed POST /episode_status.
The route exists only when a server overrides episode_status, so other
servers answer 404. It is never an MCP tool.
- The single-agent Environment Server option episode_status_poll_seconds
polls that route during the agent's turn. When the world reports done,
it calls POST /rollouts/{rollout_id}/stop on the Agent Server with the
reason. A 404 ends the polling. The legacy server inherits the option.
- OpenCode supports the rollout stop, as Codex does, including a stop
that arrives before OpenCode starts.
- When a timeout or a stop cuts the last step, OpenCode reports it as
turn_interrupted instead of missing evidence. The harness checker
accepts a tool call with status timeout, error_type
invocation_timeout and no output.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
|
/ok to test 5bbd7c0 |
…ed it A rollout stop can end OpenCode after the model's last reply returned but before OpenCode saved any part of it. The cut message then has no turn, so its completed call joins no step, and the TE-9 step join failed the row (1 of 9 stopped rows in an OSWorld run). - OpenCode emits one turn_interrupted gap for each cut session and sets its invocation_id. - steps.attempt_accounting accepts one policy call with no step only when it is the row's only such call, returned HTTP 200, and is the last call of an invocation that a turn_interrupted gap names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
|
/ok to test 7f67046 |
|
🌿 Preview your docs: https://nvidia-preview-pr-5838-910d858061d6.docs.buildwithfern.com/nemo/gym |
Resolve the conflicts with main at 82f574a: - single_agent_turn: keep main's checkpoint text and imports; the episode status poller uses main's LOGGER. - OpenCode: keep main's step-limit marker, qualified tool timing and call ID normalization next to the rollout stop and the cut-step report. The run stopped before launch now reports schema 2.0 observations with the session's agent instance, like a run that started. - Harness checks: take main's TE-5 rule. Since #5507 OpenCode keeps a tool that a stop cut as incomplete, and main accepts an incomplete tool without output (#5082), so this PR's timeout exception, its test and its doc sentence are removed. The TE-9 exception stays. - Docs: name every agent on main that supports the rollout stop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
|
/ok to test 910d858 |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP WITH CARE
I found three RISK issues: the stop request can queue behind the agent turn it must stop, a timeout can hide missing evidence from an already-completed child session, and TE-9 can excuse a call that isn't the invocation's final attempt.
I inspected all 20 changed files and traced the affected session, sandbox, trajectory-joining, and conformance paths. This was a static review; I couldn't execute tests or independently confirm the validation reported in the PR.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| continue | ||
| reason = f"episode over: {status.reason}" if status.reason else "episode over" | ||
| try: | ||
| stop_response = await self.server_client.post( |
There was a problem hiding this comment.
RISK — Send the watcher requests through the reserved control pool.
Failure: With global_aiohttp_connector_limit_per_host=1, the outstanding agent /v1/responses request holds the only data connection to that host. Resources can return done: true, but this stop request queues behind the turn it must stop. The harness therefore continues until it finishes or reaches its own deadline. The same problem occurs at higher concurrency when all per-host slots are occupied. ServerClient.request() defaults to the data pool; the shared client already has a separate control pool specifically to avoid this starvation.
Correction: Could you use that pool for both watcher calls? The status poll also needs independence when the total data-connection limit is exhausted. In _stop_agent_when_episode_ends(), replace each self.server_client.post(...) with self.server_client.request(method="POST", _control=True, ...), retaining all existing arguments. For example, the stop becomes:
stop_response = await self.server_client.request(
method="POST",
_control=True,
server_name=self.config.agent_server.name,
url_path=f"/rollouts/{rollout_id}/stop",
json=RolloutStopRequest(reason=reason[:200]).model_dump(),
)Apply the same transport change to status_response, preserving its Resources cookies and empty body. The invariant is that watcher traffic must not wait for connection capacity held by the running turn. This suggested code hasn't been executed; please verify it before adopting it.
Acceptance (proposed): Use the real HTTP client with a per-host limit of one and an agent response handler that waits for its stop handler. Have Resources return done: true; assert that the stop arrives and the episode reaches verification before a short test deadline. Also retain the existing cases where 404 ends polling, transient status failures retry, and an unset option sends neither watcher request.
| unavailable = set() | ||
|
|
||
| def missing(session_id: str, message_id: str) -> None: | ||
| if last_assistant.get(session_id) == message_id: |
There was a problem hiding this comment.
RISK — Preserve missing evidence from sessions that finished before the timeout.
Failure: interrupted describes the whole harness run, but last_assistant includes every retained session. Consider a child assistant with finish="stop", a valid completion timestamp, step-start and text, but no step-finish; then let a later root step time out. _parse_opencode_session() reports that child as completed, yet this branch converts its missing finish into turn_interrupted. The root timeout cannot have cut the already-completed child. If the child turn joins its captured call normally, steps.gap now accepts its missing evidence, weakening conformance results for interrupted multi-session runs.
Correction: Could you qualify missing() with the session/message's native terminal evidence, not only its position and the global timeout flag? At minimum, a message with explicit finish="stop" and a valid time.completed must retain turns_unavailable when its finish part is missing. Only a genuinely unfinished final policy step should gain turn_interrupted. Don't reject interruption merely because any completion timestamp exists: a tool-calls reply can complete while its invocation still runs tools. The complete terminal qualification needs to preserve that distinction, so I haven't suggested a blanket timestamp-based patch.
Acceptance and preservation (proposed, not executed): Add the completed-child/root-cut database above and assert that the child produces turns_unavailable, while the root produces turn_interrupted with the root invocation ID. After joining their captured calls, steps.gap must fail because of the child's lost evidence. Retain the assertions in the existing cut-step tests: a running-tool cut keeps its function call, an empty cut creates no turn, and an earlier unfinished step remains unavailable. Keep valid completed children gap-free, summary/synthetic messages excluded, and continuation gap IDs remapped to their activation.
| if invocation.get("invocation_id") == owners[stepless[0]][0] | ||
| for reference in _objects(invocation.get("model_calls")) | ||
| for index in _resolve(reference, self.calls) | ||
| if index in policy - http_errors |
There was a problem hiding this comment.
RISK — Compare the cut call against every later invocation attempt.
Failure: Starting from stop_cut_record(), append another invocation-owned call after cut, with HTTP 500 and no step. This filter removes that later attempt from owned, so owned[-1:] still equals cut and steps.attempt_accounting passes. But the harness proceeded to another request: cut is no longer the invocation's last attempt, as the exception requires. Later compaction-helper calls are also ignored. This lets the new exception falsely certify a missing step outside its terminal-stop window.
Correction: Keep the helper/HTTP-error exclusions for ordinary step accounting, but remove them when deciding whether the candidate is terminal. The smallest correction is:
for reference in _objects(invocation.get("model_calls"))
for index in _resolve(reference, self.calls)
- if index in policy - http_errorsThe existing one-stepless-policy-call, HTTP-200, and invocation-owner requirements remain unchanged; the terminal comparison then includes every resolved attempt owned by that invocation. This is a source-reviewed suggestion, not an executed patch; please verify it before adopting it.
Acceptance and preservation (proposed): Add cases with a later owned HTTP-500 attempt and a later explicit compaction helper; both must make the earlier stepless HTTP-200 call fail steps.attempt_accounting. An HTTP error or helper before a genuinely terminal cut must not prevent that cut from being excused. Retain the terminal-cut pass and the existing no-gap, two-stepless, not-last, and wrong-owner failures. Also retain ordinary retry/helper accounting: HTTP errors may remain unbound, helpers stay outside policy steps, and duplicate step references still fail. I couldn't execute these checks.
Problem
Codex and OpenCode end their turn only when the model replies without a tool call. In a live world that ends the task on its own, such as a desktop task that ends on DONE, FAIL or its action limit, the harness keeps acting after the episode is over. This wastes model calls and time.
Fix
POST /episode_status(ResourcesEpisodeStatusRequest,ResourcesEpisodeStatusResponse). The route exists only when a server overridesepisode_status, so every other server answers 404. It is never an MCP tool.episode_status_poll_secondsset, the single-agent Environment Server polls that route during the agent's turn. When the world reportsdone, it callsPOST /rollouts/{rollout_id}/stop(feat(agent): let a caller stop a running rollout and still grade it #5145) with the reason. A 404 ends the polling. The legacy server inherits the option. When the option is unset, nothing changes.maindo. A stop that arrives before OpenCode starts gives an empty incomplete run, not a failure. Its observations use the same schema (2.0) and agent instance as a session that ran.turn_interruptedinstead ofturns_unavailable, with the cut session'sinvocation_id. The gap keeps the capture diagnostics from Record native capture diagnostics for missing OpenCode steps #5237. Earlier steps with no step finish are stillturns_unavailable. A tool that the stop cut staysincomplete, asmainrecords it since feat(opencode): capture qualified tool timing #5507, andmain's TE-5 check already accepts an incomplete tool without output (feat(copilot): add a sandboxed Copilot CLI agent server #5082). The harness checker accepts one more thing a cut leaves behind:turn_interruptedgap names.No issue: we found this while running OpenCode on a desktop benchmark.
Evidence
All runs used 10 OSWorld tasks (5 tasks, 2 repeats each), OpenCode 1.17.11, the same model and settings, and a step limit of 25. "Prototype" is an earlier version of this change without the cut-step part. "This PR" is the first commit of this PR with the OSWorld server's
episode_statusanswer added. These runs are on the PR's earlier base, before the merge withmain; no run was made after the merge.steps.gapfailurestools.outputfailuressteps.attempt_accountingfailuresturn_interruptedand none hasturns_unavailable.timeout, and this PR accepted that in TE-5. Onmain, the same tool staysincomplete, whichmain's TE-5 check accepts, so the merge removed this PR's own TE-5 exception.The TE-9 row is covered by the second commit. The run used the first commit. In one stopped row, the model's last reply had already returned, but the stop ended OpenCode before it saved any part of that reply. The cut message had no parts and no turn, so its completed call joined no step (
assistant_message_call_unmatched), and TE-9 failed. The call itself was still captured. With the second commit, the gap names the invocation and the checker accepts that one call. I checked the recorded row again with the gap as the new code writes it, and every check passes.Not in this PR
A desktop benchmark needs more before Codex or OpenCode can use this stop:
POST /episode_status. The OSWorld server onmaindoes not yet, and it also lacks the typed seed and close and the desktop tools that a session harness calls. That is a separate OSWorld change.episode_status_poll_seconds(5 s in the runs above).operations.background_exec: true. On other providers, the stop of a running harness answers HTTP 501.turn_interrupted. A stopped Codex row was not tested against the harness checks.Validation
At the new head (merge of
mainat82f574a83):pytest environment_servers/single_agent_turn/tests tests/unit_tests/server_runtime tests/unit_tests/rollouts tests/unit_tests/agents tests/unit_tests/trajectory tests/unit_tests/test_public_api.py tests/unit_tests/test_fern_docs_links.py: 6296 passed, 12 skipped.pytest environment_servers/single_agent_turn_legacy/tests: 48 passed. It runs on its own, because both server suites have atests/test_app.py.pytest responses_api_agents/opencode_agent/tests(its own venv, from the repository root): 925 passed, 23 skipped.pre-commit run --fileson the PR's files: passed.invocation_id. A new test failed each time.Compatibility
New, opt-in API and option. Rows without a stop are unchanged. The only change for existing rows is the cut step: a run that timed out on its own now gets
turn_interruptedfor its last unfinished step instead ofturns_unavailable.🤖 Generated with Claude Code