Skip to content

fix(opencode): preserve failed answers after sandbox termination - #5099

Closed
Kipok wants to merge 4 commits into
mainfrom
codex/opencode-ended-sandbox-public
Closed

Kipok wants to merge 4 commits into
mainfrom
codex/opencode-ended-sandbox-public

Conversation

@Kipok

@Kipok Kipok commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

When OpenCode execution fails and its sandbox has ended, the subsequent MCP setup-marker check can raise another provider exception. This aborts normal failed-answer recording and can cause the unanswered task to be retried on resume.

Preserve the existing failed-answer path only when execution already failed and the marker-check exception explicitly contains [SANDBOX::ENDED]. Actual MCP initialization failures, unrelated transport errors, and sandbox-ended errors after successful execution still propagate. The existing zero-reward option remains configurable; prompts, budgets, and graders are unchanged. Existing artifacts are not repaired.

Adds a three-case regression for this exception and both propagation guards. Follow-up to #3575; no separate issue is needed because the test provides a reproducer.

Validation on exact PR head 92771f8516f7b58a185ca9a7326e5d4db0a3110f:

  • Complete responses_api_agents/opencode_sandboxed_agent/tests/test_app.py: 72 passed, including ended-sandbox guards, MCP setup failures, export, grading and cleanup unit coverage.
  • pre-commit run --all-files and the PR diff whitespace check passed; commits have DCO sign-offs.
  • Fresh live OpenCode 1.17.11 smoke through the unmodified native model, authenticated Tavily MCP and equivalence-judge servers: one completed tavily_web_search call, exported final answer “Jensen Huang,” judge verdict [[A=B]], reward 1.0, HTTP 200, OpenCode exit 0, and no agent failure. Policy/judge: openai/openai/gpt-5.6-luna. CPU job 1555942 exited 0.
  • Real sandbox cleanup completed; a subsequent provider lookup confirmed KUBERNETES::SANDBOX_NOT_FOUND. Inspected the native export, Gym response, judge output and teardown receipt.

The smoke used the shipped benchmark agent preset with OpenSandbox and network_access=model_and_tools, retaining its context/step budgets, disabled compaction, remaining-context output policy and configurable zero-reward setting. It used a synthetic search question, not a benchmark evaluation. Full configuration, command and evidence location are recorded in the validation comment. The live successful path validates integration and teardown; the terminated-sandbox failure remains covered by the deterministic regression. Trusted CI and reviewer sign-off remain outstanding.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok Kipok added area:agent Agent harnesses and Responses API agent behavior bug Something isn't working labels Oct 26, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 26, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Kipok
Kipok marked this pull request as ready for review October 26, 2026 21:50
@Kipok Kipok added the needs-review PR is ready for code review and waiting on a reviewer label Oct 26, 2026

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP WITH CARE — no implementation defect found; required rollout validation remains outstanding.

NOTE — Complete the required model-backed smoke validation. The protected repository quality bar requires representative real model rollouts for behavior-changing agent work. The PR description explicitly says none was run on this branch. The new regression meaningfully checks the exception guards, but mocks the sandbox and calls responses() directly; it does not validate real provider/MCP/verifier integration. This is a validation gap, not a demonstrated runtime failure.

Correction: Run a representative MCP-enabled OpenCode task with a real model at the reviewed head, inspect agent and verifier behavior, and record the exact configuration, command, and results in the PR. No source patch is indicated.

Preservation and acceptance: Keep prompts, budgets, graders, and the configurable zero-reward option unchanged. The proposed smoke check should confirm MCP initialization and an actual tool call, a collected response/export, a consistent verifier result, and sandbox cleanup. Retain these deterministic assertions:

  • Execution raises, then the marker check raises [SANDBOX::ENDED]: empty output is returned and the execution failure is recorded.
  • An unrelated marker exception, or [SANDBOX::ENDED] after successful execution: the exception propagates without recording a result.
  • A returned MCP setup failure still raises before export/download or scoring, as asserted by the existing setup-failure test.

Both changed files and directly affected execution, export, observation, verification, and cleanup paths were reviewed. Validation here was static only; no tests or rollouts were executed.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok

Kipok commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

Merged upstream main 778a37bf3. pre-commit run --all-files and the PR diff whitespace check pass; the merge commit has a DCO sign-off. Resolved the test-file conflict by retaining both the ended-sandbox regression and main’s new observation/timing tests. The complete OpenCode adapter test file now passes all 72 tests, including ended-sandbox handling, propagation guards, MCP setup failure, export, grading and cleanup unit coverage. Existing production rollout evidence is now verified: the October 27 max-harness suite is pinned to Gym eb0d449529a0b4f5fdff1e37565391e27d0ad1e2, whose ended-sandbox handling block matches this fix. In its Kimi K3 HLE Diamond OpenCode+search job 1492522, task 1/repeat 0 has five completed tavily_web_search MCP calls, three completed bash calls, an exported OpenCode transcript, exit code 0, no agent failure, and a native judge reward of 1 (answer 8). The record includes a completed OpenSandbox execution. I inspected the saved response, tool observations and judge result, and compared the source block; no new generation or score alteration was performed.

Configuration: benchmarks/hle_diamond/rf_opencode_full_search.yaml in the pinned experiment checkout, opencode_sandboxed_agent, network_access=model_and_tools, Tavily exposed over MCP, OpenSandbox, Kimi K3 with max model-side thinking effort. The suite's source receipt and exact resolved/submission configurations are retained locally under .runs/max-harness-suite-20261027/; the evidence sample is from the q000-099 shard.

Limits: this is production evidence for the same fix, not a fresh run of PR head 92771f851. The experiment checkout also contains prompt/image and final-answer changes outside this PR. Sandbox execution is recorded, but teardown success is not independently attested by the rollout (lifecycle timing is unavailable); cleanup is covered by the current branch's deterministic suite. A strict current-head live smoke/teardown check therefore remains open. No prompt, budget, grader or zero-reward setting was changed in this update.

@Kipok

Kipok commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Completed the requested live MCP-enabled smoke at exact PR head 92771f8516f7b58a185ca9a7326e5d4db0a3110f (CPU job 1555942). No source changes were needed.

Inspected both the saved native OpenCode export and Gym verify response:

  • Authenticated Tavily MCP initialized; one tavily_web_search call completed and its real search results reached the model.
  • Final answer: Jensen Huang; native equivalence judge returned [[A=B]], reward 1.0.
  • /run returned HTTP 200, OpenCode exited 0, opencode_finished=true, opencode_export_found=true, opencode_failed=false.
  • Production cleanup completed (stopped=true). A subsequent provider lookup confirmed sandbox 628d4b96-3f22-405a-9ec6-28ceaa2e802d was deleted: KUBERNETES::SANDBOX_NOT_FOUND. The smoke process and Slurm job both exited 0.

Configuration: shipped responses_api_agents/opencode_sandboxed_agent/configs/benchmark.yaml, OpenCode 1.17.11, OpenSandbox, network_access=model_and_tools, authenticated tavily_search with the HLE exclusion list, and the shipped equivalence-judge prompt. Policy and judge used openai/openai/gpt-5.6-luna through the native openai_model adapter. The benchmark preset's context limit (262144), 400-step limit, remaining-context output policy, disabled compaction, and configurable execution_failure_reward_zero=true were retained. This was a synthetic integration task, not a benchmark score: “Use the Tavily search tool to look up who co-founded NVIDIA in 1993 and is its CEO. Give that person's name as your final answer.” Expected answer: Jensen Huang.

Exact launch: sbatch --parsable smoke.sbatch; inside the allocation: .venv/bin/python "$SMOKE_OUTPUT_DIR/opencode_smoke_remote.py". The harness, launch script, input, response, exported transcript and teardown receipt are retained under /scratch/fsw/portfolios/nemotron/projects/nemotron_n4_post/users/igitman/gym-pr-followup-20261029/ on oci-aga. The harness instantiates the unmodified PR servers and observes the real sandbox handle; model calls, MCP, execution, export, verification and cleanup are not mocked. Temporary copied credentials were removed after completion.

Also reran the complete OpenCode adapter test file: 72 passed, including ended-sandbox and exception-propagation guards. pre-commit run --all-files passes. The successful live run validates integration and cleanup; the ended-sandbox failure itself remains covered by the deterministic regression. Trusted CI/reviewer sign-off remain outstanding.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok

Kipok commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Merged upstream main 483674910 in ad7846e237c0bcccb68c51472e7c94d0f62f25cd and reconciled the package-layout changes.

83 OpenCode adapter tests pass, including all ended-sandbox propagation guards. The current merge preserves main’s observation and timing changes. The prior real MCP/model smoke and verified sandbox teardown remain documented above.

pre-commit run --all-files, diff whitespace checks and 724 public-API/import-boundary tests pass. This update merges main and adapts imports; no new model rollout or full local coverage run is claimed. Trusted CI needs vetter authorization for the new head.

@cmunley1

cmunley1 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

/ok to test ad7846e

@cmunley1

cmunley1 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

/review

yaoyu-33
yaoyu-33 previously approved these changes Nov 4, 2026

@yaoyu-33 yaoyu-33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The catch only applies when execution already failed and the sandbox has ended, so real MCP setup failures still surface. The three-case regression covers both guards.

There's one small fix inline; with it, this is good to merge. CI is green apart from the Autobot check.

@yaoyu-33 yaoyu-33 added ready-to-merge PR is approved, current, and waiting for final CI or merge and removed needs-review PR is ready for code review and waiting on a reviewer labels Nov 4, 2026

Copy link
Copy Markdown
Contributor

🤖 On it — handling this as babysit-ready-to-merge.

Copy link
Copy Markdown
Contributor

I'll watch this ready-to-merge PR and report substantive progress. Running on nim-gpt-6-1-sol.

Copy link
Copy Markdown
Contributor

🤖 On it — handling this as babysit-ready-to-merge.

Signed-off-by: Igor Gitman <igitman@nvidia.com>
@Kipok

Kipok commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Fixed in b9f8b717a: detect SANDBOX::ENDED in both bracketed and HTTP 410 JSON error formats. The regression now exercises both formats with and without an execution failure, and retains rejection of unrelated connection failures. All 86 OpenCode app tests, pre-commit and whitespace checks pass. This extends the existing failure-preservation rule; no fresh model rollout was run. New head needs trusted-CI authorization.

Copy link
Copy Markdown
Contributor

💬 Head b9f8b717 has passed lint, DCO, copyright and secret checks, but the main test checks are missing. The author reports that this head needs maintainer CI authorization; missing checks can’t be retried. I’ll keep watching for CI and merge.

Copy link
Copy Markdown
Contributor

💬 I can’t verify the latest PR, main, or CI state because the GitHub access broker keeps returning HTTP 503. The last verified PR head was b9f8b717, open and unmerged. I haven’t changed code or retried CI without the required evidence; the PR watch remains active.

Copy link
Copy Markdown
Contributor

💬 I could read GitHub again. Head b9f8b717 has no failing checks, but trusted test checks are still absent and GitHub reports REVIEW_REQUIRED. The PR remains open and unmerged; I’ll keep watching for maintainer review and current-head CI authorization.

@adil-a

adil-a commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

/ok to test b9f8b71

@adil-a
adil-a enabled auto-merge November 5, 2026 18:38

Copy link
Copy Markdown
Contributor

💬 Trusted CI is now running on b9f8b717, and GitHub reports APPROVED. Core, sandbox, and server tests are still in progress; the PR remains open and unmerged.

Copy link
Copy Markdown
Contributor

💬 The non-blocking KiloCode conformance check failed in model_error: the rollout returned HTTP 500 without demonstrating the required model-error outcome. The matching job passed on current main. I’m comparing the artifacts before attributing the failure or retrying CI; the PR remains open.

@adil-a
adil-a added this pull request to the merge queue Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42091635253

Copy link
Copy Markdown
Contributor

💬 The KiloCode probe exposes older error handling on b9f8b717: a controlled HTTP 400 becomes an HTTP 500 without ng_agent_outcome. Merged PR #5377 already adds the missing handling on main, where the same scenario passed. The core, sandbox, and server test suites passed. There’s no evidence of flakiness, so I won’t retry the unchanged head. The author can incorporate the merged fix and recheck the probe; I’ll keep watching for an update or merge.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 5, 2026

Copy link
Copy Markdown
Contributor

💬 Merge-queue validation failed in all six variants of the new ended-sandbox regression. The mock sandbox.download doesn’t create required-mcp.js, so main’s plugin read-back check fails before reaching the intended assertions. The author needs to update that fixture and rerun CI; retrying unchanged code won’t address it. The PR remains open, and I’ll keep watching.

@Kipok

Kipok commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Closing as superseded by merged #5072 and #5343. Current main raises a typed agent failure when sandbox execution has no reliable command result and preserves that evidence for recovery; it no longer turns this provider exception into a failed answer/zero. Consequently run_error_type stays unset and this PR's MCP-marker exception fallback is unreachable. Restoring the old fallback would conflict with the new failure/recovery contract. The six old regression cases fail against current main because they encode that superseded behavior, in addition to needing the newer plugin upload/readback fixture.

@Kipok Kipok closed this Nov 5, 2026

Copy link
Copy Markdown
Contributor

The PR is no longer an open ready-to-merge PR targeting main.

This branch was previously deployed

1 inactive deployment
public — b9f8b717 Deployed Nov 5, 2026 by copy-pr-bot[bot] via release / finalize / notify #10422
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior bug Something isn't working ready-to-merge PR is approved, current, and waiting for final CI or merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants