Repository navigation
Conversation
Signed-off-by: Igor Gitman <igitman@nvidia.com>
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP WITH CARE — no implementation defect found; required rollout validation remains outstanding.
NOTE — Complete the required model-backed smoke validation. The protected repository quality bar requires representative real model rollouts for behavior-changing agent work. The PR description explicitly says none was run on this branch. The new regression meaningfully checks the exception guards, but mocks the sandbox and calls responses() directly; it does not validate real provider/MCP/verifier integration. This is a validation gap, not a demonstrated runtime failure.
Correction: Run a representative MCP-enabled OpenCode task with a real model at the reviewed head, inspect agent and verifier behavior, and record the exact configuration, command, and results in the PR. No source patch is indicated.
Preservation and acceptance: Keep prompts, budgets, graders, and the configurable zero-reward option unchanged. The proposed smoke check should confirm MCP initialization and an actual tool call, a collected response/export, a consistent verifier result, and sandbox cleanup. Retain these deterministic assertions:
- Execution raises, then the marker check raises
[SANDBOX::ENDED]: empty output is returned and the execution failure is recorded. - An unrelated marker exception, or
[SANDBOX::ENDED]after successful execution: the exception propagates without recording a result. - A returned MCP setup failure still raises before export/download or scoring, as asserted by the existing setup-failure test.
Both changed files and directly affected execution, export, observation, verification, and cleanup paths were reviewed. Validation here was static only; no tests or rollouts were executed.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
Signed-off-by: Igor Gitman <igitman@nvidia.com>
|
Merged upstream main Configuration: Limits: this is production evidence for the same fix, not a fresh run of PR head |
|
Completed the requested live MCP-enabled smoke at exact PR head Inspected both the saved native OpenCode export and Gym verify response:
Configuration: shipped Exact launch: Also reran the complete OpenCode adapter test file: 72 passed, including ended-sandbox and exception-propagation guards. |
Signed-off-by: Igor Gitman <igitman@nvidia.com>
|
Merged upstream main 83 OpenCode adapter tests pass, including all ended-sandbox propagation guards. The current merge preserves main’s observation and timing changes. The prior real MCP/model smoke and verified sandbox teardown remain documented above.
|
|
/ok to test ad7846e |
|
/review |
yaoyu-33
left a comment
There was a problem hiding this comment.
Thanks. The catch only applies when execution already failed and the sandbox has ended, so real MCP setup failures still surface. The three-case regression covers both guards.
There's one small fix inline; with it, this is good to merge. CI is green apart from the Autobot check.
|
🤖 On it — handling this as babysit-ready-to-merge. |
|
I'll watch this ready-to-merge PR and report substantive progress. Running on |
|
🤖 On it — handling this as babysit-ready-to-merge. |
Signed-off-by: Igor Gitman <igitman@nvidia.com>
|
Fixed in |
|
💬 Head |
|
💬 I can’t verify the latest PR, main, or CI state because the GitHub access broker keeps returning HTTP 503. The last verified PR head was |
|
💬 I could read GitHub again. Head |
|
/ok to test b9f8b71 |
|
💬 Trusted CI is now running on |
|
💬 The non-blocking KiloCode conformance check failed in |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42091635253 |
|
💬 The KiloCode probe exposes older error handling on |
|
💬 Merge-queue validation failed in all six variants of the new ended-sandbox regression. The mock |
|
Closing as superseded by merged #5072 and #5343. Current main raises a typed agent failure when sandbox execution has no reliable command result and preserves that evidence for recovery; it no longer turns this provider exception into a failed answer/zero. Consequently |
|
The PR is no longer an open ready-to-merge PR targeting main. |
When OpenCode execution fails and its sandbox has ended, the subsequent MCP setup-marker check can raise another provider exception. This aborts normal failed-answer recording and can cause the unanswered task to be retried on resume.
Preserve the existing failed-answer path only when execution already failed and the marker-check exception explicitly contains
[SANDBOX::ENDED]. Actual MCP initialization failures, unrelated transport errors, and sandbox-ended errors after successful execution still propagate. The existing zero-reward option remains configurable; prompts, budgets, and graders are unchanged. Existing artifacts are not repaired.Adds a three-case regression for this exception and both propagation guards. Follow-up to #3575; no separate issue is needed because the test provides a reproducer.
Validation on exact PR head
92771f8516f7b58a185ca9a7326e5d4db0a3110f:responses_api_agents/opencode_sandboxed_agent/tests/test_app.py: 72 passed, including ended-sandbox guards, MCP setup failures, export, grading and cleanup unit coverage.pre-commit run --all-filesand the PR diff whitespace check passed; commits have DCO sign-offs.tavily_web_searchcall, exported final answer “Jensen Huang,” judge verdict[[A=B]], reward 1.0, HTTP 200, OpenCode exit 0, and no agent failure. Policy/judge:openai/openai/gpt-5.6-luna. CPU job 1555942 exited 0.KUBERNETES::SANDBOX_NOT_FOUND. Inspected the native export, Gym response, judge output and teardown receipt.The smoke used the shipped benchmark agent preset with OpenSandbox and
network_access=model_and_tools, retaining its context/step budgets, disabled compaction, remaining-context output policy and configurable zero-reward setting. It used a synthetic search question, not a benchmark evaluation. Full configuration, command and evidence location are recorded in the validation comment. The live successful path validates integration and teardown; the terminated-sandbox failure remains covered by the deterministic regression. Trusted CI and reviewer sign-off remain outstanding.