Repository navigation
feat(terminal-bench): support Pi, Codex, OpenCode and OpenClaw - #3902
Conversation
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> (cherry picked from commit 68981f0)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> (cherry picked from commit 5363dc6)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> (cherry picked from commit 7311f2d)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> (cherry picked from commit f5387fc)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> (cherry picked from commit 2759092)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> (cherry picked from commit 5052bce)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> (cherry picked from commit fd72679)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> (cherry picked from commit 030ca08)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> (cherry picked from commit 672b1f5)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Extract benchmark-independent native Pi defaults and compose them from the SWE-bench recipe. Support Python 3.8 and x86_64 musl task images, preserve installer diagnostics, and bound tool/model waits without replacing task runtimes. Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Remove the Pi-specific SWE-bench recipe and combined deployment aliases. Load benchmark Resources and Pi configs independently, bind them in the run configuration, and declare Pi compatibility on the shared benchmark config. Exercise both benchmark and independent task contracts through the native collector lifecycle. Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Promote the EnvironmentServer configuration to pi_agent.yaml and remove the separate suffixed configuration. Document task sessions as the standard entry point and local execution as compatibility. Use session/sandbox terminology consistently in the adapter, installer, tests, and examples. Follow the standard-entry-point cleanup in Hermes PR NVIDIA-NeMo#3554 (6dc9c53). Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Integrate the approved #11 changes. Route Hermes-specific callers and tests to NVIDIA-NeMo#3961. Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Follow the second NVIDIA-NeMo#3902 review as resolved in ab9f3cc: - On timeout or stop, the supervisor records already adopted children, stops the harness group, then drains newly adopted children across sessions. Natural exit only drains the group. State the remaining limits and that harnesses start each tool command in its own session. - Keep every harness process in the supervised group; disable self-respawn, as with OPENCLAW_NO_RESPAWN=1. - Make harness hooks fail closed with a readiness receipt, and bound downloaded logs. - Note the Terminal Bench 2.1 close path that has no automatic retry. - Add the AGENTS.md runtime-install rule for sandbox and host paths. - Expand termination validation and test hygiene. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
There was a problem hiding this comment.
Looks great!
Non-blocking follow-ups:
- The sentence saying the new runs validate the adoption-cutoff policy could be scoped. The certificate runs exercised the deadline drain under a real model. Preserving a service that was adopted before the stop is covered only by the Linux regressions.
- The sandboxed Hermes README says the cutoff applies "on timeout or close", but Hermes passes no stop file, so it applies only at timeout (README.md#L113).
sandbox_session.py#L63still says detached task services remain until the owner destroys the sandbox. A service launched by a tool that is still running at the stop is drained.
|
|
||
| The supervisor stops live processes in the harness group and reaps its children before | ||
| capture or verification. On timeout or close, it first records its already adopted | ||
| children, typically services from completed tool calls. It then stops the harness and |
There was a problem hiding this comment.
Two things here would help recipe authors. First, anything the supervisor adopted before the stop survives, not only services. That includes a background job such as make & left by an earlier tool call, or a backgrounded child of a tool whose shell already exited, and both can keep writing through verification. Second, the service rule has a timing side: the tool call that launches a service has to return before the harness deadline or stop. A service whose launching tool is still running at that point, or that is orphaned during the stop's grace period, is drained along with the in-flight tools.
|
/ok to test 30b121d |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38605636154 |
… pin the per-callback budget main (#3902) already bounds each cleanup callback on its own. This keeps the two pieces that were not there: a log line naming the callback that timed out, and a test that pins the per-callback budget (a slow close no longer eats the time a hung one left, and the sandbox stop at the bottom of the stack still runs). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>
… pin the per-callback budget (NVIDIA-NeMo#4603) ## What main already bounds each cleanup callback on its own since NVIDIA-NeMo#3902 (`CleanupContext.aclose` runs every callback under its own `cleanup_timeout_seconds`). This PR keeps the two pieces that were not part of that change: - an explicit `TimeoutError` branch that logs the name of the callback that timed out (the generic branch logged a traceback without saying which owner hung); - `test_each_cleanup_has_its_own_budget`, which pins that a slow close no longer eats the time a hung one left and that the sandbox stop at the bottom of the stack still runs, with a bound on total time. Rebased onto main after NVIDIA-NeMo#3902; the earlier version of this PR carried the per-callback loop itself. ## Test ``` pytest tests/unit_tests/test_environment_server.py -q # 21 passed ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Adil Asif <adasif@nvidia.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Clarify the cleanup contract for task recipes and shared sandbox sessions, following the [non-blocking approval notes on #3902](#3902 (review)) and the [service-timing comment](#3902 (comment)). - Explain that previously adopted children outside the harness group include unfinished background jobs such as `make &`, which can keep writing during verification. Describe the service-launch timing needed before a forced stop and the cleanup of newly orphaned children. - Correct sandboxed Hermes guidance: its adoption cutoff applies at the harness timeout, not on close, because the runner never signals the supervisor (a stop file only fences delayed launches). - Qualify `SandboxSession`'s service-lifetime docstring, including the natural-exit limitation. - Correct the drain timing: in-flight tools are killed during the drain after the grace period, adoption follows the launching process's exit, and descendants of a preserved process also survive. The associated #3902 description correction scopes the evidence: Pi/Codex/OpenCode certificate runs exercise deadline drain under a real model; preservation of services already adopted before a forced stop is covered only by Linux regressions. Synthetic service controls exercise normal completion. No separate issue is needed for these small corrections to the existing review. Runtime behavior, configuration and benchmark scoring are unchanged. Validation: `pre-commit run --from-ref origin/main --to-ref HEAD` and `git diff --check` pass. A Python AST comparison confirms `sandbox_session.py` has identical executable code to `main` after stripping docstrings. `process_supervisor.py`, `supervisor_client.py` and `hermes_sandboxed_agent/app.py` are unchanged on `main` since the reviewed #3902 head (`30b121d4b`), so the documented behavior still matches the code. Runtime tests and fresh model smoke: N/A for prose-only changes. Supersedes #4604, which closed automatically when #3902 merged and its `pull-request/3902` CI mirror base was deleted. This branch is rebased onto `main` (after the #3902 squash merge, 8bd25f3), so the diff contains only these three files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Enable Pi, Codex, OpenCode and OpenClaw on Terminal-Bench 2.1 through the shared Resources/Agent session protocol. Resources prepares one task sandbox, the harness borrows it, verification runs in that same sandbox, and Resources closes it afterward.
Consolidates #4064, #4241 and #4242 into #3902 after their parent integrations merged into main. Those sibling PRs remain closed; these review repairs stay in this PR. The existing benchmark integration and author-requested consolidation do not need another feature issue. Legacy lifecycle retirement is tracked in #4570 under #2763.
Remaining non-blocking documentation follow-ups are in #4604: task-service adoption timing, Hermes timeout-only cleanup, and
SandboxSessionlifetime wording. The smoke-evidence scope is already corrected below. #4604 is a draft based on this PR; after #3902 merges, rebase and retarget #4604 tomain.Changes and compatibility
materialize_task, andsingle_agent_turn. Existing direct-agent recipes remain available.session_close_retry_window_seconds(default 300 seconds); retries do not extend it. Close fences slow setup before waiting for its lock, and late setup attempts bounded cleanup; failed stops retain ownership. After a cancelled final close and failed late stop, no durable automatic retry is scheduled: explicit close retry or shutdown can use the retained handle, while provider TTL can stop the sandbox but does not purge the in-memory record.preserve_descendants_on_successpolicy switch.shell.envhook. A fresh successful-initialization receipt is required before accepting a gradable result; missing/invalid receipt fails activation even when the CLI ignored a plugin error. This check prevents grading, not earlier execution. Borrowed agent close removes the private session directory; Resources retains the task sandbox.OPENCLAW_NO_RESPAWN=1so its pinned CLI does not detach the agent during non-TTY startup. Compile-cache wrappers may still spawn within the supervised group.keep_sandbox_for_close. Session verification grades the existing sandbox; legacyverifyowns its bounded cleanup; golden verification owns a separate sandbox and boundedfinallycleanup. A legacy seed created before golden verification is also closed.This changes shared process-lifetime behavior. Services from completed tools must use task-owned files and avoid retaining transport streams; the pinned harnesses already run shell tools in separate sessions. Natural CLI exit has no stop-time cutoff, so unfinished detached tools cannot be distinguished from services and may still change task files. Adoption timing, PID reuse and later service forks also limit the forced-stop rule. A positive receipt does not guarantee an idle sandbox and is not a security boundary. SWE-Pro still extracts a patch and grades it in a fresh sandbox; TB grades the original task state. No original TB task prompt, dataset, scoring formula, or default permission profile is changed. Sessions remain process-local and require one worker.
Lifecycle
The Agent and Resources servers run outside the task sandbox. Normal gradable completion is:
sequenceDiagram participant E as Environment server participant R as Resources server participant A as Agent server participant S as Task sandbox E->>R: Seed Resources session R->>S: Create and set up sandbox R-->>E: Sandbox access E->>A: Seed agent with sandbox access E->>A: Run agent A->>S: Harness CLI and tools Note over A,S: Stop live harness-group processes and reap children Note over A,S: On timeout/close, also drain newly adopted tools Note over A,S: Confirm termination, then capture artifacts A-->>E: Gradable agent response E->>A: Close agent session A->>S: Remove private harness session files A-->>E: Release borrowed connection Note over R,S: Task files and preserved services remain E->>R: Verify R->>S: Run task tests in this same sandbox R-->>E: Verdict E->>R: Final cleanup R->>S: Stop sandbox and remaining processesOn failure or cancellation, final cleanup still attempts each registered owner. Successful session release alone does not make a failed activation gradable. Confirmed harness termination permits capture, not a passing reward. Golden verification creates and closes its own sandbox and has no agent session.
Validation — current review repair
Runtime repair
8d8809e13, merged with maine085a1d2ain6f6f9ac3e; final test-only follow-upab9f3cc27. Three independent agents implemented, challenged and cross-reviewed the nine comment fixes over multiple rounds. No additional PR was created.7765f9bcac97fd7a7f81e398ff9286c0329896bf. Includes shared supervisor/session/environment/checkpoint lifecycle, both environment adapters, TB/SWE-Pro, four harness suites and affected Hermes suites. Pinned CLI tests were enabled, not skipped.OPENCLAW_NO_RESPAWNto 0 fails its PGID assertion. Pinned OpenCode loader tests reproduce ignored plugin failures with no receipt. Final pinned-runtime selection: 13 passed, 0 skipped (overlaps the full suite).pre-commit run --all-files, Ruff and diff checks pass; Fernnpm run check: zero errors, one warning. All three new commits have DCO sign-off.Fresh real-model smoke: Qwen2.5-7B-Instruct revision
a09a35458c702b33eeacc393d103063234e8bc28, vLLM 0.8.5 on RTX A6000; temperature 0, 32,768 context, 4,096 provider output tokens/call, 300s harness/verifier, 1,200s episode, 1,800s sandbox TTL, 2 CPU/4,096 MiB. Dataset7131e4375048a0e408a8fb404b5f499d726b695b; original task/test bytes and images retained, current taskset routing and default harness permissions unchanged.12/12 attempts graded; no failed activation or missing row; all four strict 3/3 profiles pass. Four synthetic controls each contain an actual native tool call and verify task HOME/custom XDG markers, an HTTP service alive after agent close, removed private session directories, and unchanged task OpenCode config. Eight original-task zeros have
evaluation_completed=trueand task-test failures, not verifier infrastructure exceptions. These scores are not benchmark successes.Native responses include five completed and seven incomplete results. Pi log-summary and OpenClaw certificate/log-summary have no actual tool calls despite textual tool markup. Pi/Codex/OpenCode certificate runs exercise partial-result grading after time limits. Native observations/token joins have gaps; strict profile completion does not certify rollout quality or full training-token parity. Automated health reports Codex 2 unhealthy/1 unobserved (failed model endings, including output/context limits) and OpenCode 1 unhealthy/2 healthy (token-count mismatch). Pi and OpenClaw are 3 healthy each; that does not imply task success. The explicitly detached control service does not establish compatibility for every service-launch recipe.
Smoke source is tree
4f455a30592b106eafba853cf64db193a5e3c4e6(6f6f9ac3e). The only later changes are three test files inab9f3cc27; all runtime/docs bytes match. Each arm's source hashes stayed unchanged. All 12 containers have normal destroy events, no fallback removal or forced Gym stop; task-owned model/Gym processes, control image and ports are cleaned, and raw evidence is archived. Existing unrelated workloads were preserved.Historical evidence and remaining gates
Earlier repair
c2a6dec84had 1,926 Linux passes plus a 397-test OpenCode follow-up. Its 16 retained model attempts included one OpenCode startup failure before any model call; a separate repeat graded zero but did not explain or erase that failure. Earlier repair report. That evidence and the October 1206bd366b5smoke predate the current adoption-cutoff policy. The new Pi/Codex/OpenCode certificate runs exercise deadline drain under a real model. Preserving a service already adopted before a forced stop is covered by the Linux regressions only; the synthetic service controls cover normal completion and do not establish that forced-stop preservation behavior.The full PR touches shared core code and requires full CI and maintainer review. Earlier CI at
3dd3ae027passed core/server shards but reported a Hermes P0model_errorconformance failure; this local validation does not claim to clear that scenario. Exact-current-head CI remains a separate gate. No trust-gate command was posted.single_agent_turn_legacydoes not retire those agents.