Skip to content

feat(terminal-bench): support Pi, Codex, OpenCode and OpenClaw - #3902

Merged
yaoyu-33 merged 100 commits into
NVIDIA-NeMo:mainfrom
yaoyu-33:codex/pi-tb21-native
Oct 14, 2026
Merged

yaoyu-33 merged 100 commits into
NVIDIA-NeMo:mainfrom
yaoyu-33:codex/pi-tb21-native

Conversation

@yaoyu-33

@yaoyu-33 yaoyu-33 commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Enable Pi, Codex, OpenCode and OpenClaw on Terminal-Bench 2.1 through the shared Resources/Agent session protocol. Resources prepares one task sandbox, the harness borrows it, verification runs in that same sandbox, and Resources closes it afterward.

Consolidates #4064, #4241 and #4242 into #3902 after their parent integrations merged into main. Those sibling PRs remain closed; these review repairs stay in this PR. The existing benchmark integration and author-requested consolidation do not need another feature issue. Legacy lifecycle retirement is tracked in #4570 under #2763.

Remaining non-blocking documentation follow-ups are in #4604: task-service adoption timing, Hermes timeout-only cleanup, and SandboxSession lifetime wording. The smoke-evidence scope is already corrected below. #4604 is a draft based on this PR; after #3902 merges, rebase and retarget #4604 to main.

Changes and compatibility

  • One TB Resources session implementation and one benchmark recipe selecting any of the four harnesses. The recipe uses taskset routing, materialize_task, and single_agent_turn. Existing direct-agent recipes remain available.
  • Session seed/verification replay validates task identity; Resources keeps ownership through grading and retries failed stops. Completed close records expire after session_close_retry_window_seconds (default 300 seconds); retries do not extend it. Close fences slow setup before waiting for its lock, and late setup attempts bounded cleanup; failed stops retain ownership. After a cancelled final close and failed late stop, no durable automatic retry is scheduled: explicit close retry or shutdown can use the retained handle, while provider TTL can stop the sandbox but does not purge the in-memory record.
  • The Linux supervisor stops live harness-group processes and reaps its children. Before timeout/close, it snapshots already adopted children, typically services from completed tools. After stopping the harness, it repeatedly kills/reaps newly adopted children, including in-flight tools in separate sessions, within the existing cleanup budget. Previously adopted services outside the group remain for the sandbox owner. Zombie-only groups held by another live parent do not falsely fail termination. There is no preserve_descendants_on_success policy switch.
  • A valid final negative termination receipt fails activation and skips capture/verification, but permits borrowed-session release so Resources can tear down the sandbox. Missing or inconclusive termination evidence stays retryable; positive confirmation tolerates malformed optional diagnostics. Each final cleanup callback gets its own deadline, so an unresponsive agent close cannot consume the Resources owner's close attempt.
  • Task shell HOME is preserved. Pi/Codex/OpenClaw use their harness-specific state locations. OpenCode keeps private CLI HOME/XDG state and restores task HOME/XDG for shell commands through its shell.env hook. A fresh successful-initialization receipt is required before accepting a gradable result; missing/invalid receipt fails activation even when the CLI ignored a plugin error. This check prevents grading, not earlier execution. Borrowed agent close removes the private session directory; Resources retains the task sandbox.
  • OpenClaw sets OPENCLAW_NO_RESPAWN=1 so its pinned CLI does not detach the agent during non-TTY startup. Compile-cache wrappers may still spawn within the supervised group.
  • Removed keep_sandbox_for_close. Session verification grades the existing sandbox; legacy verify owns its bounded cleanup; golden verification owns a separate sandbox and bounded finally cleanup. A legacy seed created before golden verification is also closed.
  • Direct sandboxed Hermes redirects inherited stdout/stderr to a session log, preventing detached services from holding the provider exec pipe open. It downloads only a bounded 1 MiB tail, including a truncation marker, for diagnostics. Shared caller docs and Linux regressions cover Hermes and SWE-Pro as well as the four TB harnesses.

This changes shared process-lifetime behavior. Services from completed tools must use task-owned files and avoid retaining transport streams; the pinned harnesses already run shell tools in separate sessions. Natural CLI exit has no stop-time cutoff, so unfinished detached tools cannot be distinguished from services and may still change task files. Adoption timing, PID reuse and later service forks also limit the forced-stop rule. A positive receipt does not guarantee an idle sandbox and is not a security boundary. SWE-Pro still extracts a patch and grades it in a fresh sandbox; TB grades the original task state. No original TB task prompt, dataset, scoring formula, or default permission profile is changed. Sessions remain process-local and require one worker.

Lifecycle

The Agent and Resources servers run outside the task sandbox. Normal gradable completion is:

sequenceDiagram
    participant E as Environment server
    participant R as Resources server
    participant A as Agent server
    participant S as Task sandbox
    E->>R: Seed Resources session
    R->>S: Create and set up sandbox
    R-->>E: Sandbox access
    E->>A: Seed agent with sandbox access
    E->>A: Run agent
    A->>S: Harness CLI and tools
    Note over A,S: Stop live harness-group processes and reap children
    Note over A,S: On timeout/close, also drain newly adopted tools
    Note over A,S: Confirm termination, then capture artifacts
    A-->>E: Gradable agent response
    E->>A: Close agent session
    A->>S: Remove private harness session files
    A-->>E: Release borrowed connection
    Note over R,S: Task files and preserved services remain
    E->>R: Verify
    R->>S: Run task tests in this same sandbox
    R-->>E: Verdict
    E->>R: Final cleanup
    R->>S: Stop sandbox and remaining processes
Loading

On failure or cancellation, final cleanup still attempts each registered owner. Successful session release alone does not make a failed activation gradable. Confirmed harness termination permits capture, not a passing reward. Golden verification creates and closes its own sandbox and has no agent session.

Validation — current review repair

Runtime repair 8d8809e13, merged with main e085a1d2a in 6f6f9ac3e; final test-only follow-up ab9f3cc27. Three independent agents implemented, challenged and cross-reviewed the nine comment fixes over multiple rounds. No additional PR was created.

  • Final Linux/Python 3.13.14 affected suite: 2,012 passed, 0 skipped, tree 7765f9bcac97fd7a7f81e398ff9286c0329896bf. Includes shared supervisor/session/environment/checkpoint lifecycle, both environment adapters, TB/SWE-Pro, four harness suites and affected Hermes suites. Pinned CLI tests were enabled, not skipped.
  • The first full attempt retained 2,011 passed, 1 failed: a Codex fixture modeled an in-flight child as a completed-tool service. The corrected fixture preserves the original alive-service/dead-CLI assertions; its whole session file passes 133 tests before the complete rerun.
  • All eight new detached-writer regressions fail against the previous supervisor; all 40 isolated supervisor tests pass after repair. Pinned OpenClaw non-TTY startup reaches a held model request in one group and stops; changing only OPENCLAW_NO_RESPAWN to 0 fails its PGID assertion. Pinned OpenCode loader tests reproduce ignored plugin failures with no receipt. Final pinned-runtime selection: 13 passed, 0 skipped (overlaps the full suite).
  • Removing only the second final-negative-receipt check fails the new regression; restoring unbounded Hermes output fails its diagnostic test.
  • Final pre-commit run --all-files, Ruff and diff checks pass; Fern npm run check: zero errors, one warning. All three new commits have DCO sign-off.

Fresh real-model smoke: Qwen2.5-7B-Instruct revision a09a35458c702b33eeacc393d103063234e8bc28, vLLM 0.8.5 on RTX A6000; temperature 0, 32,768 context, 4,096 provider output tokens/call, 300s harness/verifier, 1,200s episode, 1,800s sandbox TTL, 2 CPU/4,096 MiB. Dataset 7131e4375048a0e408a8fb404b5f499d726b695b; original task/test bytes and images retained, current taskset routing and default harness permissions unchanged.

Harness Original certificate task Original log-summary task Synthetic HOME/XDG/service control
Pi 0.80.2 graded 0 graded 0 reward 1
Codex 0.144.4 graded 0 graded 0 reward 1
OpenCode 1.17.11 graded 0 graded 0 reward 1
OpenClaw 2026.6.11 graded 0 graded 0 reward 1

12/12 attempts graded; no failed activation or missing row; all four strict 3/3 profiles pass. Four synthetic controls each contain an actual native tool call and verify task HOME/custom XDG markers, an HTTP service alive after agent close, removed private session directories, and unchanged task OpenCode config. Eight original-task zeros have evaluation_completed=true and task-test failures, not verifier infrastructure exceptions. These scores are not benchmark successes.

Native responses include five completed and seven incomplete results. Pi log-summary and OpenClaw certificate/log-summary have no actual tool calls despite textual tool markup. Pi/Codex/OpenCode certificate runs exercise partial-result grading after time limits. Native observations/token joins have gaps; strict profile completion does not certify rollout quality or full training-token parity. Automated health reports Codex 2 unhealthy/1 unobserved (failed model endings, including output/context limits) and OpenCode 1 unhealthy/2 healthy (token-count mismatch). Pi and OpenClaw are 3 healthy each; that does not imply task success. The explicitly detached control service does not establish compatibility for every service-launch recipe.

Smoke source is tree 4f455a30592b106eafba853cf64db193a5e3c4e6 (6f6f9ac3e). The only later changes are three test files in ab9f3cc27; all runtime/docs bytes match. Each arm's source hashes stayed unchanged. All 12 containers have normal destroy events, no fallback removal or forced Gym stop; task-owned model/Gym processes, control image and ports are cleaned, and raw evidence is archived. Existing unrelated workloads were preserved.

Historical evidence and remaining gates

Earlier repair c2a6dec84 had 1,926 Linux passes plus a 397-test OpenCode follow-up. Its 16 retained model attempts included one OpenCode startup failure before any model call; a separate repeat graded zero but did not explain or erase that failure. Earlier repair report. That evidence and the October 12 06bd366b5 smoke predate the current adoption-cutoff policy. The new Pi/Codex/OpenCode certificate runs exercise deadline drain under a real model. Preserving a service already adopted before a forced stop is covered by the Linux regressions only; the synthetic service controls cover normal completion and do not establish that forced-stop preservation behavior.

The full PR touches shared core code and requires full CI and maintainer review. Earlier CI at 3dd3ae027 passed core/server shards but reported a Hermes P0 model_error conformance failure; this local validation does not claim to clear that scenario. Exact-current-head CI remains a separate gate. No trust-gate command was posted.

@yaoyu-33 yaoyu-33 added feature New capabilities, enhancements, or enablement work area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers labels Oct 1, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
(cherry picked from commit 68981f0)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
(cherry picked from commit 5363dc6)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
(cherry picked from commit 7311f2d)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
(cherry picked from commit f5387fc)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
(cherry picked from commit 2759092)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
(cherry picked from commit 5052bce)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
(cherry picked from commit fd72679)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
(cherry picked from commit 030ca08)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
(cherry picked from commit 672b1f5)
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Extract benchmark-independent native Pi defaults and compose them from the SWE-bench recipe. Support Python 3.8 and x86_64 musl task images, preserve installer diagnostics, and bound tool/model waits without replacing task runtimes.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Remove the Pi-specific SWE-bench recipe and combined deployment aliases. Load benchmark Resources and Pi configs independently, bind them in the run configuration, and declare Pi compatibility on the shared benchmark config. Exercise both benchmark and independent task contracts through the native collector lifecycle.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Promote the EnvironmentServer configuration to pi_agent.yaml and remove the separate suffixed configuration. Document task sessions as the standard entry point and local execution as compatibility. Use session/sandbox terminology consistently in the adapter, installer, tests, and examples.

Follow the standard-entry-point cleanup in Hermes PR NVIDIA-NeMo#3554 (6dc9c53).

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Integrate the approved #11 changes. Route Hermes-specific callers and tests to NVIDIA-NeMo#3961.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
yaoyu-33 added a commit to yaoyu-33/Gym that referenced this pull request Oct 14, 2026
Follow the second NVIDIA-NeMo#3902 review as resolved in ab9f3cc:

- On timeout or stop, the supervisor records already adopted children,
  stops the harness group, then drains newly adopted children across
  sessions. Natural exit only drains the group. State the remaining
  limits and that harnesses start each tool command in its own session.
- Keep every harness process in the supervised group; disable
  self-respawn, as with OPENCLAW_NO_RESPAWN=1.
- Make harness hooks fail closed with a readiness receipt, and bound
  downloaded logs.
- Note the Terminal Bench 2.1 close path that has no automatic retry.
- Add the AGENTS.md runtime-install rule for sandbox and host paths.
- Expand termination validation and test hygiene.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>

@ananthsub ananthsub left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

Non-blocking follow-ups:

  1. The sentence saying the new runs validate the adoption-cutoff policy could be scoped. The certificate runs exercised the deadline drain under a real model. Preserving a service that was adopted before the stop is covered only by the Linux regressions.
  2. The sandboxed Hermes README says the cutoff applies "on timeout or close", but Hermes passes no stop file, so it applies only at timeout (README.md#L113).
  3. sandbox_session.py#L63 still says detached task services remain until the owner destroys the sandbox. A service launched by a tool that is still running at the stop is drained.


The supervisor stops live processes in the harness group and reaps its children before
capture or verification. On timeout or close, it first records its already adopted
children, typically services from completed tool calls. It then stops the harness and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things here would help recipe authors. First, anything the supervisor adopted before the stop survives, not only services. That includes a background job such as make & left by an earlier tool call, or a backgrounded child of a tool whose shell already exited, and both can keep writing through verification. Second, the service rule has a timing side: the tool call that launches a service has to return before the harness deadline or stop. A service whose launching tool is still running at that point, or that is orphaned during the stop's grace period, is drained along with the in-flight tools.

@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test 30b121d

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38605636154

Merged via the queue into NVIDIA-NeMo:main with commit 8bd25f3 Oct 14, 2026
51 checks passed
@yaoyu-33
yaoyu-33 deleted the codex/pi-tb21-native branch October 14, 2026 17:37
adil-a added a commit that referenced this pull request Oct 14, 2026
… pin the per-callback budget

main (#3902) already bounds each cleanup callback on its own. This keeps the two
pieces that were not there: a log line naming the callback that timed out, and a
test that pins the per-callback budget (a slow close no longer eats the time a hung
one left, and the sandbox stop at the bottom of the stack still runs).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Adil Asif <adasif@nvidia.com>
anson627 pushed a commit to anson627/Gym that referenced this pull request Oct 14, 2026
… pin the per-callback budget (NVIDIA-NeMo#4603)

## What

main already bounds each cleanup callback on its own since NVIDIA-NeMo#3902
(`CleanupContext.aclose` runs every callback under its own
`cleanup_timeout_seconds`). This PR keeps the two pieces that were not
part of that change:

- an explicit `TimeoutError` branch that logs the name of the callback
that timed out (the generic branch logged a traceback without saying
which owner hung);
- `test_each_cleanup_has_its_own_budget`, which pins that a slow close
no longer eats the time a hung one left and that the sandbox stop at the
bottom of the stack still runs, with a bound on total time.

Rebased onto main after NVIDIA-NeMo#3902; the earlier version of this PR carried
the per-callback loop itself.

## Test

```
pytest tests/unit_tests/test_environment_server.py -q   # 21 passed
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Adil Asif <adasif@nvidia.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
github-merge-queue Bot pushed a commit that referenced this pull request Oct 14, 2026
Clarify the cleanup contract for task recipes and shared sandbox
sessions, following the [non-blocking approval notes on
#3902](#3902 (review))
and the [service-timing
comment](#3902 (comment)).

- Explain that previously adopted children outside the harness group
include unfinished background jobs such as `make &`, which can keep
writing during verification. Describe the service-launch timing needed
before a forced stop and the cleanup of newly orphaned children.
- Correct sandboxed Hermes guidance: its adoption cutoff applies at the
harness timeout, not on close, because the runner never signals the
supervisor (a stop file only fences delayed launches).
- Qualify `SandboxSession`'s service-lifetime docstring, including the
natural-exit limitation.
- Correct the drain timing: in-flight tools are killed during the drain
after the grace period, adoption follows the launching process's exit,
and descendants of a preserved process also survive.

The associated #3902 description correction scopes the evidence:
Pi/Codex/OpenCode certificate runs exercise deadline drain under a real
model; preservation of services already adopted before a forced stop is
covered only by Linux regressions. Synthetic service controls exercise
normal completion.

No separate issue is needed for these small corrections to the existing
review. Runtime behavior, configuration and benchmark scoring are
unchanged.

Validation: `pre-commit run --from-ref origin/main --to-ref HEAD` and
`git diff --check` pass. A Python AST comparison confirms
`sandbox_session.py` has identical executable code to `main` after
stripping docstrings. `process_supervisor.py`, `supervisor_client.py`
and `hermes_sandboxed_agent/app.py` are unchanged on `main` since the
reviewed #3902 head (`30b121d4b`), so the documented behavior still
matches the code. Runtime tests and fresh model smoke: N/A for
prose-only changes.

Supersedes #4604, which closed automatically when #3902 merged and its
`pull-request/3902` CI mirror base was deleted. This branch is rebased
onto `main` (after the #3902 squash merge, 8bd25f3), so the diff
contains only these three files.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
public — 30b121d4 Deployed Oct 14, 2026 by copy-pr-bot[bot] via release / finalize / notify #5307
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants