Skip to content

feat(anyterminal): OpenSandbox oracle config, resource knobs, cgroup metrics, TB2.1 prepare - #2980

Draft
adil-a wants to merge 1 commit into
anyterminal-oracle-modefrom
anyterminal-oracle-opensandbox-arm
Draft

adil-a wants to merge 1 commit into
anyterminal-oracle-modefrom
anyterminal-oracle-opensandbox-arm

Conversation

@adil-a

@adil-a adil-a commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #2595 (oracle mode); both rebased onto current main. Diff vs base is the anyterminal changes only.

What

Everything needed to run Terminal-Bench 2.1 gold (oracle) verification on a remote OpenSandbox cluster, validated end-to-end on arm64 (86/89 pass) and amd64 (82/89) with all remaining failures attributed to upstream task rot or cluster egress, not architecture or images.

  • configs/anyterminal_oracle_opensandbox_arm.yaml — oracle config: provider referenced by name, image_auth/platform for a private arm64 image mirror, uniform 2c/4GiB sandboxes, CPU caps.
  • cpu_pin_enabled — injects main's nemo_gym.sandbox.utils.cpu_cap_env (derived from the sandbox CPU limit) into the agent/oracle/verifier execs so thread pools size to the cgroup quota (replaces the closed feat(sandbox): add per-exec CPU pinning #2601 approach).
  • override_cpus / override_memory_mb — uniform sandbox sizing (mirrors harbor_agent).
  • ray_task_num_cpus — local Ray reservation decoupled from task cpus so remote-sandbox concurrency isn't capped by the driver's core count (mirrors harbor_ray_task_num_cpus).
  • Per-rollout resource metrics from cgroup counters read at phase boundaries: cpu_solve/verify/total_sec, cpu_util_*, cpu_throttled_sec, mem_peak_mib (fail-open).
  • prepare.py gains a prepare() entrypoint for gym eval prepare (pinned terminal-bench-2-1 clone, same pin as harbor_agent).
  • .gitignore for the cloned task checkout and local results/.

Testing

  • pytest responses_api_agents/anyterminal_agent/tests/ — 95 passed (6 TestBuildProvider cases need a docker binary; unchanged, deselected locally).
  • Real rollouts: full TB2.1 oracle sweeps on arm64 and amd64 via OpenSandbox with the new metrics on every row (cpu_util 0.02–0.83 of the 2-core limit, throttle ≈0, mem peak 539 MiB p50 / 4 GiB max); 5-task smoke re-run on the rebased code.
  • pre-commit run --files … clean.

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Sep 2, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Sep 3, 2026
…metrics, TB2.1 prepare

Adds the pieces needed to run Terminal-Bench 2.1 gold verification on a
remote OpenSandbox cluster (validated on arm64 and amd64):

- configs/anyterminal_oracle_opensandbox_arm.yaml: oracle_mode config with
  the sandbox provider referenced by name and image_auth/platform for a
  private arm64 image mirror.
- cpu_pin_enabled: inject nemo_gym.sandbox.utils.cpu_cap_env (derived from
  the sandbox CPU limit) into the agent/oracle/verifier execs so thread
  pools size to the cgroup quota instead of the host's core count.
- override_cpus / override_memory_mb: uniform sandbox sizing replacing each
  task.toml's own values (mirrors harbor_agent).
- ray_task_num_cpus: decouple the local Ray reservation from the task's
  cpus/memory so remote-sandbox runs are not capped by the driver's cores.
- Resource metrics per rollout from cgroup v2 (v1 fallback) counters read
  at phase boundaries: cpu_solve/verify/total_sec, cpu_util_solve/verify,
  cpu_throttled_sec, mem_peak_mib. Fail-open when unavailable.
- prepare.py: `prepare()` entrypoint for `gym eval prepare` that clones the
  pinned terminal-bench-2-1 checkout (same pin as harbor_agent) and writes
  the input JSONL; TERMINAL_BENCH_2_1_TASKS_DIR overrides the checkout.
- .gitignore the cloned task checkout and local results/ output.

Stacked on the oracle-mode commit (#2595).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Adil Asif <adasif@nvidia.com>
@adil-a
adil-a force-pushed the anyterminal-oracle-mode branch from ac76555 to a25c14a Compare September 8, 2026 05:26
@adil-a
adil-a force-pushed the anyterminal-oracle-opensandbox-arm branch from 5996ea0 to d5ec76e Compare September 8, 2026 05:26
@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:agent Agent harnesses and Responses API agent behavior feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Oct 20, 2026
@yaoyu-33
yaoyu-33 marked this pull request as draft November 18, 2026 02:17
@yaoyu-33

Copy link
Copy Markdown
Contributor

We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft.

If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks!

@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants