Repository navigation
Conversation
…metrics, TB2.1 prepare Adds the pieces needed to run Terminal-Bench 2.1 gold verification on a remote OpenSandbox cluster (validated on arm64 and amd64): - configs/anyterminal_oracle_opensandbox_arm.yaml: oracle_mode config with the sandbox provider referenced by name and image_auth/platform for a private arm64 image mirror. - cpu_pin_enabled: inject nemo_gym.sandbox.utils.cpu_cap_env (derived from the sandbox CPU limit) into the agent/oracle/verifier execs so thread pools size to the cgroup quota instead of the host's core count. - override_cpus / override_memory_mb: uniform sandbox sizing replacing each task.toml's own values (mirrors harbor_agent). - ray_task_num_cpus: decouple the local Ray reservation from the task's cpus/memory so remote-sandbox runs are not capped by the driver's cores. - Resource metrics per rollout from cgroup v2 (v1 fallback) counters read at phase boundaries: cpu_solve/verify/total_sec, cpu_util_solve/verify, cpu_throttled_sec, mem_peak_mib. Fail-open when unavailable. - prepare.py: `prepare()` entrypoint for `gym eval prepare` that clones the pinned terminal-bench-2-1 checkout (same pin as harbor_agent) and writes the input JSONL; TERMINAL_BENCH_2_1_TASKS_DIR overrides the checkout. - .gitignore the cloned task checkout and local results/ output. Stacked on the oracle-mode commit (#2595). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>
adil-a
force-pushed
the
anyterminal-oracle-mode
branch
from
September 8, 2026 05:26
ac76555 to
a25c14a
Compare
adil-a
force-pushed
the
anyterminal-oracle-opensandbox-arm
branch
from
September 8, 2026 05:26
5996ea0 to
d5ec76e
Compare
This was referenced Oct 22, 2026
yaoyu-33
marked this pull request as draft
November 18, 2026 02:17
Contributor
|
We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft. If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks! |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2595 (oracle mode); both rebased onto current
main. Diff vs base is the anyterminal changes only.What
Everything needed to run Terminal-Bench 2.1 gold (oracle) verification on a remote OpenSandbox cluster, validated end-to-end on arm64 (86/89 pass) and amd64 (82/89) with all remaining failures attributed to upstream task rot or cluster egress, not architecture or images.
configs/anyterminal_oracle_opensandbox_arm.yaml— oracle config: provider referenced by name,image_auth/platformfor a private arm64 image mirror, uniform 2c/4GiB sandboxes, CPU caps.cpu_pin_enabled— injects main'snemo_gym.sandbox.utils.cpu_cap_env(derived from the sandbox CPU limit) into the agent/oracle/verifier execs so thread pools size to the cgroup quota (replaces the closed feat(sandbox): add per-exec CPU pinning #2601 approach).override_cpus/override_memory_mb— uniform sandbox sizing (mirrors harbor_agent).ray_task_num_cpus— local Ray reservation decoupled from task cpus so remote-sandbox concurrency isn't capped by the driver's core count (mirrorsharbor_ray_task_num_cpus).cpu_solve/verify/total_sec,cpu_util_*,cpu_throttled_sec,mem_peak_mib(fail-open).prepare.pygains aprepare()entrypoint forgym eval prepare(pinned terminal-bench-2-1 clone, same pin as harbor_agent)..gitignorefor the cloned task checkout and localresults/.Testing
pytest responses_api_agents/anyterminal_agent/tests/— 95 passed (6TestBuildProvidercases need adockerbinary; unchanged, deselected locally).cpu_util0.02–0.83 of the 2-core limit, throttle ≈0, mem peak 539 MiB p50 / 4 GiB max); 5-task smoke re-run on the rebased code.pre-commit run --files …clean.🤖 Generated with Claude Code