Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
31a7d72
fix(nooa): forward invocation correlation from the sandbox
init-nikhil Oct 10, 2026
e10a4ac
feat(nooa): persist partial sandbox checkpoints at execution boundaries
init-nikhil Oct 19, 2026
fa61ac0
fix(nooa): import the relocated process supervisor
init-nikhil Oct 10, 2026
c54b13d
fix(nooa): recover partial evidence after sandbox worker failure
init-nikhil Oct 19, 2026
82090b9
refactor(nooa): use shared sandbox sessions
init-nikhil Oct 20, 2026
7690229
test(nooa): remove unused collector test setup
init-nikhil Oct 20, 2026
83dd1a7
fix(nooa): record capture gaps for unconfirmed cleanup
init-nikhil Oct 20, 2026
cdb801a
fix(nooa): recover checkpoints before optional diagnostics
init-nikhil Oct 20, 2026
9d15dea
[wip] feat(nooa): wire sandbox execution into native sessions
init-nikhil Oct 10, 2026
2233a26
fix(nooa): preserve sandbox session close evidence
init-nikhil Oct 20, 2026
135041d
fix(nooa): remove the default SWE benchmark call limit
init-nikhil Oct 10, 2026
09f21ee
chore(nooa): sync PR 3850 with current main
init-nikhil Oct 20, 2026
5f82d13
chore(nooa): sync PR 3851 with its updated parent
init-nikhil Oct 20, 2026
a9a1672
chore(nooa): sync PR 3849 with its updated parent
init-nikhil Oct 20, 2026
53196dc
chore(nooa): sync PR 3845 with its updated parent
init-nikhil Oct 20, 2026
3c332f4
fix(nooa): distinguish partial and final worker evidence
init-nikhil Oct 21, 2026
98f59e5
fix(nooa): retain sandbox capture failures and worker diagnostics
init-nikhil Oct 21, 2026
0b4ad6b
chore(nooa): integrate sandbox supervision review fixes
init-nikhil Oct 21, 2026
d82a4d0
fix(nooa): keep sandbox session seeding responsive
init-nikhil Oct 21, 2026
18d8d09
chore(nooa): carry entrypoint review fixes into supervision
init-nikhil Oct 21, 2026
bf7e086
test(nooa): verify SWE routing and correct setup guidance
init-nikhil Oct 21, 2026
ec52176
chore(nooa): carry entrypoint review fixes into sessions
init-nikhil Oct 21, 2026
08399ab
docs(nooa): describe configurable worker supervision budget
init-nikhil Oct 21, 2026
318bd29
chore(nooa): carry reviewed session fixes into SWE recipe
init-nikhil Oct 21, 2026
096e5d1
chore(nooa): carry configurable budget wording into sessions
init-nikhil Oct 21, 2026
9c06350
ch(nooa): carry budget wording into SWE recipe
init-nikhil Oct 21, 2026
dcc8bbd
test(nooa): assert close-race HTTP responses in lifecycle checks
init-nikhil Oct 21, 2026
d147360
chore(nooa): carry close-race regression updates into PR 3845
init-nikhil Oct 21, 2026
307a969
fix(nooa): retain worker diagnostics after capture failures
init-nikhil Oct 21, 2026
c229e72
Merge commit '307a96916f3823cb95fcbbeb25506cf2b0262bab' into HEAD
init-nikhil Oct 21, 2026
0b4b97b
Merge commit 'c229e723be82bffd40c4b44611f2a4eebfb1ccd3' into HEAD
init-nikhil Oct 21, 2026
309e8d5
fix(nooa): drain ordered background checkpoint writes
init-nikhil Oct 23, 2026
45f4649
Merge commit 'cac55a5fca93baf73e202770f71097b2d952f046' into HEAD
init-nikhil Oct 23, 2026
ee433d0
Merge commit '45f4649854fedd039fe64d1ad8d9d6d6d632abcf' into HEAD
init-nikhil Oct 23, 2026
3d8c38e
Merge commit 'ee433d0914fc8b94ad81636897819f1b55765899' into HEAD
init-nikhil Oct 23, 2026
843b65a
Merge commit '3d8c38e17' into HEAD
init-nikhil Oct 23, 2026
4d2d3e6
chore(nooa): merge current main into sandbox sessions
init-nikhil Oct 23, 2026
c847a73
Merge updated sandbox sessions into SWE-bench Pro recipe
init-nikhil Oct 23, 2026
63e6886
docs(nooa): clarify worker and episode timeout boundaries
init-nikhil Oct 23, 2026
46ab918
Merge clarified NOOA timeout documentation into SWE-bench Pro recipe
init-nikhil Oct 23, 2026
a7a6ba5
fix(nooa): extend SWE-bench Pro sandbox TTL beyond run budgets
init-nikhil Oct 23, 2026
a174558
test(nooa): use supervisor filename in session fixture
init-nikhil Oct 28, 2026
c570b13
chore(stack): merge updated parent into #3845
init-nikhil Oct 28, 2026
fa9fd09
chore(stack): merge main into SWE-bench Pro recipe
init-nikhil Oct 28, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions benchmarks/swebench/pro/nooa.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
config_paths:
- resources_servers/swebench_pro/configs/swebench_pro.yaml
- responses_api_agents/nooa_agent/configs/nooa_agent.yaml
- environment_servers/single_agent_turn_legacy/configs/single_agent_turn_legacy.yaml

# This flat-row recipe binds its own environment instead of the generic NOOA environment.
nooa_environment_server:
_delete_key: environment_servers

swebench_pro_nooa_resources_server:
_inherit_from: swebench_pro_resources_server
resources_servers:
swebench_pro:
allowed_agents: [nooa_agent]
sandbox_config:
# Outlive the episode, patch extraction, and final cleanup budgets.
ttl_s: 25200
datasets:
- name: swebench_pro
type: benchmark
jsonl_fpath: benchmarks/swebench/data/swebench_pro_benchmark.jsonl
prepare_script: benchmarks/swebench/pro/prepare.py
num_repeats: 1

swebench_pro_nooa_agent:
_inherit_from: nooa_agent
responses_api_agents:
nooa_agent:
resources_server:
type: resources_servers
name: swebench_pro_nooa_resources_server
nooa:
execution_mode: sandboxed
max_policy_calls: null
Comment thread
init-nikhil marked this conversation as resolved.
Resolved
Comment thread
init-nikhil marked this conversation as resolved.
Resolved

swebench_pro_nooa:
_inherit_from: single_agent_turn_legacy
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: swebench_pro_nooa_resources_server
agent_server:
type: responses_api_agents
name: swebench_pro_nooa_agent
104 changes: 104 additions & 0 deletions fern/versions/latest/pages/evaluation-tutorials/nooa-swe-bench-pro.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
---
Comment thread
init-nikhil marked this conversation as resolved.
Resolved
title: "NOOA sessions and SWE-bench Pro"
description: "Run NOOA through single_agent_turn and a Resources-owned sandbox."
position: 5
---

# NOOA sessions and SWE-bench Pro

NOOA uses Gym's agent sessions and existing `single_agent_turn` Environment Server. It seeds Resources,
opens and invokes NOOA, closes the agent to collect evidence and final Resources cookies, asks Resources to
verify, and closes Resources. Call the Environment Server's `/run`; NOOA
no longer implements `/run` or assigns rewards. Sessions support one activation, with identical retries sharing it.
Use one agent-server worker; independent sessions can still execute concurrently.

The default harness is `nooa_bench.bench_agent:BenchAgent`, invoked through
`responses_api_agents.nooa_agent.bench_agent_adapter:invoke_bench_agent`. The standard NOOA requirements
install `nooa`, `nooa-cli`, and `nooa-bench` at the same revision. Normal `gym env prefetch` prepares
these dependencies for the agent server; sandbox runtime preparation installs the same pinned packages.

## Execution modes

`nooa.execution_mode: embedded` runs NOOA in the agent server. Its Python execution can access files, processes,
and network on that host; embedded mode provides no sandbox protection. Custom embedded invocation adapters
must close their own processes and resources before returning; agent session close cannot track arbitrary
host-side effects.

`nooa.execution_mode: sandboxed` runs the complete NOOA agent in the Resources-provided sandbox. Resources must
return direct `SandboxAccess`; NOOA never provisions or destroys the sandbox. NOOA prepares its runtime under
`/opt/nemo-gym-nooa`, keeps session files under `/tmp/nemo-gym-nooa`, and executes in the supplied task workdir.
NOOA closes its worker and session before Resources verifies; the Resources-owned sandbox remains available.
Verification can inspect persisted artifacts after agent cleanup. Tasks whose verifier requires a
NOOA-started service to remain live are not supported by this lifecycle.

Cold runtime setup requires Linux x86_64 or aarch64 with glibc, `bash`, `tar`, `gzip`, `sha256sum`, `flock`,
`tee`, `mv` with `-T` support, and writable `/opt/nemo-gym-nooa` and `/tmp`. The controller supplies Python;
the sandbox needs outbound package and build-dependency access. Alpine/musl images are not supported.
Model and Resources URLs must be reachable from the sandbox. A host-loopback URL is not automatically
rewritten; configure `use_absolute_ip` / `default_host` for the deployment. No host virtual environment
or provider credentials are copied into the runner.

## Resource tools and observations

Direct HTTP tool grants arrive through the agent session's `tool_accesses`. Generated `self.*` resource methods, `GymResponsesLLM`,
and `GymTraceHooks` execute alongside NOOA. Model and resource cookies remain separate; final resource cookies
return on close for verification. Model/resource infrastructure failures fail the episode instead of becoming a
reward-zero response. Policy-budget exhaustion remains a gradable partial response.

Hooks capture Python execution and nested resource calls. Sandbox results are exported on completion or handled
errors; a hard kill can lose in-memory events. Sandbox-written results are agent-reported evidence, not trusted
training records. Resources remains responsible for rewards.

## Stateful resource example

Use `resources_servers/example_session_state_mgmt/configs/example_session_state_mgmt_nooa.yaml` for an embedded
real-model smoke of the instance-method resource bindings. Its materialized task rows are in
`resources_servers/example_session_state_mgmt/data/example_nooa_tasks.jsonl`; other agents retain the original rows.
The smoke requires `POLICY_BASE_URL`, `POLICY_API_KEY`, and `POLICY_MODEL_NAME`:

```bash
bash responses_api_agents/nooa_agent/smoke/run_stateful_e2e.sh
```

The script routes `nooa-counter` tasks to `example_session_state_mgmt_environment_server` and checks the server-side
counter reward, model usage, resource calls, and trajectory. This example does not exercise sandbox provisioning.

## SWE-bench Pro

The recipe reuses the existing Resources-owned task sandbox, patch extraction, and separate grading sandbox:
Comment thread
init-nikhil marked this conversation as resolved.
Resolved

```bash
python benchmarks/swebench/pro/prepare.py
gym env start --config benchmarks/swebench/pro/nooa.yaml --config model-provider.yaml

gym eval run --no-serve \
--config benchmarks/swebench/pro/nooa.yaml --config model-provider.yaml \
--input benchmarks/swebench/data/swebench_pro_benchmark.jsonl \
--output /tmp/nooa-swebench-pro.jsonl --limit 1 --concurrency 1 \
+agent_name=swebench_pro_nooa_agent
```

Use `model-provider.yaml` to supply the policy model and sandbox provider settings and credentials. Pass the
same recipe and overlay to both commands: `--no-serve` does not inherit routing from the running head server.
For Docker, configure Gym's published model
and Resources addresses to be reachable from the container before running. Preparation preserves upstream grading
metadata in the shared flat dataset. The existing `single_agent_turn_legacy` adapter separates that metadata
from the Responses request before invoking NOOA. Swap the provider configuration for
remote infrastructure without changing the agent recipe.

The overall deadline belongs to the Environment Server (`default_episode_timeout_seconds`). The standard
task contract has no separate NOOA agent deadline. This recipe sets the task sandbox TTL to **25,200 seconds
(seven hours)** through
`swebench_pro_nooa_resources_server.resources_servers.swebench_pro.sandbox_config.ttl_s`.
Override that Resources key when changing the budget; provider overlays do not set this TTL.
Keep the worker ceiling plus patch extraction and cleanup below `ttl_s`, and also keep the episode deadline
plus final cleanup below it. The defaults are a 21,420-second worker ceiling and a 21,600-second episode timeout.
Patch generation and patch-file removal each use the sandbox command timeout (180 seconds); stopping the task
sandbox has a 120-second budget. Patch download has no separate timeout; the episode deadline bounds the caller's wait.
Final cleanup allows 180 seconds per callback, with at most three registered callbacks including the Resources retry.
The episode budget includes setup, so it can expire before the worker reaches its ceiling. Protected grading uses a
separate sandbox after patch extraction; a model-call limit alone is not a wall-clock guarantee.
NOOA has no model-call limit by default (`max_policy_calls: null`). Set a positive `max_policy_calls` to opt into a
shared model-call budget for the rollout. The optional `resources_server` agent reference identifies the Resources
server that owns the dataset and its schema. Runtime tool and sandbox access comes from the standard environment's
agent session grants. The old `run_timeout_secs` agent configuration field is removed.
2 changes: 2 additions & 0 deletions responses_api_agents/nooa_agent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ fire first. Final cleanup runs after the episode deadline under its own `cleanup
`session_close_timeout_seconds` budgets each cleanup/capture phase and the remaining activation join
(default: 30 seconds per phase). Both settings must be positive and finite.

See [NOOA sessions and SWE-bench Pro](../../fern/versions/latest/pages/evaluation-tutorials/nooa-swe-bench-pro.mdx)
for setup, recipes, service lifecycle and observation limits.

Sandboxed mode requires the Resources server to provide `sandbox_access`; otherwise seeding returns HTTP 422.
Its provider must support `AsyncSandbox.connect`. The Gym model server address must be reachable from inside
Expand Down
159 changes: 159 additions & 0 deletions responses_api_agents/nooa_agent/tests/test_recipe.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

import json
from inspect import signature
from pathlib import Path
from unittest.mock import MagicMock

import pytest

from environment_servers.single_agent_turn.app import SingleAgentTurnEnvironmentServerConfig
from environment_servers.single_agent_turn_legacy.app import SingleAgentTurnLegacyEnvironmentServer
from nemo_gym.server_utils import ServerClient


ROOT = Path(__file__).resolve().parents[3]


def test_swe_shared_preparation_preserves_grading_metadata_through_legacy_adapter(tmp_path: Path) -> None:
from benchmarks.swebench.pro.prepare import prepare, render_prompt

task = {
"instance_id": "repo-1",
"dockerhub_tag": "fixture:1",
"problem_statement": "Fix the task.",
"requirements": "Keep the public interface.",
"interface": "main()",
"patch": "private ground truth",
}
for name in (
"run_scripts/repo-1/run_script.sh",
"run_scripts/repo-1/parser.py",
"dockerfiles/base_dockerfile/repo-1/Dockerfile",
"dockerfiles/instance_dockerfile/repo-1/Dockerfile",
):
asset = tmp_path / "upstream" / name
asset.parent.mkdir(parents=True, exist_ok=True)
asset.write_text("private grading asset")
output = prepare(
dataset=[task],
upstream_root=tmp_path / "upstream",
output_fpath=tmp_path / "shared.jsonl",
image_digest_resolver=lambda tag: "sha256:" + "a" * 64,
)
original = output.read_bytes()
row = json.loads(original)
adapter = SingleAgentTurnLegacyEnvironmentServer(
config=SingleAgentTurnEnvironmentServerConfig(
name="swebench_pro_nooa",
host="localhost",
port=8000,
entrypoint="app.py",
cleanup_timeout_seconds=180,
resources_server={"type": "resources_servers", "name": "swebench_pro_nooa_resources_server"},
agent_server={"type": "responses_api_agents", "name": "swebench_pro_nooa_agent"},
),
server_client=MagicMock(spec=ServerClient),
)
request = adapter._episode_request_from_row(row | {"_ng_task_index": 0, "_ng_rollout_index": 0})
assert request.task.task_id.taskset == "swebench_pro_nooa_resources_server"
assert request.task.task_id.task_id == "repo-1"
assert request.task.task_input.task_data == {k: v for k, v in row.items() if k != "responses_create_params"}
params = request.task.task_input.responses_create_params
assert params.input[0].content == render_prompt(task)
assert "private ground truth" not in params.model_dump_json()
assert "private grading asset" not in params.model_dump_json()
for task_source in (adapter.config.name, adapter.config.agent_server.name, adapter.config.resources_server.name):
routed = adapter._episode_request_from_row(
row
| {
"_ng_task_index": 0,
"_ng_rollout_index": 0,
"task_source": task_source,
"agent_ref": {"name": adapter.config.agent_server.name},
}
)
assert routed.task.task_id.taskset == task_source
assert routed.task.task_input.responses_create_params == params
with pytest.raises(ValueError, match="does not match configured agent server"):
adapter._episode_request_from_row(row | {"agent_ref": {"name": "different_agent"}})
with pytest.raises(ValueError, match="names none of this Environment Server"):
adapter._episode_request_from_row(row | {"task_source": "different_resources"})
assert output.read_bytes() == original


def test_swe_recipe_uses_borrowed_sandbox_and_single_agent_turn_environment() -> None:
from omegaconf import OmegaConf

from environment_servers.single_agent_turn.app import SingleAgentTurnEnvironmentServerConfig
from nemo_gym.global_config import GlobalConfigDictParser
from nemo_gym.rollout_collection import RolloutCollectionHelper
from nemo_gym.sandbox import AsyncSandbox
from resources_servers.swebench_pro.app import SWEBenchProResourcesServerConfig
from responses_api_agents.nooa_agent.config import NOOAAgentConfig

parser = GlobalConfigDictParser()
_, configs = parser.load_extra_config_paths([str(ROOT / "benchmarks/swebench/pro/nooa.yaml")])
config = OmegaConf.merge(*configs)
parser._recursively_swap_keys(config)
environment = SingleAgentTurnEnvironmentServerConfig(
name="swebench_pro_nooa",
host="localhost",
port=8000,
**OmegaConf.to_container(config.swebench_pro_nooa.environment_servers.single_agent_turn_legacy, resolve=True),
)
agent = NOOAAgentConfig(
name=environment.agent_server.name,
host="localhost",
port=8001,
**OmegaConf.to_container(config[environment.agent_server.name].responses_api_agents.nooa_agent, resolve=True),
)
assert agent.nooa.execution_mode == "sandboxed"
assert agent.num_workers == 1
assert agent.model_server.name == "policy_model"
assert agent.max_policy_calls is None
assert environment.resources_server.name == "swebench_pro_nooa_resources_server"
assert not environment.resources_tool_transports
resources = config[environment.resources_server.name].resources_servers.swebench_pro
Comment thread
init-nikhil marked this conversation as resolved.
Resolved
assert resources.allowed_agents == ["nooa_agent"]
assert resources.datasets[0].jsonl_fpath == "benchmarks/swebench/data/swebench_pro_benchmark.jsonl"
assert resources.datasets[0].prepare_script == "benchmarks/swebench/pro/prepare.py"
assert resources.get("task_network_policy") is None
resource_config = SWEBenchProResourcesServerConfig(
name=environment.resources_server.name,
host="localhost",
port=8002,
**OmegaConf.to_container(resources, resolve=True),
)
ttl = resource_config.sandbox_config["ttl_s"]
# Patch extraction and patch-file removal each use AsyncSandbox.exec's default timeout.
command_timeout = signature(AsyncSandbox.exec).parameters["timeout_s"].default
assert isinstance(command_timeout, (int, float))
assert resource_config.verification_stop_timeout is not None
assert environment.default_episode_timeout_seconds is not None
# SingleAgentTurn registers Resources, agent, and post-verification Resources retry callbacks.
final_cleanup_budget = 3 * environment.cleanup_timeout_seconds
assert ttl > (
agent.sandbox_runner_timeout_seconds
+ 2 * command_timeout
+ resource_config.verification_stop_timeout
+ final_cleanup_budget
)
# The episode deadline bounds the caller's wait for setup and patch download, which has no separate timeout.
assert ttl > environment.default_episode_timeout_seconds + final_cleanup_budget
instances = parser.filter_for_server_instance_configs(config)
assert {(instance.name, instance.SERVER_TYPE) for instance in instances} == {
("swebench_pro_nooa_agent", "responses_api_agents"),
("swebench_pro_nooa_resources_server", "resources_servers"),
("swebench_pro_nooa", "environment_servers"),
}
rows = [{"agent_ref": {"name": agent.name}, "task_source": environment.resources_server.name}]
RolloutCollectionHelper._validate_agent_pairings(rows, config)
resources.allowed_agents = ["different_agent"]
with pytest.raises(ValueError, match="does not accept the agent"):
RolloutCollectionHelper._validate_agent_pairings(rows, config)
environments = [instance for instance in instances if instance.SERVER_TYPE == "environment_servers"]
assert [(instance.name, set(instance.environment_servers)) for instance in environments] == [
("swebench_pro_nooa", {"single_agent_turn_legacy"})
]
Loading