Skip to content
Merged
Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
31a7d72
fix(nooa): forward invocation correlation from the sandbox
init-nikhil Oct 10, 2026
e10a4ac
feat(nooa): persist partial sandbox checkpoints at execution boundaries
init-nikhil Oct 19, 2026
fa61ac0
fix(nooa): import the relocated process supervisor
init-nikhil Oct 10, 2026
c54b13d
fix(nooa): recover partial evidence after sandbox worker failure
init-nikhil Oct 19, 2026
82090b9
refactor(nooa): use shared sandbox sessions
init-nikhil Oct 20, 2026
7690229
test(nooa): remove unused collector test setup
init-nikhil Oct 20, 2026
83dd1a7
fix(nooa): record capture gaps for unconfirmed cleanup
init-nikhil Oct 20, 2026
cdb801a
fix(nooa): recover checkpoints before optional diagnostics
init-nikhil Oct 20, 2026
9d15dea
[wip] feat(nooa): wire sandbox execution into native sessions
init-nikhil Oct 10, 2026
2233a26
fix(nooa): preserve sandbox session close evidence
init-nikhil Oct 20, 2026
135041d
fix(nooa): remove the default SWE benchmark call limit
init-nikhil Oct 10, 2026
09f21ee
chore(nooa): sync PR 3850 with current main
init-nikhil Oct 20, 2026
5f82d13
chore(nooa): sync PR 3851 with its updated parent
init-nikhil Oct 20, 2026
a9a1672
chore(nooa): sync PR 3849 with its updated parent
init-nikhil Oct 20, 2026
53196dc
chore(nooa): sync PR 3845 with its updated parent
init-nikhil Oct 20, 2026
3c332f4
fix(nooa): distinguish partial and final worker evidence
init-nikhil Oct 21, 2026
98f59e5
fix(nooa): retain sandbox capture failures and worker diagnostics
init-nikhil Oct 21, 2026
0b4ad6b
chore(nooa): integrate sandbox supervision review fixes
init-nikhil Oct 21, 2026
d82a4d0
fix(nooa): keep sandbox session seeding responsive
init-nikhil Oct 21, 2026
18d8d09
chore(nooa): carry entrypoint review fixes into supervision
init-nikhil Oct 21, 2026
bf7e086
test(nooa): verify SWE routing and correct setup guidance
init-nikhil Oct 21, 2026
ec52176
chore(nooa): carry entrypoint review fixes into sessions
init-nikhil Oct 21, 2026
08399ab
docs(nooa): describe configurable worker supervision budget
init-nikhil Oct 21, 2026
318bd29
chore(nooa): carry reviewed session fixes into SWE recipe
init-nikhil Oct 21, 2026
096e5d1
chore(nooa): carry configurable budget wording into sessions
init-nikhil Oct 21, 2026
9c06350
ch(nooa): carry budget wording into SWE recipe
init-nikhil Oct 21, 2026
dcc8bbd
test(nooa): assert close-race HTTP responses in lifecycle checks
init-nikhil Oct 21, 2026
d147360
chore(nooa): carry close-race regression updates into PR 3845
init-nikhil Oct 21, 2026
307a969
fix(nooa): retain worker diagnostics after capture failures
init-nikhil Oct 21, 2026
c229e72
Merge commit '307a96916f3823cb95fcbbeb25506cf2b0262bab' into HEAD
init-nikhil Oct 21, 2026
0b4b97b
Merge commit 'c229e723be82bffd40c4b44611f2a4eebfb1ccd3' into HEAD
init-nikhil Oct 21, 2026
309e8d5
fix(nooa): drain ordered background checkpoint writes
init-nikhil Oct 23, 2026
45f4649
Merge commit 'cac55a5fca93baf73e202770f71097b2d952f046' into HEAD
init-nikhil Oct 23, 2026
ee433d0
Merge commit '45f4649854fedd039fe64d1ad8d9d6d6d632abcf' into HEAD
init-nikhil Oct 23, 2026
3d8c38e
Merge commit 'ee433d0914fc8b94ad81636897819f1b55765899' into HEAD
init-nikhil Oct 23, 2026
843b65a
Merge commit '3d8c38e17' into HEAD
init-nikhil Oct 23, 2026
4d2d3e6
chore(nooa): merge current main into sandbox sessions
init-nikhil Oct 23, 2026
c847a73
Merge updated sandbox sessions into SWE-bench Pro recipe
init-nikhil Oct 23, 2026
63e6886
docs(nooa): clarify worker and episode timeout boundaries
init-nikhil Oct 23, 2026
46ab918
Merge clarified NOOA timeout documentation into SWE-bench Pro recipe
init-nikhil Oct 23, 2026
a7a6ba5
fix(nooa): extend SWE-bench Pro sandbox TTL beyond run budgets
init-nikhil Oct 23, 2026
a174558
test(nooa): use supervisor filename in session fixture
init-nikhil Oct 28, 2026
c570b13
chore(stack): merge updated parent into #3845
init-nikhil Oct 28, 2026
fa9fd09
chore(stack): merge main into SWE-bench Pro recipe
init-nikhil Oct 28, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions benchmarks/swebench/pro/nooa.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
config_paths:
- resources_servers/swebench_pro/configs/swebench_pro.yaml
- responses_api_agents/nooa_agent/configs/nooa_agent.yaml
- environment_servers/single_agent_turn_legacy/configs/single_agent_turn_legacy.yaml

# This flat-row recipe binds its own environment instead of the generic NOOA environment.
nooa_environment_server:
_delete_key: environment_servers

swebench_pro_nooa_resources_server:
_inherit_from: swebench_pro_resources_server
resources_servers:
swebench_pro:
allowed_agents: [nooa_agent]
datasets:
- name: swebench_pro
type: benchmark
jsonl_fpath: benchmarks/swebench/data/swebench_pro_benchmark.jsonl
prepare_script: benchmarks/swebench/pro/prepare.py
num_repeats: 1

swebench_pro_nooa_agent:
_inherit_from: nooa_agent
responses_api_agents:
nooa_agent:
resources_server:
type: resources_servers
name: swebench_pro_nooa_resources_server
nooa:
execution_mode: sandboxed
max_policy_calls: null
Comment thread
init-nikhil marked this conversation as resolved.
Resolved
Comment thread
init-nikhil marked this conversation as resolved.
Resolved

swebench_pro_nooa:
_inherit_from: single_agent_turn_legacy
environment_servers:
single_agent_turn_legacy:
resources_server:
type: resources_servers
name: swebench_pro_nooa_resources_server
agent_server:
type: responses_api_agents
name: swebench_pro_nooa_agent
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
---
Comment thread
init-nikhil marked this conversation as resolved.
Resolved
title: "NOOA sessions and SWE-bench Pro"
description: "Run NOOA through single_agent_turn and a Resources-owned sandbox."
position: 5
---

# NOOA sessions and SWE-bench Pro

NOOA uses Gym's agent sessions and existing `single_agent_turn` Environment Server. It seeds Resources,
opens and invokes NOOA, closes the agent to collect evidence and final Resources cookies, asks Resources to
verify, and closes Resources. Call the Environment Server's `/run`; NOOA
no longer implements `/run` or assigns rewards. Sessions support one activation, with identical retries sharing it.
Use one agent-server worker; independent sessions can still execute concurrently.

The default harness is `nooa_bench.bench_agent:BenchAgent`, invoked through
`responses_api_agents.nooa_agent.bench_agent_adapter:invoke_bench_agent`. The standard NOOA requirements
install `nooa`, `nooa-cli`, and `nooa-bench` at the same revision. Normal `gym env prefetch` prepares
these dependencies for the agent server; sandbox runtime preparation installs the same pinned packages.

## Execution modes

`nooa.execution_mode: embedded` runs NOOA in the agent server. Its Python execution can access files, processes,
and network on that host; embedded mode provides no sandbox protection. Custom embedded invocation adapters
must close their own processes and resources before returning; agent session close cannot track arbitrary
host-side effects.

`nooa.execution_mode: sandboxed` runs the complete NOOA agent in the Resources-provided sandbox. Resources must
return direct `SandboxAccess`; NOOA never provisions or destroys the sandbox. NOOA prepares its runtime under
`/opt/nemo-gym-nooa`, keeps session files under `/tmp/nemo-gym-nooa`, and executes in the supplied task workdir.
NOOA closes its worker and session before Resources verifies; the Resources-owned sandbox remains available.
Verification can inspect persisted artifacts after agent cleanup. Tasks whose verifier requires a
NOOA-started service to remain live are not supported by this lifecycle.

Cold runtime setup requires Linux x86_64 or aarch64 with glibc, `bash`, `tar`, `gzip`, `sha256sum`, `flock`,
`tee`, `mv` with `-T` support, and writable `/opt/nemo-gym-nooa` and `/tmp`. The controller supplies Python;
the sandbox needs outbound package and build-dependency access. Alpine/musl images are not supported.
Model and Resources URLs must be reachable from the sandbox. A host-loopback URL is not automatically
rewritten; configure `use_absolute_ip` / `default_host` for the deployment. No host virtual environment
or provider credentials are copied into the runner.

## Resource tools and observations

Direct HTTP tool grants arrive through the agent session's `tool_accesses`. Generated `self.*` resource methods, `GymResponsesLLM`,
and `GymTraceHooks` execute alongside NOOA. Model and resource cookies remain separate; final resource cookies
return on close for verification. Model/resource infrastructure failures fail the episode instead of becoming a
reward-zero response. Policy-budget exhaustion remains a gradable partial response.

Hooks capture Python execution and nested resource calls. Sandbox results are exported on completion or handled
errors; a hard kill can lose in-memory events. Sandbox-written results are agent-reported evidence, not trusted
training records. Resources remains responsible for rewards.

## Stateful resource example

Use `resources_servers/example_session_state_mgmt/configs/example_session_state_mgmt_nooa.yaml` for an embedded
real-model smoke of the instance-method resource bindings. Its materialized task rows are in
`resources_servers/example_session_state_mgmt/data/example_nooa_tasks.jsonl`; other agents retain the original rows.
The smoke requires `POLICY_BASE_URL`, `POLICY_API_KEY`, and `POLICY_MODEL_NAME`:

```bash
bash responses_api_agents/nooa_agent/smoke/run_stateful_e2e.sh
```

The script routes `nooa-counter` tasks to `example_session_state_mgmt_environment_server` and checks the server-side
counter reward, model usage, resource calls, and trajectory. This example does not exercise sandbox provisioning.

## SWE-bench Pro

The recipe reuses the existing Resources-owned task sandbox, patch extraction, and separate grading sandbox:
Comment thread
init-nikhil marked this conversation as resolved.
Resolved

```bash
python benchmarks/swebench/pro/prepare.py
gym env start --config benchmarks/swebench/pro/nooa.yaml --config model-provider.yaml

gym eval run --no-serve \
--config benchmarks/swebench/pro/nooa.yaml --config model-provider.yaml \
--input benchmarks/swebench/data/swebench_pro_benchmark.jsonl \
--output /tmp/nooa-swebench-pro.jsonl --limit 1 --concurrency 1 \
+agent_name=swebench_pro_nooa_agent
```

Use `model-provider.yaml` to supply the policy model and sandbox provider settings and credentials. Pass the
same recipe and overlay to both commands: `--no-serve` does not inherit routing from the running head server.
For Docker, configure Gym's published model
and Resources addresses to be reachable from the container before running. Preparation preserves upstream grading
metadata in the shared flat dataset. The existing `single_agent_turn_legacy` adapter separates that metadata
from the Responses request before invoking NOOA. Swap the provider configuration for
remote infrastructure without changing the agent recipe.

The overall deadline belongs to the Environment Server (`default_episode_timeout_seconds`). The standard
task contract has no separate NOOA agent deadline. The inherited SWE-bench Pro sandbox TTL is **18,000 seconds
(five hours)**, shorter than the default six-hour episode deadline and 21,420-second worker ceiling. The sandbox may expire
first. For long runs, set both deadlines below the TTL and reserve time for verification and cleanup, or raise
the TTL in the provider overlay; a model-call limit alone is not a wall-clock guarantee.
Comment thread
init-nikhil marked this conversation as resolved.
Resolved
NOOA has no model-call limit by default (`max_policy_calls: null`). Set a positive `max_policy_calls` to opt into a
shared model-call budget for the rollout. The optional `resources_server` agent reference identifies the Resources
server that owns the dataset and its schema. Runtime tool and sandbox access comes from the standard environment's
agent session grants. The old `run_timeout_secs` agent configuration field is removed.
38 changes: 36 additions & 2 deletions responses_api_agents/nooa_agent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,39 @@ Embedded CodeAct persists for the duration of one invocation. `execute_python` i
read and modify files, start processes, use the network, and change other node state. Embedded mode provides no
sandbox protection. Only injected resource methods are Gym/verifier-observable and authoritative; direct CodeAct
side effects belong to NOOA and are not reliably visible to verification or replay. The trajectory distinguishes
the outer `execute_python` call from nested resource calls. A sandboxed or remote executor is a future integration;
sandboxed execution is not implemented here.
the outer `execute_python` call from nested resource calls.

## Sandboxed execution

`nooa.execution_mode: sandboxed` runs the complete NOOA agent in the task sandbox supplied by Resources.
NOOA stages its own runtime and closes its worker before disconnecting; Resources owns sandbox creation and teardown.
Agent and resource sessions use Gym's existing `single_agent_turn` environment. Agent close returns the evidence and final
Resources cookies before verification; Resources closes its sandbox afterward.

`sandbox_runner_timeout_seconds` sets a finite worker ceiling (default: 21420 seconds). Request metadata
`agent_timeout_sec` can shorten that ceiling. This is separate from the environment's episode timeout;
a longer episode timeout does not extend the worker ceiling. The default leaves 180 seconds before the
default 21600-second episode deadline for finalization. Custom worker and episode budgets should preserve that headroom.
`session_close_timeout_seconds` budgets each cleanup/capture phase and the remaining activation join
(default: 30 seconds per phase). Both settings must be positive and finite.

See [NOOA sessions and SWE-bench Pro](../../fern/versions/latest/pages/evaluation-tutorials/nooa-swe-bench-pro.mdx)
for setup, recipes, service lifecycle and observation limits.

Sandboxed mode requires the Resources server to provide `sandbox_access`; otherwise seeding returns HTTP 422.
Its provider must support `AsyncSandbox.connect`. The Gym model server address must be reachable from inside
the task sandbox: configure `use_absolute_ip` / `default_host` for the deployment instead of a controller-only
loopback address. Runtime packaging runs in a thread so seeding does not block other agent requests.

Close raises if the worker stop is unconfirmed or the remaining activation cannot be joined within its budget;
the session stays available for a close retry. The bounded join also applies to embedded mode. An activation
interrupted by session close returns HTTP 409 to a caller whose request was not itself cancelled.


## Sandbox worker supervision

`sandbox_runner_timeout_seconds` bounds the worker execution budget; task metadata can shorten it. Exhausting that
budget is an ungradable, non-retryable execution failure, rather than a retryable model or provider outage.
Partial checkpoints remain available for failure inspection; they never authorize grading. Transient artifact
transfer failures receive one retry within the existing capture deadline. Worker startup and supervisor failures
retain up to 2,000 characters of output-log diagnostics after artifact capture.
112 changes: 103 additions & 9 deletions responses_api_agents/nooa_agent/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,16 @@
from nemo_gym.base_responses_api_agent import (
AgentCloseSessionResponse,
AgentSeedSessionRequest,
AgentSessionSetupError,
AgentSessionState,
SimpleResponsesAPIAgent,
)
from nemo_gym.openai_utils import NeMoGymResponse, NeMoGymResponseCreateParamsNonStreaming
from nemo_gym.rollout_correlation import rollout_context
from nemo_gym.rollout_observability import AgentInvocation
from nemo_gym.sandbox import AsyncSandbox
from nemo_gym.sandbox.config import resolve_provider_config
from nemo_gym.sandbox.providers import create_provider
from nemo_gym.tool_access import DirectHTTPToolAccess
from responses_api_agents.nooa_agent.config import NOOAAgentConfig
from responses_api_agents.nooa_agent.result import finalize_run_result, is_transient_infrastructure_error
Expand All @@ -38,6 +43,8 @@
NOOARunRequest,
NOOARunResult,
)
from responses_api_agents.nooa_agent.sandbox_runner import SandboxNOOARunner
from responses_api_agents.nooa_agent.sandbox_runtime import prepare_nooa_runtime
from responses_api_agents.nooa_agent.termination import NOOATerminationReason


Expand Down Expand Up @@ -80,12 +87,45 @@ async def _seed_agent_session_state(self, body: AgentSeedSessionRequest) -> NOOA
tool_access=access,
resources_cookies=dict(access.cookies) if access else {},
)
state.runner = InProcessNOOARunner(
invocation=self.config.nooa,
server_client=self.server_client,
model_server_name=self.config.model_server.name,
max_policy_calls=self.config.max_policy_calls,
)
if self.config.nooa.execution_mode == "embedded":
state.runner = InProcessNOOARunner(
invocation=self.config.nooa,
server_client=self.server_client,
model_server_name=self.config.model_server.name,
max_policy_calls=self.config.max_policy_calls,
)
else:
if body.sandbox_access is None:
raise HTTPException(422, "Sandboxed NOOA requires Resources-provided sandbox_access")
model_base_url = self.resolve_model_base_url(self.config.model_server.name).removesuffix("/v1")
connection = body.sandbox_access.connection
provider = create_provider(
resolve_provider_config(connection.provider_config_ref, self.server_client.global_config_dict)
)
try:
sandbox = await AsyncSandbox.connect(connection.descriptor, provider=provider)
except BaseException:
await provider.aclose()
raise
runner = SandboxNOOARunner(
sandbox=sandbox,
workdir=body.sandbox_access.workdir,
python="",
invocation=self.config.nooa,
model_base_url=model_base_url,
model_server_name=self.config.model_server.name,
timeout=self.config.sandbox_runner_timeout_seconds,
close_timeout=self.config.session_close_timeout_seconds,
max_policy_calls=self.config.max_policy_calls,
)
state.runner = runner
try:
runner.python = await prepare_nooa_runtime(sandbox)
await runner.prepare()
except BaseException as error:
# Main retains this state for close while rejecting activation
# and seed retries against an incompletely initialized runtime.
raise AgentSessionSetupError(state, error=error) from error
return state

async def responses(
Expand All @@ -111,7 +151,13 @@ async def responses(
state.activation_key = key
state.execution = asyncio.create_task(self._execute_session(state, body, route))
execution = state.execution
return await asyncio.shield(execution)
try:
return await asyncio.shield(execution)
except asyncio.CancelledError:
request_task = asyncio.current_task()
if state.closing and request_task is not None and not request_task.cancelling():
raise HTTPException(409, "Agent session is closing") from None
raise

async def _execute_session(
self, state: NOOASessionState, body: NeMoGymResponseCreateParamsNonStreaming, route: str
Expand Down Expand Up @@ -150,16 +196,64 @@ async def _close_agent_session_state(self, state: AgentSessionState) -> AgentClo
async with state.lock:
state.closing = True
execution = state.execution
if isinstance(state.runner, SandboxNOOARunner):
# Stop the worker and capture its final evidence before cancelling
# the controller task that may still be waiting for provider I/O.
# An unconfirmed stop raises here and leaves this state retryable.
await state.runner.close()
if execution is not None:
if not execution.done():
execution.cancel()
done, _ = await asyncio.wait({execution}, timeout=self.config.session_close_timeout_seconds)
if not done:
raise TimeoutError(
"NOOA activation did not stop within the session close timeout; close must retry"
)
await asyncio.gather(execution, return_exceptions=True)
if isinstance(state.runner, SandboxNOOARunner):
recovered = state.runner.artifact or state.runner.checkpoint
if recovered is not None:
state.resources_cookies.update(recovered.resource_cookies)
state.model_cookies.update(recovered.model_cookies)
if state.result is None and recovered.response is not None:
state.result = recovered.run_result()
# Late evidence cannot turn a failed activation into a
# successful or gradable result.
if execution is not None and execution.cancelled():
state.result.termination_reason = NOOATerminationReason.CANCELLED
elif execution is not None and (error := execution.exception()) is not None:
state.result.termination_reason = NOOATerminationReason.INFRASTRUCTURE_ERROR
state.result.termination_error = str(error)
if state.result is not None:
state.result.resource_cookies.update(recovered.resource_cookies)
state.result.model_cookies.update(recovered.model_cookies)
return self._session_evidence(state)

def _session_evidence(self, state: NOOASessionState) -> AgentCloseSessionResponse:
observations = None
observations = state.runner.observations if isinstance(state.runner, SandboxNOOARunner) else None
if state.result is not None:
_, observations = finalize_run_result(state.result)
_, finalized = finalize_run_result(state.result)
if observations is None:
observations = finalized
else:
# The runner may have captured newer records and cleanup gaps
# after the activation produced its result. Keep both sets of
# gaps, including the normalized activation failure.
observations = observations.model_copy(
update={
"records": list(
{
("invocation", record.invocation_id)
if isinstance(record, AgentInvocation)
else ("record", record.model_dump_json()): record
for record in [*finalized.records, *observations.records]
}.values()
),
"gaps": list(
{gap.model_dump_json(): gap for gap in [*observations.gaps, *finalized.gaps]}.values()
),
}
)
return AgentCloseSessionResponse(
agent_session_id=state.request.agent_session_id,
agent_observations=observations,
Expand Down
Loading
Loading