Repository navigation
feat(nooa): add the SWE-bench Pro session recipe #3845
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+312
−0
Merged
Changes from 31 commits
Commits
Show all changes
44 commits
Select commit
Hold shift + click to select a range
31a7d72
fix(nooa): forward invocation correlation from the sandbox
init-nikhil e10a4ac
feat(nooa): persist partial sandbox checkpoints at execution boundaries
init-nikhil fa61ac0
fix(nooa): import the relocated process supervisor
init-nikhil c54b13d
fix(nooa): recover partial evidence after sandbox worker failure
init-nikhil 82090b9
refactor(nooa): use shared sandbox sessions
init-nikhil 7690229
test(nooa): remove unused collector test setup
init-nikhil 83dd1a7
fix(nooa): record capture gaps for unconfirmed cleanup
init-nikhil cdb801a
fix(nooa): recover checkpoints before optional diagnostics
init-nikhil 9d15dea
[wip] feat(nooa): wire sandbox execution into native sessions
init-nikhil 2233a26
fix(nooa): preserve sandbox session close evidence
init-nikhil 135041d
fix(nooa): remove the default SWE benchmark call limit
init-nikhil 09f21ee
chore(nooa): sync PR 3850 with current main
init-nikhil 5f82d13
chore(nooa): sync PR 3851 with its updated parent
init-nikhil a9a1672
chore(nooa): sync PR 3849 with its updated parent
init-nikhil 53196dc
chore(nooa): sync PR 3845 with its updated parent
init-nikhil 3c332f4
fix(nooa): distinguish partial and final worker evidence
init-nikhil 98f59e5
fix(nooa): retain sandbox capture failures and worker diagnostics
init-nikhil 0b4ad6b
chore(nooa): integrate sandbox supervision review fixes
init-nikhil d82a4d0
fix(nooa): keep sandbox session seeding responsive
init-nikhil 18d8d09
chore(nooa): carry entrypoint review fixes into supervision
init-nikhil bf7e086
test(nooa): verify SWE routing and correct setup guidance
init-nikhil ec52176
chore(nooa): carry entrypoint review fixes into sessions
init-nikhil 08399ab
docs(nooa): describe configurable worker supervision budget
init-nikhil 318bd29
chore(nooa): carry reviewed session fixes into SWE recipe
init-nikhil 096e5d1
chore(nooa): carry configurable budget wording into sessions
init-nikhil 9c06350
ch(nooa): carry budget wording into SWE recipe
init-nikhil dcc8bbd
test(nooa): assert close-race HTTP responses in lifecycle checks
init-nikhil d147360
chore(nooa): carry close-race regression updates into PR 3845
init-nikhil 307a969
fix(nooa): retain worker diagnostics after capture failures
init-nikhil c229e72
Merge commit '307a96916f3823cb95fcbbeb25506cf2b0262bab' into HEAD
init-nikhil 0b4b97b
Merge commit 'c229e723be82bffd40c4b44611f2a4eebfb1ccd3' into HEAD
init-nikhil 309e8d5
fix(nooa): drain ordered background checkpoint writes
init-nikhil 45f4649
Merge commit 'cac55a5fca93baf73e202770f71097b2d952f046' into HEAD
init-nikhil ee433d0
Merge commit '45f4649854fedd039fe64d1ad8d9d6d6d632abcf' into HEAD
init-nikhil 3d8c38e
Merge commit 'ee433d0914fc8b94ad81636897819f1b55765899' into HEAD
init-nikhil 843b65a
Merge commit '3d8c38e17' into HEAD
init-nikhil 4d2d3e6
chore(nooa): merge current main into sandbox sessions
init-nikhil c847a73
Merge updated sandbox sessions into SWE-bench Pro recipe
init-nikhil 63e6886
docs(nooa): clarify worker and episode timeout boundaries
init-nikhil 46ab918
Merge clarified NOOA timeout documentation into SWE-bench Pro recipe
init-nikhil a7a6ba5
fix(nooa): extend SWE-bench Pro sandbox TTL beyond run budgets
init-nikhil a174558
test(nooa): use supervisor filename in session fixture
init-nikhil c570b13
chore(stack): merge updated parent into #3845
init-nikhil fa9fd09
chore(stack): merge main into SWE-bench Pro recipe
init-nikhil File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| config_paths: | ||
| - resources_servers/swebench_pro/configs/swebench_pro.yaml | ||
| - responses_api_agents/nooa_agent/configs/nooa_agent.yaml | ||
| - environment_servers/single_agent_turn_legacy/configs/single_agent_turn_legacy.yaml | ||
|
|
||
| # This flat-row recipe binds its own environment instead of the generic NOOA environment. | ||
| nooa_environment_server: | ||
| _delete_key: environment_servers | ||
|
|
||
| swebench_pro_nooa_resources_server: | ||
| _inherit_from: swebench_pro_resources_server | ||
| resources_servers: | ||
| swebench_pro: | ||
| allowed_agents: [nooa_agent] | ||
| datasets: | ||
| - name: swebench_pro | ||
| type: benchmark | ||
| jsonl_fpath: benchmarks/swebench/data/swebench_pro_benchmark.jsonl | ||
| prepare_script: benchmarks/swebench/pro/prepare.py | ||
| num_repeats: 1 | ||
|
|
||
| swebench_pro_nooa_agent: | ||
| _inherit_from: nooa_agent | ||
| responses_api_agents: | ||
| nooa_agent: | ||
| resources_server: | ||
| type: resources_servers | ||
| name: swebench_pro_nooa_resources_server | ||
| nooa: | ||
| execution_mode: sandboxed | ||
| max_policy_calls: null | ||
|
init-nikhil marked this conversation as resolved.
Resolved
|
||
|
|
||
| swebench_pro_nooa: | ||
| _inherit_from: single_agent_turn_legacy | ||
| environment_servers: | ||
| single_agent_turn_legacy: | ||
| resources_server: | ||
| type: resources_servers | ||
| name: swebench_pro_nooa_resources_server | ||
| agent_server: | ||
| type: responses_api_agents | ||
| name: swebench_pro_nooa_agent | ||
97 changes: 97 additions & 0 deletions
97
fern/versions/latest/pages/evaluation-tutorials/nooa-swe-bench-pro.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| --- | ||
|
init-nikhil marked this conversation as resolved.
Resolved
|
||
| title: "NOOA sessions and SWE-bench Pro" | ||
| description: "Run NOOA through single_agent_turn and a Resources-owned sandbox." | ||
| position: 5 | ||
| --- | ||
|
|
||
| # NOOA sessions and SWE-bench Pro | ||
|
|
||
| NOOA uses Gym's agent sessions and existing `single_agent_turn` Environment Server. It seeds Resources, | ||
| opens and invokes NOOA, closes the agent to collect evidence and final Resources cookies, asks Resources to | ||
| verify, and closes Resources. Call the Environment Server's `/run`; NOOA | ||
| no longer implements `/run` or assigns rewards. Sessions support one activation, with identical retries sharing it. | ||
| Use one agent-server worker; independent sessions can still execute concurrently. | ||
|
|
||
| The default harness is `nooa_bench.bench_agent:BenchAgent`, invoked through | ||
| `responses_api_agents.nooa_agent.bench_agent_adapter:invoke_bench_agent`. The standard NOOA requirements | ||
| install `nooa`, `nooa-cli`, and `nooa-bench` at the same revision. Normal `gym env prefetch` prepares | ||
| these dependencies for the agent server; sandbox runtime preparation installs the same pinned packages. | ||
|
|
||
| ## Execution modes | ||
|
|
||
| `nooa.execution_mode: embedded` runs NOOA in the agent server. Its Python execution can access files, processes, | ||
| and network on that host; embedded mode provides no sandbox protection. Custom embedded invocation adapters | ||
| must close their own processes and resources before returning; agent session close cannot track arbitrary | ||
| host-side effects. | ||
|
|
||
| `nooa.execution_mode: sandboxed` runs the complete NOOA agent in the Resources-provided sandbox. Resources must | ||
| return direct `SandboxAccess`; NOOA never provisions or destroys the sandbox. NOOA prepares its runtime under | ||
| `/opt/nemo-gym-nooa`, keeps session files under `/tmp/nemo-gym-nooa`, and executes in the supplied task workdir. | ||
| NOOA closes its worker and session before Resources verifies; the Resources-owned sandbox remains available. | ||
| Verification can inspect persisted artifacts after agent cleanup. Tasks whose verifier requires a | ||
| NOOA-started service to remain live are not supported by this lifecycle. | ||
|
|
||
| Cold runtime setup requires Linux x86_64 or aarch64 with glibc, `bash`, `tar`, `gzip`, `sha256sum`, `flock`, | ||
| `tee`, `mv` with `-T` support, and writable `/opt/nemo-gym-nooa` and `/tmp`. The controller supplies Python; | ||
| the sandbox needs outbound package and build-dependency access. Alpine/musl images are not supported. | ||
| Model and Resources URLs must be reachable from the sandbox. A host-loopback URL is not automatically | ||
| rewritten; configure `use_absolute_ip` / `default_host` for the deployment. No host virtual environment | ||
| or provider credentials are copied into the runner. | ||
|
|
||
| ## Resource tools and observations | ||
|
|
||
| Direct HTTP tool grants arrive through the agent session's `tool_accesses`. Generated `self.*` resource methods, `GymResponsesLLM`, | ||
| and `GymTraceHooks` execute alongside NOOA. Model and resource cookies remain separate; final resource cookies | ||
| return on close for verification. Model/resource infrastructure failures fail the episode instead of becoming a | ||
| reward-zero response. Policy-budget exhaustion remains a gradable partial response. | ||
|
|
||
| Hooks capture Python execution and nested resource calls. Sandbox results are exported on completion or handled | ||
| errors; a hard kill can lose in-memory events. Sandbox-written results are agent-reported evidence, not trusted | ||
| training records. Resources remains responsible for rewards. | ||
|
|
||
| ## Stateful resource example | ||
|
|
||
| Use `resources_servers/example_session_state_mgmt/configs/example_session_state_mgmt_nooa.yaml` for an embedded | ||
| real-model smoke of the instance-method resource bindings. Its materialized task rows are in | ||
| `resources_servers/example_session_state_mgmt/data/example_nooa_tasks.jsonl`; other agents retain the original rows. | ||
| The smoke requires `POLICY_BASE_URL`, `POLICY_API_KEY`, and `POLICY_MODEL_NAME`: | ||
|
|
||
| ```bash | ||
| bash responses_api_agents/nooa_agent/smoke/run_stateful_e2e.sh | ||
| ``` | ||
|
|
||
| The script routes `nooa-counter` tasks to `example_session_state_mgmt_environment_server` and checks the server-side | ||
| counter reward, model usage, resource calls, and trajectory. This example does not exercise sandbox provisioning. | ||
|
|
||
| ## SWE-bench Pro | ||
|
|
||
| The recipe reuses the existing Resources-owned task sandbox, patch extraction, and separate grading sandbox: | ||
|
init-nikhil marked this conversation as resolved.
Resolved
|
||
|
|
||
| ```bash | ||
| python benchmarks/swebench/pro/prepare.py | ||
| gym env start --config benchmarks/swebench/pro/nooa.yaml --config model-provider.yaml | ||
|
|
||
| gym eval run --no-serve \ | ||
| --config benchmarks/swebench/pro/nooa.yaml --config model-provider.yaml \ | ||
| --input benchmarks/swebench/data/swebench_pro_benchmark.jsonl \ | ||
| --output /tmp/nooa-swebench-pro.jsonl --limit 1 --concurrency 1 \ | ||
| +agent_name=swebench_pro_nooa_agent | ||
| ``` | ||
|
|
||
| Use `model-provider.yaml` to supply the policy model and sandbox provider settings and credentials. Pass the | ||
| same recipe and overlay to both commands: `--no-serve` does not inherit routing from the running head server. | ||
| For Docker, configure Gym's published model | ||
| and Resources addresses to be reachable from the container before running. Preparation preserves upstream grading | ||
| metadata in the shared flat dataset. The existing `single_agent_turn_legacy` adapter separates that metadata | ||
| from the Responses request before invoking NOOA. Swap the provider configuration for | ||
| remote infrastructure without changing the agent recipe. | ||
|
|
||
| The overall deadline belongs to the Environment Server (`default_episode_timeout_seconds`). The standard | ||
| task contract has no separate NOOA agent deadline. The inherited SWE-bench Pro sandbox TTL is **18,000 seconds | ||
| (five hours)**, shorter than the default six-hour episode deadline and 21,420-second worker ceiling. The sandbox may expire | ||
| first. For long runs, set both deadlines below the TTL and reserve time for verification and cleanup, or raise | ||
| the TTL in the provider overlay; a model-call limit alone is not a wall-clock guarantee. | ||
|
init-nikhil marked this conversation as resolved.
Resolved
|
||
| NOOA has no model-call limit by default (`max_policy_calls: null`). Set a positive `max_policy_calls` to opt into a | ||
| shared model-call budget for the rollout. The optional `resources_server` agent reference identifies the Resources | ||
| server that owns the dataset and its schema. Runtime tool and sandbox access comes from the standard environment's | ||
| agent session grants. The old `run_timeout_secs` agent configuration field is removed. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.