Skip to content

feat(nooa): add the SWE-bench Pro session recipe - #3845

Merged
init-nikhil merged 44 commits into
mainfrom
init-nikhil/nooa-swebench-pro
Oct 29, 2026
Merged

init-nikhil merged 44 commits into
mainfrom
init-nikhil/nooa-swebench-pro

Conversation

@init-nikhil

@init-nikhil init-nikhil commented Oct 1, 2026 •

Copy link
Copy Markdown
Member
  • run NOOA on shared swe-bench pro preparation and the existing legacy adapter
  • retain canonical patch grading while validating allowed-agent and row routing
  • document runtime prerequisites, collector configuration and the inherited sandbox lifetime

Tests and validation

Head a7a6ba544f2592c1bb2712a765f48db5b8e8b168; base 63e6886bd11d269cf273e76b41d863fd5cf508f5. The existing stack tracks this work; no separate issue is needed.

34 config/recipe checks and independent parser/preparer/adapter probes passed. Fresh source-bound recipe verification passed both tests in responses_api_agents/nooa_agent/tests/test_recipe.py, including the Resources TTL override and worker/extraction/cleanup budget relationships.

A fresh three-call model rollout exercised the real NOOA worker and legacy adapter, native tools, patch extraction, the separate canonical SWE verifier and collection/cleanup; the synthetic SWE-shaped task earned reward 1. A controlled real Linux SIGKILL case also passed. Earlier setup/timing stops and the live verifier's overly strict intermediate-output assertion remain recorded; the live verdict was corrected separately from retained evidence without repeating inference.

The recipe sets Resources sandbox TTL to 25,200 seconds. Policy/output limits remain gradable; infrastructure and cleanup failures remain ungradable. This used local Linux provisioning and a preselected pinned runtime. It validates integration on a synthetic task, not an official SWE-bench Pro result, cloud networking or runtime installation.

pre-commit run --all-files passed. Fresh official Codex review returned no actionable findings; the independent bug hunt passed for this head and base.

CI snapshot (2026-10-24 03:47 UTC) on this head: failing: Harness conformance (non-blocking) / kilocode conformance probes; Harness conformance (non-blocking) / Publish check; Harness conformance (non-blocking). Current checks. The all-green readiness gate remains unmet; no final verification-success post or new review request is implied.

@init-nikhil init-nikhil added area:agent Agent harnesses and Responses API agent behavior area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters area:docs Generic documentation without another dominant domain labels Oct 1, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 1, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@init-nikhil
init-nikhil added this pull request to stack #3852 October 1, 2026 06:39
@init-nikhil
init-nikhil force-pushed the init-nikhil/nooa-swebench-pro branch from 95305a6 to b0448e6 Compare October 8, 2026 02:23
@init-nikhil
init-nikhil force-pushed the init-nikhil/nooa-swebench-pro branch from b0448e6 to b46ff42 Compare October 8, 2026 21:01
@init-nikhil
init-nikhil force-pushed the init-nikhil/nooa-swebench-pro branch from b46ff42 to f4a2cbb Compare October 9, 2026 21:58
@init-nikhil
init-nikhil removed this pull request from stack #3852 October 9, 2026 21:59
@init-nikhil
init-nikhil added this pull request to stack #4355 October 9, 2026 22:00
@init-nikhil
init-nikhil removed this pull request from stack #4355 October 10, 2026 05:25
@init-nikhil
init-nikhil force-pushed the init-nikhil/nooa-swebench-pro branch from f4a2cbb to ede986f Compare October 10, 2026 05:25
@init-nikhil
init-nikhil added this pull request to stack #4367 October 10, 2026 05:25
@init-nikhil init-nikhil changed the title feat(nooa): [wip] add the native SWE-bench Pro recipe feat(nooa): add the native SWE-bench Pro recipe Oct 10, 2026
@init-nikhil
init-nikhil force-pushed the init-nikhil/nooa-swebench-pro branch 2 times, most recently from 391b56c to d8e40f4 Compare October 13, 2026 08:22
@copy-pr-bot

copy-pr-bot Bot commented Oct 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@init-nikhil
init-nikhil force-pushed the init-nikhil/nooa-swebench-pro branch from d8e40f4 to e1ba061 Compare October 14, 2026 20:28
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>

# Conflicts:
#	responses_api_agents/nooa_agent/tests/test_config.py
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>

@yaoyu-33 yaoyu-33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the follow-up. Four of the five earlier comments are addressed: the prerequisite list now matches the installer, the gym eval run example passes --config and +agent_name like the Hermes page, the page has position: 5 and an H1, and the recipe test now runs the real pairing check and the adapter's routing with negative cases. I ran the nooa suite locally on this head (414 passed, 8 skipped: the Linux-only installer tests).

The sandbox TTL is still the open one: the page now documents the mismatch, but the recipe still ships the 18,000 s TTL against the 21,420 s worker ceiling and 21,600 s episode deadline, and the new text points at the wrong place to fix it. See the two inline comments.

As before, the acceptance evidence is a synthetic task; a single real SWE-bench Pro instance through Docker (or an explicit N/A) would still be worth sharing.

Comment thread benchmarks/swebench/pro/nooa.yaml Resolved
yaoyu-33
yaoyu-33 previously approved these changes Oct 23, 2026

@yaoyu-33 yaoyu-33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. The two inline comments (the wrong TTL pointer in the docs, and the recipe's 18,000 s TTL vs the 21,420 s worker ceiling) are worth fixing before merge but don't block.

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>

@init-nikhil init-nikhil left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CODEX SELF REVIEW

Reviewed head a7a6ba544f2592c1bb2712a765f48db5b8e8b168 against actual base 63e6886bd11d269cf273e76b41d863fd5cf508f5 using a fresh official codex exec review process.

No actionable defects were found. The recipe resolves correctly, and its environment/resource configuration and agent pairing validate successfully. Full tests and real-model sandbox rollouts were not run.

This is a source-review result, not an approving review or a substitute for the separate validation and CI gates.

yaoyu-33
yaoyu-33 previously approved these changes Oct 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40612824022

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Propagate existing reviewed parent changes while preserving the authored layer and branch history.

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
@init-nikhil

Copy link
Copy Markdown
Member Author

/ok to test c570b13

Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
@init-nikhil

Copy link
Copy Markdown
Member Author

/ok to test fa9fd09

@init-nikhil

Copy link
Copy Markdown
Member Author

Resolved the merge conflict exposed after #3849 landed in fa9fd092540561dcfdf41f0cc50337ab38346503. This merges main f46d2f4d and retains the existing NOOA README recipe link. The resulting authored diff against main is byte-identical to the reviewed #3845 delta: four files, 312 insertions.

Validation on this commit's exact tree: 423 NOOA tests passed, including the SWE-bench Pro recipe tests; 8 Linux-only tests skipped on macOS. All-files pre-commit passed with no source changes (private guardwords catalog unavailable). No production behavior was changed by the conflict resolution. Required CI has been authorized for this head.

@furgalep furgalep left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40822890159

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40823956982

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40824748241

This branch was previously deployed

1 inactive deployment
public — fa9fd092 Deployed Oct 28, 2026 by copy-pr-bot[bot] via release / finalize / notify #7752
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior area:docs Generic documentation without another dominant domain area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters complexity:low Localized change in one scope with a small, straightforward review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants