Repository navigation
feat(nooa): add the SWE-bench Pro session recipe - #3845
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
95305a6 to
b0448e6
Compare
b0448e6 to
b46ff42
Compare
b46ff42 to
f4a2cbb
Compare
f4a2cbb to
ede986f
Compare
391b56c to
d8e40f4
Compare
d8e40f4 to
e1ba061
Compare
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com> # Conflicts: # responses_api_agents/nooa_agent/tests/test_config.py
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
yaoyu-33
left a comment
There was a problem hiding this comment.
Thanks for the follow-up. Four of the five earlier comments are addressed: the prerequisite list now matches the installer, the gym eval run example passes --config and +agent_name like the Hermes page, the page has position: 5 and an H1, and the recipe test now runs the real pairing check and the adapter's routing with negative cases. I ran the nooa suite locally on this head (414 passed, 8 skipped: the Linux-only installer tests).
The sandbox TTL is still the open one: the page now documents the mismatch, but the recipe still ships the 18,000 s TTL against the 21,420 s worker ceiling and 21,600 s episode deadline, and the new text points at the wrong place to fix it. See the two inline comments.
As before, the acceptance evidence is a synthetic task; a single real SWE-bench Pro instance through Docker (or an explicit N/A) would still be worth sharing.
yaoyu-33
left a comment
There was a problem hiding this comment.
Approving. The two inline comments (the wrong TTL pointer in the docs, and the recipe's 18,000 s TTL vs the 21,420 s worker ceiling) are worth fixing before merge but don't block.
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
init-nikhil
left a comment
There was a problem hiding this comment.
CODEX SELF REVIEW
Reviewed head a7a6ba544f2592c1bb2712a765f48db5b8e8b168 against actual base 63e6886bd11d269cf273e76b41d863fd5cf508f5 using a fresh official codex exec review process.
No actionable defects were found. The recipe resolves correctly, and its environment/resource configuration and agent pairing validate successfully. Full tests and real-model sandbox rollouts were not run.
This is a source-review result, not an approving review or a substitute for the separate validation and CI gates.
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40612824022 |
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Propagate existing reviewed parent changes while preserving the authored layer and branch history. Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
|
/ok to test c570b13 |
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
|
/ok to test fa9fd09 |
|
Resolved the merge conflict exposed after #3849 landed in Validation on this commit's exact tree: 423 NOOA tests passed, including the SWE-bench Pro recipe tests; 8 Linux-only tests skipped on macOS. All-files pre-commit passed with no source changes (private guardwords catalog unavailable). No production behavior was changed by the conflict resolution. Required CI has been authorized for this head. |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40822890159 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40823956982 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40824748241 |
Tests and validation
Head
a7a6ba544f2592c1bb2712a765f48db5b8e8b168; base63e6886bd11d269cf273e76b41d863fd5cf508f5. The existing stack tracks this work; no separate issue is needed.34 config/recipe checks and independent parser/preparer/adapter probes passed. Fresh source-bound recipe verification passed both tests in
responses_api_agents/nooa_agent/tests/test_recipe.py, including the Resources TTL override and worker/extraction/cleanup budget relationships.A fresh three-call model rollout exercised the real NOOA worker and legacy adapter, native tools, patch extraction, the separate canonical SWE verifier and collection/cleanup; the synthetic SWE-shaped task earned reward 1. A controlled real Linux SIGKILL case also passed. Earlier setup/timing stops and the live verifier's overly strict intermediate-output assertion remain recorded; the live verdict was corrected separately from retained evidence without repeating inference.
The recipe sets Resources sandbox TTL to 25,200 seconds. Policy/output limits remain gradable; infrastructure and cleanup failures remain ungradable. This used local Linux provisioning and a preselected pinned runtime. It validates integration on a synthetic task, not an official SWE-bench Pro result, cloud networking or runtime installation.
pre-commit run --all-filespassed. Fresh official Codex review returned no actionable findings; the independent bug hunt passed for this head and base.CI snapshot (2026-10-24 03:47 UTC) on this head: failing:
Harness conformance (non-blocking) / kilocode conformance probes;Harness conformance (non-blocking) / Publish check;Harness conformance (non-blocking). Current checks. The all-green readiness gate remains unmet; no final verification-success post or new review request is implied.