Repository navigation
Conversation
Signed-off-by: Christian Munley <cmunley@nvidia.com>
# Conflicts: # README.md
…b + verifier + docker provider; drop openclaw) Self-contained anyswe-on-swe_env on top of cmunley1/anyswe (now merged with latest main): - add swe_env library (responses_api_agents/swe_env/) + inline verifier (resources_servers/swe_env/) - add the docker sandbox provider + register it (main has apptainer/opensandbox, not docker) - replace anyswe_agent with the swe_env-based version: docker + apptainer backends, local-.sif support, --writable-tmpfs, claude_code anthropic_base_url=null (real Anthropic by default) - drop openclaw + orphaned dataset setup scripts + unused example_rollouts (out of scope) Supersedes the NVIDIA-NeMo#1677 stack (NVIDIA-NeMo#1677/NVIDIA-NeMo#1678/NVIDIA-NeMo#1679) and the prior NVIDIA-NeMo#1738 base (stack/3). Unit tests: anyswe 16, swe_env lib 134 (+1 skip), verifier 14 (+2 skip) = 164 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d by this stack) responses_api_agents/openclaw_agent/ exists on main; it entered the NVIDIA-NeMo#1572 base via the main-merge (22f8ead), not from cmunley's original anyswe work (pr-1572 had zero openclaw files). The earlier "drop openclaw" trim therefore deleted a main artifact, so merging the stack down (NVIDIA-NeMo#1738 -> NVIDIA-NeMo#1572 -> main) would have removed openclaw_agent/ from main. Restore the openclaw files from the base so this PR neither adds nor removes openclaw: - Restore responses_api_agents/openclaw_agent/ (11 files). - Restore responses_api_agents/anyswe_agent/configs/anyswe_openclaw.yaml and setup_scripts/openclaw_agent_deps.sh. Still trimmed (these are NVIDIA-NeMo#1572-only, absent from main): the orphaned swebench.sh / r2e_gym.sh / swebench_multilingual.sh setup scripts and the unused example_rollouts_{claude,hermes}.jsonl fixtures. openclaw + anyswe suites green (45 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>
…erified (#1249) (#1738) ## What this is Converges the **anyswe** runner (#1572) onto the **#1677 swe_env decoupling**: a thin agent drops any Gym agent into the SWE task container, extracts `git diff HEAD`, and grades **in-process** via `verify_task` — no separate resources server. Based on the #1677 stack, so this diff is just the convergence + fixes (~55 files). ## Before / after ```text BEFORE (main) task --> swe_agents (OpenHands-coupled; sandbox + grading inline; apptainer-only) AFTER (this PR) [NEW] = added in this PR task --> anyswe_agent --> swe_env/ [NEW shared library] (any Gym agent +-- harnesses/ (extracted from swe_agents) in the container) +-- parsing/ (verbatim from main) +-- verify_task (inline grade; no /verify server) | v nemo_gym/sandbox providers: docker [NEW] | apptainer ``` ## What changed | Path | Change | |---|---| | `responses_api_agents/swe_env/` | **New** shared library — `harnesses/`, `parsing/`, sandbox lifecycle, `verify_task`, `self_drive` | | `responses_api_agents/anyswe_agent/` | Thin runner (#1572) rewired onto `swe_env`; grades `git diff` in-process | | `nemo_gym/sandbox/providers/docker/` | **New** docker provider (was apptainer-only) — runs Verified on docker | | `…/swe_env/parsing/` | Byte-identical relocation of main's `swe_agents/swe_bench_ext/` (lighthouse parser) | | `…/swe_env/verify_task.py` | Moved from `resources_servers/swe_env/`; inline grader, importers repointed | | `resources_servers/swe_env/` | **Removed** — no consumer; grading is inline, not a `/verify` server | | `…/anyswe_agent/setup_scripts/{swebench,r2e_gym,swebench_multilingual}.sh` | **Removed** — superseded by `harnesses/` + pip `swebench` | | `responses_api_agents/openclaw_agent/` | Untouched (a `main` artifact — neither added nor removed) | Other fixes: docker flat-eval uses swebench's official per-repo parser (gold resolve 4%→95%); per-repo FAIL_ONLY eval-type + main's patch-apply ladder ported into the harnesses; apptainer `--pid` removed (≥1.3.6 rejects it); `get_harness` self-registers + HF-name tolerant. ## Suggested review order 1. `nemo_gym/sandbox/providers/registry.py` + `providers/docker/` — the new provider lazy-registration (small, foundational). 2. `responses_api_agents/swe_env/harness.py` — the contract: `SweTask`, the harness ABC, the registry, and the resolution rule (`compute_resolved`). **Read first** — everything else builds on it. 3. `responses_api_agents/swe_env/verify_task.py` — the inline grading orchestrator (the "no `/verify` server" core). 4. `responses_api_agents/swe_env/sandbox.py` + `self_drive.py` — sandbox lifecycle, then provision / egress / `git diff` extraction. 5. `responses_api_agents/swe_env/harnesses/swebench.py` — the primary harness + the parity fixes (FAIL_ONLY, patch-apply ladder, masking); then `swe_bench_ext.py` / `r2egym.py` / `flat_eval.py` as needed. 6. `responses_api_agents/anyswe_agent/app.py` — how it wires together: routing → `_build_swetask` → provider → `self_drive` + `verify_task`. 7. `responses_api_agents/swe_env/parsing/` — **skim only**: byte-identical relocation from main, no new logic. 8. `…/swe_env/tests/` + `…/anyswe_agent/tests/` — coverage for the above. ## Validation — SWE-bench Verified (N=100/cell), in-line grader Each agent × model run on the **same 100 instances** on **both** sandbox backends; cells are resolve rate (`mean/reward`), all clean (mask ≤ 0.01): | agent × model | docker | apptainer | |---|---|---| | hermes × gpt-5.5 | 69% | 66% | | hermes × opus-4.8 | 77% | 79% | | claude_code × opus-4.8 | 78% | 82% | | openclaw × gpt-5.5 | 70% | 71% | | openclaw × opus-4.8 | 77% | 78% | docker and apptainer agree within ±4 pts (mean Δ ≈ +1, no systematic gap) — the two backends are equivalent. (claude_code × gpt-5.5 skipped: the Claude CLI speaks Anthropic `/v1/messages` only.) Gold-patch grader sanity beforehand: docker **95/100**, `.sif` **10/10**. - A full per-failure audit found **0 our-side (infra) failures** — non-resolved rows are model wrong/no-patch. (Found + fixed an apptainer ENOSPC en route: `--writable-tmpfs`'s 64 MiB cap → per-instance disk `--overlay`.) - ruff clean, **158 unit tests pass**; harness-vs-main audit converged (0 residual behavioral drift). ## Reproduce — gpt-5.5 on hermes (docker **or** apptainer) Shared setup + dataset (gpt-5.5 is a reasoning model, so temperature/top-p must be 1.0): ```bash uv venv && uv sync --extra dev export RAY_TMPDIR=/tmp # avoids Ray AF_UNIX path-length limits export POLICY_API_KEY=sk-... # gateway key — SECRET, pass via ++ only, never commit python responses_api_agents/anyswe_agent/prepare.py --no-images --limit 100 --temperature 1.0 --top-p 1.0 DS=responses_api_agents/anyswe_agent/data/swebench_verified.jsonl MODEL=us/azure/openai/eccn-gpt-5.5 A=anyswe_hermes.responses_api_agents.anyswe_agent ``` **Docker** (default provider; each instance image pulls on demand from DockerHub — no pre-build): ```bash # Terminal A — servers (wait for "All 2 / 2 servers ready!") ng_run "+config_paths=[responses_api_agents/anyswe_agent/configs/anyswe_hermes.yaml,responses_api_models/openai_model/configs/openai_model.yaml]" \ ++policy_base_url='https://inference-api.nvidia.com/v1' ++policy_model_name="$MODEL" ++policy_api_key="$POLICY_API_KEY" # Terminal B — rollouts (num_samples_in_parallel == concurrent containers) ng_collect_rollouts +agent_name=anyswe_hermes +input_jsonl_fpath=$DS \ +output_jsonl_fpath=results/hermes_gpt_docker.jsonl +num_samples_in_parallel=16 ``` **Apptainer** (build a `.sif` per instance once, then run with no registry pulls): ```bash # Build SIFs (needs apptainer on PATH; ~1-3 GB each) -> data/sifs/{instance_id}.sif python responses_api_agents/anyswe_agent/prepare.py --no-dataset --sif-dir data/sifs # Terminal A — same servers, but point the formatter at the .sif and select the apptainer provider. # Quote container_formatter so Hydra treats {instance_id} as a literal; sandbox_provider is a dict. ng_run "+config_paths=[responses_api_agents/anyswe_agent/configs/anyswe_hermes.yaml,responses_api_models/openai_model/configs/openai_model.yaml]" \ ++policy_base_url='https://inference-api.nvidia.com/v1' ++policy_model_name="$MODEL" ++policy_api_key="$POLICY_API_KEY" \ ++$A.container_formatter='"data/sifs/{instance_id}.sif"' \ ++$A.sandbox_provider='{apptainer:{}}' # Terminal B — same collect command ng_collect_rollouts +agent_name=anyswe_hermes +input_jsonl_fpath=$DS \ +output_jsonl_fpath=results/hermes_gpt_appt.jsonl +num_samples_in_parallel=16 ``` Resolve rate = `mean/reward` in `results/<name>_aggregate_metrics.json`; a clean run has `mean/mask_sample == 0`. (For other cells: `anyswe_claude_code.yaml` + an opus model id speaks Anthropic `/v1/messages` via `++$A.agent_kwargs.anthropic_base_url=…`; `anyswe_openclaw.yaml` runs OpenClaw. Same `container_formatter` / `sandbox_provider` overrides switch either between docker and apptainer.) ## Deferred to a follow-up openclaw eval (runner restored, eval deferred); R2E-Gym / Multilingual **native nested graders** (host-side flat grading approximates those — Verified/English is exact); routing needles for `swe-bench-ext` / `swe-rebench` / `nv-internal`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Codex <codex@openai.com> Co-authored-by: Codex <codex@openai.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the **server-suite CI failure** on #1572 (all 8 shards failing). ## Root cause `ng_test_all` runs with `fail_on_total_and_test_mismatch=true` in CI. It asserts every candidate module dir under `resources_servers/*`, `responses_api_agents/*`, `responses_api_models/*` has a `README.md` (and is therefore in the tested set). The new `responses_api_agents/swe_env/` library had no README, so it was counted as a candidate but not a tested module: ``` AssertionError: Mismatch on the number of total modules found (129) and the number of actual modules tested (128)! Extra candidate paths: - responses_api_agents/swe_env ``` This assertion runs in every shard, so all 8 shards fail. ## Fix Add `responses_api_agents/swe_env/README.md`. This makes the counts match **and** runs swe_env's unit tests in CI (FakeProvider-backed — no Docker/apptainer; `swebench` is imported lazily so the `nemo-gym[dev]` test venv suffices). Data validation is skipped for swe_env since it's not under `resources_servers/`. Only `swe_env` was missing a README; no other module is affected. DCO on the squash-merge commit is a separate, known item. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Codex <codex@openai.com> Co-authored-by: Codex <codex@openai.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ffrujeri
left a comment
There was a problem hiding this comment.
This adds a provider-neutral SWE environment. anyswe_agent runs any thin Gym agent (hermes / claude_code / openclaw) inside a SWE task container, extracts its git diff HEAD, and grades the patch inline via the new swe_env library — there is no separate Resources Server. It also adds a Docker sandbox provider to the shared core nemo_gym.sandbox layer, making end-to-end SWE-bench verification runnable on a single workstation.
A couple of non-blocking notes that don't map to a single line:
- The branch is currently behind
main— a rebase before merge would be good. - Several parsing behaviors flagged inline (
parsing.pyframework coverage, fuzzy matching, XFAIL handling) are ported verbatim from the existingswe_agents/swe_bench_extcode, so they're not regressions introduced here — they're worth a follow-up against the benchmark itself.
| # API "incomplete" status. A patch that happens to pass in that case is an accidental | ||
| # reward, so mask it (mirrors main's swe_agents, which masks resolved AND | ||
| # max_iteration/context_window) to keep it out of the training gradient. | ||
| agent_truncated = bool(saved is not None and saved.status == "incomplete") |
There was a problem hiding this comment.
responses_api_agents/anyswe_agent/app.py:621
The truncation-masking path looks like it never fires. agent_truncated is derived from the inner agent's response status == "incomplete", but none of the three shipped inner agents (hermes_agent, claude_code_agent, openclaw_agent) ever set that status — they all build NeMoGymResponse with status="completed". So agent_truncated is always False and the (resolved and agent_truncated) mask on the next line never triggers. The net effect is that an agent which exhausts its turns/context but happens to land a passing patch is scored reward=1 and left unmasked — the exact accidental-reward case the comment says this guards against. The swe_agents it replaces classified the harness error string instead. Could the truncation signal be derived from the inner agent's actual turn/error state (e.g. via self_drive) rather than a status field nothing sets?
| name: aws/anthropic/bedrock-claude-opus-4-7 | ||
| api: openai-completions | ||
|
|
||
| container_formatter: data/sifs/{instance_id}.sif |
There was a problem hiding this comment.
responses_api_agents/anyswe_agent/configs/anyswe_openclaw.yaml:44
This example config may not be runnable as shipped: container_formatter points at an Apptainer .sif, but no sandbox_provider is set, so it falls back to the AnySweAgentConfig default of docker. _instance_image then returns the .sif path verbatim, yielding docker run data/sifs/<id>.sif ..., which Docker can't run. Setting sandbox_provider: {apptainer: {...}} would match the intent.
Also, apptainer_memory_limit_mb (L47) and skip_eval (L49) aren't fields on AnySweAgentConfig — they're silently swallowed by extra="allow", so they have no effect. Probably leftovers from the swe_agents schema; worth removing to avoid implying they're honored.
| args += ["-e", f"{key}={value}"] | ||
| args += [handle.sandbox_id, "bash", "-c", command] | ||
| try: | ||
| rc, out, err = await self._run(*args, timeout_s=timeout_s) |
There was a problem hiding this comment.
nemo_gym/sandbox/providers/docker/provider.py:252
A few timeout gaps in the Docker provider, grouped:
exechas no default timeout —timeout_s=Nonewaits forever. The git-diff extraction inself_drive.py:194calls exec with no timeout, so a hung in-container command would block the rollout indefinitely. The Apptainer provider applies a configurable default exec timeout.close()(rm -f, L320) andupload_file/download_file(docker cp, L278/L295) call_runwith no timeout, so a wedged daemon hangs teardown.- On exec timeout,
proc.kill()(L113) kills only the localdocker execclient, not the in-container process (bounded, sinceclose()laterrm -f's the container, but Apptainer useskillpgfor parity).
Consider a configurable default exec timeout plus bounded timeouts on close/cp.
| @@ -0,0 +1,32 @@ | |||
| # swe_env | |||
There was a problem hiding this comment.
responses_api_agents/swe_env/README.md:1
Would it be possible to add reward-profiling evidence for the benchmark? Right now the resolved=True assertions are all unit tests over scripted fake-provider output, so there's no durable in-repo proof the verifier reproduces published resolve rates on real rows. A small baseline — e.g. gold patch → resolved=True and empty patch → resolved=False rates for at least SWE-bench Verified, committed as an aggregate_metrics.json or a short note — would let reviewers trust the numbers, per the add-benchmark / reward-profiling guidance. (Not a code blocker; the grading itself looks correct. It would be the natural gate before flipping any verified: true.)
| dataset_dir = persistent_dir / "instance_datasets" | ||
| dataset_dir.mkdir(parents=True, exist_ok=True) | ||
| instance_dataset_path = dataset_dir / f"{agent_run_id}.jsonl" | ||
| instance_dict = json.loads(problem_info["instance_dict"]) |
There was a problem hiding this comment.
responses_api_agents/anyswe_agent/app.py:450
_setup_params does json.loads(problem_info["instance_dict"]), which assumes a string, while _build_swetask (L134) accepts either a str or a dict. If a row ever carries instance_dict as a dict, this raises TypeError and the whole run is masked with an opaque error. Mirroring the isinstance guard here would make the two paths consistent.
| error_type="timeout", | ||
| ) | ||
| # docker exec returns 125/126/127 for docker-level failures (container gone, not executable). | ||
| error_type = "sandbox" if rc in (125, 126, 127) and not out else None |
There was a problem hiding this comment.
nemo_gym/sandbox/providers/docker/provider.py:261
Classifying rc in (125, 126, 127) with empty stdout as a sandbox (infra) failure can misfire: 126 (not executable) and 127 (command not found) are legitimate exit codes for a user command run via bash -c. Only 125 is genuinely docker-daemon-level. Narrowing to rc == 125 (or inspecting stderr like Apptainer's _is_runtime_failure) would avoid masking a real command failure. Low severity since the misclassification is conservative (it depresses rather than inflates reward).
| # passed set. Resolution is not gated on patch application, and the | ||
| # F2P/P2P sets are not required to be non-empty (an empty set is a subset | ||
| # of any set). | ||
| resolved = (fail_to_pass_set <= passed_set) and (pass_to_pass_set <= passed_set) |
There was a problem hiding this comment.
responses_api_agents/swe_env/harnesses/swe_rebench.py:349
swe-rebench uses raw subset logic (f2p_set <= passed and p2p_set <= passed) rather than going through compute_resolved, which guards if not required: return False (harness.py:353). The consequence is that a row with empty fail_to_pass and pass_to_pass resolves to True/reward 1.0 for any non-empty patch (the empty set is a subset of anything). It's tested as intentional, but it makes swe-rebench the one family that inflates on a degenerate row while the others deflate. A one-line empty-required guard would make it consistent.
| error_kind="eval_timeout", | ||
| tests_status={"timeout_s": timeout}, | ||
| ) | ||
| except Exception as exc: # non-timeout eval-stage failure -> unmasked reward 0 |
There was a problem hiding this comment.
responses_api_agents/swe_env/verify_task.py:163
Non-timeout grading-stage failures return error_kind=None (unmasked reward 0). On Docker with pull-on-demand images, an image-pull failure or SandboxCreateError during grading would silently depress the resolve rate rather than being masked — and it's inconsistent with swe_rebench, which masks parser-crash / missing-clone as eval_error. The comment notes this mirrors main intentionally; would it be worth masking SandboxCreateError / image-pull failures on the docker path so infra hiccups don't leak into the training signal?
| return results | ||
|
|
||
|
|
||
| def parse_test_output(output: str, framework: str) -> Dict[str, str]: |
There was a problem hiding this comment.
responses_api_agents/swe_env/parsing/parsing.py:1544
This dispatcher fans out to ~13 per-framework parsers (jest/mocha/gtest/maven/cargo-nextest/bun/cppunit/minitest/…), but the test suite only exercises pytest, go, and TAP — the rest have no direct coverage, and they map raw output → PASS/FAIL → reward for multilingual instances, so a bug there silently corrupts reward with nothing to catch it. (This file is byte-identical to swe_agents/swe_bench_ext/parsing.py, so it's pre-existing test-debt rather than new — but since reward depends on it, a starter test_parsing_frameworks.py would help. The following 7 cases were verified passing locally:)
from responses_api_agents.swe_env.parsing.parsing import parse_test_output
def test_parse_test_output_jest_json():
out = (
'{"testResults":[{"name":"a.test.js","status":"failed",'
'"assertionResults":['
'{"fullName":"math adds","status":"passed"},'
'{"fullName":"math subtracts","status":"failed"},'
'{"fullName":"math pending","status":"skipped"}'
']}]}'
)
assert parse_test_output(out, 'jest') == {
'a.test.js::math adds': 'PASSED',
'a.test.js::math subtracts': 'FAILED',
'a.test.js::math pending': 'SKIPPED',
}
def test_parse_test_output_maven_summary():
out = '[INFO] Building MyModule 1.0 [1/1]\nTests run: 3, Failures: 1, Errors: 0, Skipped: 0\n'
res = parse_test_output(out, 'maven')
assert res['MyModule::test_1'] == 'PASSED'
assert res['MyModule::test_2'] == 'PASSED'
assert res['MyModule::test_failed_1'] == 'FAILED'
def test_parse_test_output_maven_build_failure():
assert parse_test_output('...\nBUILD FAILURE\n...', 'maven') == {'maven::build': 'FAILED'}
def test_parse_test_output_cargo_nextest():
out = 'PASS [ 1.588s] mycrate tests::test_alpha\nFAIL [ 0.500s] mycrate tests::test_beta\n'
assert parse_test_output(out, 'cargo-nextest') == {
'mycrate tests::test_alpha': 'PASSED',
'mycrate tests::test_beta': 'FAILED',
}
def test_parse_test_output_cargo_nextest_compile_error_is_none_to_empty():
assert parse_test_output('error[E0425]: cannot find value\n', 'cargo-nextest') == {}
def test_parse_test_output_pytest_text_shortform_fallback():
res = parse_test_output('tests/test_foo.py .F.s\n', 'pytest')
assert res == {
'tests/test_foo.py::test_1': 'PASSED',
'tests/test_foo.py::test_2': 'FAILED',
'tests/test_foo.py::test_3': 'PASSED',
'tests/test_foo.py::test_4': 'SKIPPED',
}
def test_parse_test_output_unknown_framework_returns_empty():
assert parse_test_output('some random output', 'totally-unknown-framework') == {}| | --------------------------------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----- | ---------- | --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| | Aalcr | other | - | - | - | - | - | <a href='resources_servers/aalcr/configs/aalcr.yaml'>aalcr.yaml</a> | - | | ||
| | Abstention | rlhf | Train models to abstain when unsure using three-tier reward on HotPotQA with LLM judge | Improve calibration by rewarding abstention over incorrect answers | ✓ | ✓ | Creative Commons Attribution-ShareAlike 4.0 International | <a href='resources_servers/abstention/configs/abstention.yaml'>abstention.yaml</a> | - | | ||
| | Anyswe Agent | coding | SWE-bench run by claude-code natively inside the task container. | Eval software engineering capabilities on SWE-bench with any Gym agent. | - | - | - | <a href='responses_api_agents/anyswe_agent/configs/anyswe_claude_code.yaml'>anyswe_claude_code.yaml</a> | - | |
There was a problem hiding this comment.
Since the harness-agnostic SWE environment and the new user-selectable Docker sandbox provider (sandbox.provider=docker) are notable user-facing features, a fern/ docs-site page (or a note in the existing sandbox docs) would help discoverability beyond the in-tree READMEs. Reasonable as a follow-up.
…ng tests) - docker provider: configurable default exec timeout + bounded close/cp so a hung in-container command can't block a rollout; clear error when docker is missing (wrap FileNotFoundError); classify only rc 125 (not 126/127 user-command codes) as an infra failure. - verify_task: mask SandboxCreateError / image-pull failures (error_kind=sandbox) so infra hiccups on the docker pull-on-demand path don't depress resolve rate. - swe_rebench: empty-required guard (consistent with compute_resolved) so a degenerate row no longer inflates to reward 1.0. - anyswe _setup_params: accept instance_dict as str OR dict (mirror _build_swetask). - shlex.quote dataset values interpolated into bash -c (nv_internal/harness/ swe_bench_ext). - r2egym docstring: missing eval_script is unmasked reward-0, not an eval-error mask. - openclaw config: select the apptainer provider for its .sif formatter; drop dead apptainer_memory_limit_mb/skip_eval fields. - tests: add test_parsing_frameworks.py (jest/maven/cargo/pytest/unknown); docker regression tests (rc125-only, 126/127 user-errors, default timeout, missing-docker); split verify_task into sandbox-masked vs generic-unmasked; swe_rebench empty-required. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>
…ine) (#1802) Addresses @ffrujeri's review on #1572 (all 13 inline comments) plus the reward-profiling baseline. Targets `cmunley1/anyswe`. > Note: the swe_env `README.md` / `ng_test_all` CI-unblock landed separately via #1796 (merged). This PR is the review-comment fixes + docs + gold baseline on top of that. ## Review-comment fixes (@ffrujeri) | # | Comment | Fix | |---|---|---| | 1 | truncation mask never fires | inner agents (hermes/claude_code) now emit top-level `status="incomplete"` on internal max-turns/context/timeout; anyswe masks `resolved AND incomplete` (verified the status round-trips through `response.json`). openclaw exposes no internal signal → documented. | | 2 | openclaw config not runnable | select the apptainer provider for its `.sif` formatter; drop dead `apptainer_memory_limit_mb`/`skip_eval` fields. | | 3 | docker timeout gaps | configurable default `exec` timeout + bounded `close`/`cp` so a hung in-container command can't block a rollout. | | 4 | reward-profiling evidence | gold-patch baseline in the swe_env README (below). | | 5 | `_setup_params` assumes str `instance_dict` | accept str **or** dict (mirror `_build_swetask`). | | 6 | r2egym docstring | missing `eval_script` is unmasked reward-0, not an eval-error mask. | | 7 | missing docker binary | wrap `FileNotFoundError` → clear `SandboxCreateError`. | | 8 | unquoted shell interpolation | `shlex.quote` dataset values in `bash -c` (nv_internal/harness/swe_bench_ext). | | 9 | rc 125/126/127 infra-classify | narrow to **rc 125** only (126/127 are legit user-command codes). | | 10 | swe_rebench inflates on empty-required | empty-required guard (consistent with `compute_resolved`). | | 11 | unmasked infra failure on docker grade | mask `SandboxCreateError`/image-pull (`error_kind="sandbox"`). | | 12 | per-framework parsers untested | add `test_parsing_frameworks.py` (the 7 reviewer cases). | | 13 | docs discoverability | add a fern SWE-Environment page. | ## Gold-patch baseline (review #4) — at parity with the nested reference A full 500-instance SWE-bench Verified **gold-patch census on docker** resolves **493/500** (`patch_exists` 500/500, 0 infra errors), matching the apptainer/`.sif` *nested* reference **492/500** to within environment noise; empty patch **0/500**. (docker and apptainer both use the host-side flat grader here, so they're 1-1; the `.sif` figure is swebench's nested `run_evaluation`.) The census surfaced + fixed two real flat↔nested **reconstruction** gaps (445 → 486 → 493), verified with **0 regressions**: - **`PYTEST_ADDOPTS=-rA`**: swebench 4.1.0's eval for some families (sphinx via `tox`, several sklearn) runs pytest without `-rA`, so passing tests print only as dots and the host-side parser saw zero passes → ~45 unresolved even for gold (445→486). - **drop `GIT_CONFIG_GLOBAL=/dev/null`**: older images' git can't parse `/dev/null`, so the eval's `git checkout` + test-patch `git apply` failed → required tests "absent" (486→493). The remaining 7 misses are a small symmetric difference with `.sif` (4 shared genuine env-flaky: astropy-7606/8707/8872, django-10097; 3 docker-only sphinx instance-specific quirks; docker also resolves 4 that `.sif` misses). ## Validation - 258 unit tests pass (swe_env + docker provider + anyswe + claude_code/openclaw); ruff clean; `fern check` clean. - The `#1` truncation logic was adversarially verified end-to-end (top-level `status` survives the container→host round-trip; the mask is live). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Codex <codex@openai.com> Co-authored-by: Codex <codex@openai.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve the sole conflict (claude_code_agent/tests/test_app.py): keep main's new tests (test_skills_staged_and_bare_dropped + test_bad_skills_path_does_not_leak_config_dir) AND the anyswe truncation test, named test_timeout_emits_incomplete_result to match its body (asserts usage['incomplete'] is True) and the merged app.py's emit-incomplete-on-timeout behavior. anyswe (19) + claude_code (44) tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>
|
🌿 Preview your docs: https://nvidia-preview-cmunley1-anyswe.docs.buildwithfern.com/nemo/gym Here are the markdown pages you've updated: |
The cmd_timeouts param pushed __init__ over 119 cols; ruff-format wraps it. (Surfaced by CI's ruff-format hook once the merge made #1572 mergeable and the full suite ran.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>
…(CI fix)
Server-suite shard 6 failed: test_flat_eval_script_forces_pytest_addopts did
monkeypatch.setattr('swebench.harness.test_spec.test_spec.make_test_spec', ...),
whose string target forces importing swebench to resolve it -> ModuleNotFoundError
in the swe_env unit-test venv (which intentionally does NOT pin swebench; the grader's
swebench paths are covered via fixtures + the GraderDependencyError test). Stub the
swebench module chain in sys.modules instead, so the -rA assertion still runs in CI
without the heavy dep. Verified passing both with swebench installed and with it blocked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Codex <codex@openai.com>
|
To be merged after release branch is cut |
# AnySWE: folder-structure & config fixes to align with AnyTerminal / Harbor ## Summary Small follow-up fixes on top of the AnySWE work to bring it in line with the AnyTerminal setup and remove duplication between the two. Three things: 1. **Fix the SWE Hermes config** to match the Harbor setup we already use in AnyTerminal. 2. **Refactor the AnySWE folder structure** (deps + results) to match AnyTerminal so it's easier to use. 3. **Move agent dependency scripts into their respective agent folders** so AnyTerminal and AnySWE no longer keep duplicate copies. ## Changes ### 1. Hermes config aligned with Harbor (`configs/anyswe_hermes.yaml`) Replaced the ad-hoc agent kwargs with the same knobs Harbor/AnyTerminal use: - `max_turns` 30 → 90 - Dropped `temperature`, `enabled_toolsets`, and the inline `system_prompt` override (use the agent defaults) - Added `terminal_timeout: 180`, `compression_enabled: true`, `compression_threshold: 0.85` ### 2. Folder structure refactor (`app.py`, `.gitignore`) - Agent deps now install under `deps/anyswe_<agent>_deps` instead of a flat `anyswe_<agent>_deps` at the parent level. - Setup scripts are resolved from the agent's own folder (`responses_api_agents/<agent>/scripts/`) and requirements from the agent's `requirements.txt`, rather than local copies. - `_portable_python.sh` is passed to the dep script via `PORTABLE_PYTHON_SH` so the script no longer assumes its own location. - Added a `results_dir` config option: when set, results are written there (session id derived from the dir name); otherwise falls back to the previous auto-generated `anyswe_results_<session_id>` workspace dir. - `.gitignore` updated for the new `results/` and `deps/` dirs. ### 3. De-duplicated agent dependency scripts Dep scripts moved out of `anyswe_agent/setup_scripts/` into each agent's own folder: - `hermes_agent/scripts/hermes_agent_deps.sh` - `openclaw_agent/scripts/openclaw_agent_deps.sh` - `claude_code_agent/scripts/claude_code_agent_deps.sh` (removed the duplicate under AnySWE) AnySWE and AnyTerminal now both consume these from a single source instead of maintaining their own copies. ## Tests - Updated `anyswe_agent/tests/test_app.py`, `swe_env/tests/test_swe_env.py`, and `swe_env/tests/test_swebench.py` for the new paths. ## Notes - `.gitignore` currently lists `results/` twice — harmless, worth squashing if we touch it again. --------- Signed-off-by: Elisa Martinez Abad <elisam@nvidia.com>
# anyswe_agent Update for #1572 to use sandbox api Runs any Gym agent inside a SWE-style task container and evaluates the resulting `git diff HEAD` patch with the dataset harness. Works with `hermes_agent`, `claude_code_agent`, or another compatible Gym agent. # Quickstart From the repo root, create `env.yaml` for the policy model server: ```yaml policy_base_url: http://localhost:10240/v1 policy_api_key: EMPTY policy_model_name: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 ``` Prepare 5 examples (tasks+images): ```bash python responses_api_agents/anyswe_agent/prepare.py --limit 5 ``` Start the environment: ```bash ng_run "+config_paths=[responses_api_agents/anyswe_agent/configs/anyswe_hermes.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]" \ ++anyswe_hermes.responses_api_agents.anyswe_agent.container_formatter='responses_api_agents/anyswe_agent/data/sifs/{instance_id}.sif' ``` Collect rollouts: ```bash ng_collect_rollouts \ +agent_name=anyswe_hermes \ +input_jsonl_fpath=responses_api_agents/anyswe_agent/data/swebench_verified.jsonl \ +output_jsonl_fpath=results/anyswe_rollouts.jsonl \ +limit=5 ``` Each rollout row contains `reward`, the full trajectory, and `mask_sample` for timeouts or unreliable rewards. # Agent wiring Point the config at the Gym agent server: ```yaml agent_server_module: responses_api_agents.hermes_agent.app agent_server_class: HermesAgent agent_config_class: HermesAgentConfig agent_kwargs: {max_turns: 100, terminal_backend: local} ``` Agent dependencies install once at startup into a portable prefix mounted inside the task container. Add `setup_scripts/<agent_dir>_deps.sh` for new agents. # Dataset and images `prepare.py` writes `data/swebench_verified.jsonl` and builds `data/sifs/{instance_id}.sif`. ```bash python responses_api_agents/anyswe_agent/prepare.py ``` Image builds require `apptainer`, network access to the SWE-bench registry, and substantial disk space. Use `--limit` and `--jobs N` while iterating. Dataset prep requires `pip install datasets`. Supported datasets: SWE-bench, SWE-bench Multilingual, R2E-Gym. --------- Signed-off-by: Christian Munley <cmunley@nvidia.com>
# anyswe_agent Update for NVIDIA-NeMo#1572 to use sandbox api Runs any Gym agent inside a SWE-style task container and evaluates the resulting `git diff HEAD` patch with the dataset harness. Works with `hermes_agent`, `claude_code_agent`, or another compatible Gym agent. # Quickstart From the repo root, create `env.yaml` for the policy model server: ```yaml policy_base_url: http://localhost:10240/v1 policy_api_key: EMPTY policy_model_name: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 ``` Prepare 5 examples (tasks+images): ```bash python responses_api_agents/anyswe_agent/prepare.py --limit 5 ``` Start the environment: ```bash ng_run "+config_paths=[responses_api_agents/anyswe_agent/configs/anyswe_hermes.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]" \ ++anyswe_hermes.responses_api_agents.anyswe_agent.container_formatter='responses_api_agents/anyswe_agent/data/sifs/{instance_id}.sif' ``` Collect rollouts: ```bash ng_collect_rollouts \ +agent_name=anyswe_hermes \ +input_jsonl_fpath=responses_api_agents/anyswe_agent/data/swebench_verified.jsonl \ +output_jsonl_fpath=results/anyswe_rollouts.jsonl \ +limit=5 ``` Each rollout row contains `reward`, the full trajectory, and `mask_sample` for timeouts or unreliable rewards. # Agent wiring Point the config at the Gym agent server: ```yaml agent_server_module: responses_api_agents.hermes_agent.app agent_server_class: HermesAgent agent_config_class: HermesAgentConfig agent_kwargs: {max_turns: 100, terminal_backend: local} ``` Agent dependencies install once at startup into a portable prefix mounted inside the task container. Add `setup_scripts/<agent_dir>_deps.sh` for new agents. # Dataset and images `prepare.py` writes `data/swebench_verified.jsonl` and builds `data/sifs/{instance_id}.sif`. ```bash python responses_api_agents/anyswe_agent/prepare.py ``` Image builds require `apptainer`, network access to the SWE-bench registry, and substantial disk space. Use `--limit` and `--jobs N` while iterating. Dataset prep requires `pip install datasets`. Supported datasets: SWE-bench, SWE-bench Multilingual, R2E-Gym. --------- Signed-off-by: Christian Munley <cmunley@nvidia.com>
anyswe_agent
Runs any Gym agent inside a SWE-style task container and evaluates the resulting
git diff HEADpatch with the dataset harness. Works withhermes_agent,claude_code_agent, or another compatible Gym agent.Quickstart
From the repo root, create
env.yamlfor the policy model server:Prepare 5 examples (tasks+images):
Start the environment:
Collect rollouts:
Each rollout row contains
reward, the full trajectory, andmask_samplefortimeouts or unreliable rewards.
Agent wiring
Point the config at the Gym agent server:
Agent dependencies install once at startup into a portable prefix mounted inside
the task container. Add
setup_scripts/<agent_dir>_deps.shfor new agents.Dataset and images
prepare.pywritesdata/swebench_verified.jsonland buildsdata/sifs/{instance_id}.sif.Image builds require
apptainer, network access to the SWE-bench registry, andsubstantial disk space. Use
--limitand--jobs Nwhile iterating. Datasetprep requires
pip install datasets.Supported datasets: SWE-bench, SWE-bench Multilingual, R2E-Gym.