Skip to content

feat: harness agnostic swe environment - #1572

Closed
cmunley1 wants to merge 26 commits into
mainfrom
cmunley1/anyswe
Closed

cmunley1 wants to merge 26 commits into
mainfrom
cmunley1/anyswe

Conversation

@cmunley1

@cmunley1 cmunley1 commented Jun 10, 2026 •

Copy link
Copy Markdown
Contributor

anyswe_agent

Runs any Gym agent inside a SWE-style task container and evaluates the resulting
git diff HEAD patch with the dataset harness. Works with hermes_agent,
claude_code_agent, or another compatible Gym agent.

Quickstart

From the repo root, create env.yaml for the policy model server:

policy_base_url: http://localhost:10240/v1
policy_api_key: EMPTY
policy_model_name: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

Prepare 5 examples (tasks+images):

python responses_api_agents/anyswe_agent/prepare.py --limit 5

Start the environment:

ng_run "+config_paths=[responses_api_agents/anyswe_agent/configs/anyswe_hermes.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]" \
  ++anyswe_hermes.responses_api_agents.anyswe_agent.container_formatter='responses_api_agents/anyswe_agent/data/sifs/{instance_id}.sif'

Collect rollouts:

ng_collect_rollouts \
  +agent_name=anyswe_hermes \
  +input_jsonl_fpath=responses_api_agents/anyswe_agent/data/swebench_verified.jsonl \
  +output_jsonl_fpath=results/anyswe_rollouts.jsonl \
  +limit=5

Each rollout row contains reward, the full trajectory, and mask_sample for
timeouts or unreliable rewards.

Agent wiring

Point the config at the Gym agent server:

agent_server_module: responses_api_agents.hermes_agent.app
agent_server_class: HermesAgent
agent_config_class: HermesAgentConfig
agent_kwargs: {max_turns: 100, terminal_backend: local}

Agent dependencies install once at startup into a portable prefix mounted inside
the task container. Add setup_scripts/<agent_dir>_deps.sh for new agents.

Dataset and images

prepare.py writes data/swebench_verified.jsonl and builds
data/sifs/{instance_id}.sif.

python responses_api_agents/anyswe_agent/prepare.py

Image builds require apptainer, network access to the SWE-bench registry, and
substantial disk space. Use --limit and --jobs N while iterating. Dataset
prep requires pip install datasets.

Supported datasets: SWE-bench, SWE-bench Multilingual, R2E-Gym.

cmunley1 added 4 commits June 10, 2026 03:26
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jun 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

cmunley1 added 4 commits June 10, 2026 16:42
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
@cmunley1
cmunley1 marked this pull request as ready for review June 10, 2026 23:57
cmunley1 added 5 commits June 10, 2026 16:58
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
@cmunley1
cmunley1 requested a review from sdevare-nv June 11, 2026 00:21
cmunley1 added 5 commits June 10, 2026 17:22
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
adil-a pushed a commit to adil-a/Gym that referenced this pull request Jun 26, 2026
…b + verifier + docker provider; drop openclaw)

Self-contained anyswe-on-swe_env on top of cmunley1/anyswe (now merged with latest main):
- add swe_env library (responses_api_agents/swe_env/) + inline verifier (resources_servers/swe_env/)
- add the docker sandbox provider + register it (main has apptainer/opensandbox, not docker)
- replace anyswe_agent with the swe_env-based version: docker + apptainer backends, local-.sif
  support, --writable-tmpfs, claude_code anthropic_base_url=null (real Anthropic by default)
- drop openclaw + orphaned dataset setup scripts + unused example_rollouts (out of scope)

Supersedes the NVIDIA-NeMo#1677 stack (NVIDIA-NeMo#1677/NVIDIA-NeMo#1678/NVIDIA-NeMo#1679) and the prior NVIDIA-NeMo#1738 base (stack/3).
Unit tests: anyswe 16, swe_env lib 134 (+1 skip), verifier 14 (+2 skip) = 164 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
adil-a pushed a commit to adil-a/Gym that referenced this pull request Jun 26, 2026
…d by this stack)

responses_api_agents/openclaw_agent/ exists on main; it entered the NVIDIA-NeMo#1572 base
via the main-merge (22f8ead), not from cmunley's original anyswe work (pr-1572
had zero openclaw files). The earlier "drop openclaw" trim therefore deleted a
main artifact, so merging the stack down (NVIDIA-NeMo#1738 -> NVIDIA-NeMo#1572 -> main) would have
removed openclaw_agent/ from main. Restore the openclaw files from the base so
this PR neither adds nor removes openclaw:

- Restore responses_api_agents/openclaw_agent/ (11 files).
- Restore responses_api_agents/anyswe_agent/configs/anyswe_openclaw.yaml and
  setup_scripts/openclaw_agent_deps.sh.

Still trimmed (these are NVIDIA-NeMo#1572-only, absent from main): the orphaned
swebench.sh / r2e_gym.sh / swebench_multilingual.sh setup scripts and the unused
example_rollouts_{claude,hermes}.jsonl fixtures.

openclaw + anyswe suites green (45 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Codex <codex@openai.com>
…erified (#1249) (#1738)

## What this is

Converges the **anyswe** runner (#1572) onto the **#1677 swe_env
decoupling**: a thin agent drops any Gym agent into the SWE task
container, extracts `git diff HEAD`, and grades **in-process** via
`verify_task` — no separate resources server. Based on the #1677 stack,
so this diff is just the convergence + fixes (~55 files).

## Before / after

```text
BEFORE (main)

  task --> swe_agents     (OpenHands-coupled; sandbox + grading inline; apptainer-only)


AFTER (this PR)                                       [NEW] = added in this PR

  task --> anyswe_agent --> swe_env/   [NEW shared library]
             (any Gym agent       +-- harnesses/    (extracted from swe_agents)
              in the container)    +-- parsing/      (verbatim from main)
                                   +-- verify_task   (inline grade; no /verify server)
                                          |
                                          v
                      nemo_gym/sandbox providers:   docker [NEW]  |  apptainer
```

## What changed

| Path | Change |
|---|---|
| `responses_api_agents/swe_env/` | **New** shared library —
`harnesses/`, `parsing/`, sandbox lifecycle, `verify_task`, `self_drive`
|
| `responses_api_agents/anyswe_agent/` | Thin runner (#1572) rewired
onto `swe_env`; grades `git diff` in-process |
| `nemo_gym/sandbox/providers/docker/` | **New** docker provider (was
apptainer-only) — runs Verified on docker |
| `…/swe_env/parsing/` | Byte-identical relocation of main's
`swe_agents/swe_bench_ext/` (lighthouse parser) |
| `…/swe_env/verify_task.py` | Moved from `resources_servers/swe_env/`;
inline grader, importers repointed |
| `resources_servers/swe_env/` | **Removed** — no consumer; grading is
inline, not a `/verify` server |
|
`…/anyswe_agent/setup_scripts/{swebench,r2e_gym,swebench_multilingual}.sh`
| **Removed** — superseded by `harnesses/` + pip `swebench` |
| `responses_api_agents/openclaw_agent/` | Untouched (a `main` artifact
— neither added nor removed) |

Other fixes: docker flat-eval uses swebench's official per-repo parser
(gold resolve 4%→95%); per-repo FAIL_ONLY eval-type + main's patch-apply
ladder ported into the harnesses; apptainer `--pid` removed (≥1.3.6
rejects it); `get_harness` self-registers + HF-name tolerant.

## Suggested review order

1. `nemo_gym/sandbox/providers/registry.py` + `providers/docker/` — the
new provider lazy-registration (small, foundational).
2. `responses_api_agents/swe_env/harness.py` — the contract: `SweTask`,
the harness ABC, the registry, and the resolution rule
(`compute_resolved`). **Read first** — everything else builds on it.
3. `responses_api_agents/swe_env/verify_task.py` — the inline grading
orchestrator (the "no `/verify` server" core).
4. `responses_api_agents/swe_env/sandbox.py` + `self_drive.py` — sandbox
lifecycle, then provision / egress / `git diff` extraction.
5. `responses_api_agents/swe_env/harnesses/swebench.py` — the primary
harness + the parity fixes (FAIL_ONLY, patch-apply ladder, masking);
then `swe_bench_ext.py` / `r2egym.py` / `flat_eval.py` as needed.
6. `responses_api_agents/anyswe_agent/app.py` — how it wires together:
routing → `_build_swetask` → provider → `self_drive` + `verify_task`.
7. `responses_api_agents/swe_env/parsing/` — **skim only**:
byte-identical relocation from main, no new logic.
8. `…/swe_env/tests/` + `…/anyswe_agent/tests/` — coverage for the
above.

## Validation — SWE-bench Verified (N=100/cell), in-line grader

Each agent × model run on the **same 100 instances** on **both** sandbox
backends; cells are resolve rate (`mean/reward`), all clean (mask ≤
0.01):

| agent × model | docker | apptainer |
|---|---|---|
| hermes × gpt-5.5 | 69% | 66% |
| hermes × opus-4.8 | 77% | 79% |
| claude_code × opus-4.8 | 78% | 82% |
| openclaw × gpt-5.5 | 70% | 71% |
| openclaw × opus-4.8 | 77% | 78% |

docker and apptainer agree within ±4 pts (mean Δ ≈ +1, no systematic
gap) — the two backends are equivalent. (claude_code × gpt-5.5 skipped:
the Claude CLI speaks Anthropic `/v1/messages` only.) Gold-patch grader
sanity beforehand: docker **95/100**, `.sif` **10/10**.

- A full per-failure audit found **0 our-side (infra) failures** —
non-resolved rows are model wrong/no-patch. (Found + fixed an apptainer
ENOSPC en route: `--writable-tmpfs`'s 64 MiB cap → per-instance disk
`--overlay`.)
- ruff clean, **158 unit tests pass**; harness-vs-main audit converged
(0 residual behavioral drift).

## Reproduce — gpt-5.5 on hermes (docker **or** apptainer)

Shared setup + dataset (gpt-5.5 is a reasoning model, so
temperature/top-p must be 1.0):
```bash
uv venv && uv sync --extra dev
export RAY_TMPDIR=/tmp                  # avoids Ray AF_UNIX path-length limits
export POLICY_API_KEY=sk-...            # gateway key — SECRET, pass via ++ only, never commit
python responses_api_agents/anyswe_agent/prepare.py --no-images --limit 100 --temperature 1.0 --top-p 1.0
DS=responses_api_agents/anyswe_agent/data/swebench_verified.jsonl
MODEL=us/azure/openai/eccn-gpt-5.5
A=anyswe_hermes.responses_api_agents.anyswe_agent
```

**Docker** (default provider; each instance image pulls on demand from
DockerHub — no pre-build):
```bash
# Terminal A — servers (wait for "All 2 / 2 servers ready!")
ng_run "+config_paths=[responses_api_agents/anyswe_agent/configs/anyswe_hermes.yaml,responses_api_models/openai_model/configs/openai_model.yaml]" \
  ++policy_base_url='https://inference-api.nvidia.com/v1' ++policy_model_name="$MODEL" ++policy_api_key="$POLICY_API_KEY"
# Terminal B — rollouts (num_samples_in_parallel == concurrent containers)
ng_collect_rollouts +agent_name=anyswe_hermes +input_jsonl_fpath=$DS \
  +output_jsonl_fpath=results/hermes_gpt_docker.jsonl +num_samples_in_parallel=16
```

**Apptainer** (build a `.sif` per instance once, then run with no
registry pulls):
```bash
# Build SIFs (needs apptainer on PATH; ~1-3 GB each) -> data/sifs/{instance_id}.sif
python responses_api_agents/anyswe_agent/prepare.py --no-dataset --sif-dir data/sifs
# Terminal A — same servers, but point the formatter at the .sif and select the apptainer provider.
# Quote container_formatter so Hydra treats {instance_id} as a literal; sandbox_provider is a dict.
ng_run "+config_paths=[responses_api_agents/anyswe_agent/configs/anyswe_hermes.yaml,responses_api_models/openai_model/configs/openai_model.yaml]" \
  ++policy_base_url='https://inference-api.nvidia.com/v1' ++policy_model_name="$MODEL" ++policy_api_key="$POLICY_API_KEY" \
  ++$A.container_formatter='"data/sifs/{instance_id}.sif"' \
  ++$A.sandbox_provider='{apptainer:{}}'
# Terminal B — same collect command
ng_collect_rollouts +agent_name=anyswe_hermes +input_jsonl_fpath=$DS \
  +output_jsonl_fpath=results/hermes_gpt_appt.jsonl +num_samples_in_parallel=16
```

Resolve rate = `mean/reward` in `results/<name>_aggregate_metrics.json`;
a clean run has `mean/mask_sample == 0`. (For other cells:
`anyswe_claude_code.yaml` + an opus model id speaks Anthropic
`/v1/messages` via `++$A.agent_kwargs.anthropic_base_url=…`;
`anyswe_openclaw.yaml` runs OpenClaw. Same `container_formatter` /
`sandbox_provider` overrides switch either between docker and
apptainer.)

## Deferred to a follow-up

openclaw eval (runner restored, eval deferred); R2E-Gym / Multilingual
**native nested graders** (host-side flat grading approximates those —
Verified/English is exact); routing needles for `swe-bench-ext` /
`swe-rebench` / `nv-internal`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the **server-suite CI failure** on #1572 (all 8 shards failing).

## Root cause
`ng_test_all` runs with `fail_on_total_and_test_mismatch=true` in CI. It
asserts every candidate module dir under `resources_servers/*`,
`responses_api_agents/*`, `responses_api_models/*` has a `README.md`
(and is therefore in the tested set). The new
`responses_api_agents/swe_env/` library had no README, so it was counted
as a candidate but not a tested module:

```
AssertionError: Mismatch on the number of total modules found (129) and the number of actual modules tested (128)!
Extra candidate paths:
- responses_api_agents/swe_env
```

This assertion runs in every shard, so all 8 shards fail.

## Fix
Add `responses_api_agents/swe_env/README.md`. This makes the counts
match **and** runs swe_env's unit tests in CI (FakeProvider-backed — no
Docker/apptainer; `swebench` is imported lazily so the `nemo-gym[dev]`
test venv suffices). Data validation is skipped for swe_env since it's
not under `resources_servers/`.

Only `swe_env` was missing a README; no other module is affected. DCO on
the squash-merge commit is a separate, known item.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ffrujeri
ffrujeri previously approved these changes Jun 27, 2026

@ffrujeri ffrujeri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds a provider-neutral SWE environment. anyswe_agent runs any thin Gym agent (hermes / claude_code / openclaw) inside a SWE task container, extracts its git diff HEAD, and grades the patch inline via the new swe_env library — there is no separate Resources Server. It also adds a Docker sandbox provider to the shared core nemo_gym.sandbox layer, making end-to-end SWE-bench verification runnable on a single workstation.

A couple of non-blocking notes that don't map to a single line:

  • The branch is currently behind main — a rebase before merge would be good.
  • Several parsing behaviors flagged inline (parsing.py framework coverage, fuzzy matching, XFAIL handling) are ported verbatim from the existing swe_agents/swe_bench_ext code, so they're not regressions introduced here — they're worth a follow-up against the benchmark itself.

# API "incomplete" status. A patch that happens to pass in that case is an accidental
# reward, so mask it (mirrors main's swe_agents, which masks resolved AND
# max_iteration/context_window) to keep it out of the training gradient.
agent_truncated = bool(saved is not None and saved.status == "incomplete")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

responses_api_agents/anyswe_agent/app.py:621

The truncation-masking path looks like it never fires. agent_truncated is derived from the inner agent's response status == "incomplete", but none of the three shipped inner agents (hermes_agent, claude_code_agent, openclaw_agent) ever set that status — they all build NeMoGymResponse with status="completed". So agent_truncated is always False and the (resolved and agent_truncated) mask on the next line never triggers. The net effect is that an agent which exhausts its turns/context but happens to land a passing patch is scored reward=1 and left unmasked — the exact accidental-reward case the comment says this guards against. The swe_agents it replaces classified the harness error string instead. Could the truncation signal be derived from the inner agent's actual turn/error state (e.g. via self_drive) rather than a status field nothing sets?

name: aws/anthropic/bedrock-claude-opus-4-7
api: openai-completions

container_formatter: data/sifs/{instance_id}.sif

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

responses_api_agents/anyswe_agent/configs/anyswe_openclaw.yaml:44

This example config may not be runnable as shipped: container_formatter points at an Apptainer .sif, but no sandbox_provider is set, so it falls back to the AnySweAgentConfig default of docker. _instance_image then returns the .sif path verbatim, yielding docker run data/sifs/<id>.sif ..., which Docker can't run. Setting sandbox_provider: {apptainer: {...}} would match the intent.

Also, apptainer_memory_limit_mb (L47) and skip_eval (L49) aren't fields on AnySweAgentConfig — they're silently swallowed by extra="allow", so they have no effect. Probably leftovers from the swe_agents schema; worth removing to avoid implying they're honored.

args += ["-e", f"{key}={value}"]
args += [handle.sandbox_id, "bash", "-c", command]
try:
rc, out, err = await self._run(*args, timeout_s=timeout_s)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nemo_gym/sandbox/providers/docker/provider.py:252

A few timeout gaps in the Docker provider, grouped:

  1. exec has no default timeout — timeout_s=None waits forever. The git-diff extraction in self_drive.py:194 calls exec with no timeout, so a hung in-container command would block the rollout indefinitely. The Apptainer provider applies a configurable default exec timeout.
  2. close() (rm -f, L320) and upload_file/download_file (docker cp, L278/L295) call _run with no timeout, so a wedged daemon hangs teardown.
  3. On exec timeout, proc.kill() (L113) kills only the local docker exec client, not the in-container process (bounded, since close() later rm -f's the container, but Apptainer uses killpg for parity).

Consider a configurable default exec timeout plus bounded timeouts on close/cp.

@@ -0,0 +1,32 @@
# swe_env

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

responses_api_agents/swe_env/README.md:1

Would it be possible to add reward-profiling evidence for the benchmark? Right now the resolved=True assertions are all unit tests over scripted fake-provider output, so there's no durable in-repo proof the verifier reproduces published resolve rates on real rows. A small baseline — e.g. gold patch → resolved=True and empty patch → resolved=False rates for at least SWE-bench Verified, committed as an aggregate_metrics.json or a short note — would let reviewers trust the numbers, per the add-benchmark / reward-profiling guidance. (Not a code blocker; the grading itself looks correct. It would be the natural gate before flipping any verified: true.)

dataset_dir = persistent_dir / "instance_datasets"
dataset_dir.mkdir(parents=True, exist_ok=True)
instance_dataset_path = dataset_dir / f"{agent_run_id}.jsonl"
instance_dict = json.loads(problem_info["instance_dict"])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

responses_api_agents/anyswe_agent/app.py:450

_setup_params does json.loads(problem_info["instance_dict"]), which assumes a string, while _build_swetask (L134) accepts either a str or a dict. If a row ever carries instance_dict as a dict, this raises TypeError and the whole run is masked with an opaque error. Mirroring the isinstance guard here would make the two paths consistent.

error_type="timeout",
)
# docker exec returns 125/126/127 for docker-level failures (container gone, not executable).
error_type = "sandbox" if rc in (125, 126, 127) and not out else None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nemo_gym/sandbox/providers/docker/provider.py:261

Classifying rc in (125, 126, 127) with empty stdout as a sandbox (infra) failure can misfire: 126 (not executable) and 127 (command not found) are legitimate exit codes for a user command run via bash -c. Only 125 is genuinely docker-daemon-level. Narrowing to rc == 125 (or inspecting stderr like Apptainer's _is_runtime_failure) would avoid masking a real command failure. Low severity since the misclassification is conservative (it depresses rather than inflates reward).

# passed set. Resolution is not gated on patch application, and the
# F2P/P2P sets are not required to be non-empty (an empty set is a subset
# of any set).
resolved = (fail_to_pass_set <= passed_set) and (pass_to_pass_set <= passed_set)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

responses_api_agents/swe_env/harnesses/swe_rebench.py:349

swe-rebench uses raw subset logic (f2p_set <= passed and p2p_set <= passed) rather than going through compute_resolved, which guards if not required: return False (harness.py:353). The consequence is that a row with empty fail_to_pass and pass_to_pass resolves to True/reward 1.0 for any non-empty patch (the empty set is a subset of anything). It's tested as intentional, but it makes swe-rebench the one family that inflates on a degenerate row while the others deflate. A one-line empty-required guard would make it consistent.

error_kind="eval_timeout",
tests_status={"timeout_s": timeout},
)
except Exception as exc: # non-timeout eval-stage failure -> unmasked reward 0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

responses_api_agents/swe_env/verify_task.py:163

Non-timeout grading-stage failures return error_kind=None (unmasked reward 0). On Docker with pull-on-demand images, an image-pull failure or SandboxCreateError during grading would silently depress the resolve rate rather than being masked — and it's inconsistent with swe_rebench, which masks parser-crash / missing-clone as eval_error. The comment notes this mirrors main intentionally; would it be worth masking SandboxCreateError / image-pull failures on the docker path so infra hiccups don't leak into the training signal?

return results


def parse_test_output(output: str, framework: str) -> Dict[str, str]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

responses_api_agents/swe_env/parsing/parsing.py:1544

This dispatcher fans out to ~13 per-framework parsers (jest/mocha/gtest/maven/cargo-nextest/bun/cppunit/minitest/…), but the test suite only exercises pytest, go, and TAP — the rest have no direct coverage, and they map raw output → PASS/FAIL → reward for multilingual instances, so a bug there silently corrupts reward with nothing to catch it. (This file is byte-identical to swe_agents/swe_bench_ext/parsing.py, so it's pre-existing test-debt rather than new — but since reward depends on it, a starter test_parsing_frameworks.py would help. The following 7 cases were verified passing locally:)

from responses_api_agents.swe_env.parsing.parsing import parse_test_output


def test_parse_test_output_jest_json():
    out = (
        '{"testResults":[{"name":"a.test.js","status":"failed",'
        '"assertionResults":['
        '{"fullName":"math adds","status":"passed"},'
        '{"fullName":"math subtracts","status":"failed"},'
        '{"fullName":"math pending","status":"skipped"}'
        ']}]}'
    )
    assert parse_test_output(out, 'jest') == {
        'a.test.js::math adds': 'PASSED',
        'a.test.js::math subtracts': 'FAILED',
        'a.test.js::math pending': 'SKIPPED',
    }


def test_parse_test_output_maven_summary():
    out = '[INFO] Building MyModule 1.0 [1/1]\nTests run: 3, Failures: 1, Errors: 0, Skipped: 0\n'
    res = parse_test_output(out, 'maven')
    assert res['MyModule::test_1'] == 'PASSED'
    assert res['MyModule::test_2'] == 'PASSED'
    assert res['MyModule::test_failed_1'] == 'FAILED'


def test_parse_test_output_maven_build_failure():
    assert parse_test_output('...\nBUILD FAILURE\n...', 'maven') == {'maven::build': 'FAILED'}


def test_parse_test_output_cargo_nextest():
    out = 'PASS [   1.588s] mycrate tests::test_alpha\nFAIL [   0.500s] mycrate tests::test_beta\n'
    assert parse_test_output(out, 'cargo-nextest') == {
        'mycrate tests::test_alpha': 'PASSED',
        'mycrate tests::test_beta': 'FAILED',
    }


def test_parse_test_output_cargo_nextest_compile_error_is_none_to_empty():
    assert parse_test_output('error[E0425]: cannot find value\n', 'cargo-nextest') == {}


def test_parse_test_output_pytest_text_shortform_fallback():
    res = parse_test_output('tests/test_foo.py .F.s\n', 'pytest')
    assert res == {
        'tests/test_foo.py::test_1': 'PASSED',
        'tests/test_foo.py::test_2': 'FAILED',
        'tests/test_foo.py::test_3': 'PASSED',
        'tests/test_foo.py::test_4': 'SKIPPED',
    }


def test_parse_test_output_unknown_framework_returns_empty():
    assert parse_test_output('some random output', 'totally-unknown-framework') == {}

Comment thread README.md
| --------------------------------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----- | ---------- | --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Aalcr | other | - | - | - | - | - | <a href='resources_servers/aalcr/configs/aalcr.yaml'>aalcr.yaml</a> | - |
| Abstention | rlhf | Train models to abstain when unsure using three-tier reward on HotPotQA with LLM judge | Improve calibration by rewarding abstention over incorrect answers | ✓ | ✓ | Creative Commons Attribution-ShareAlike 4.0 International | <a href='resources_servers/abstention/configs/abstention.yaml'>abstention.yaml</a> | - |
| Anyswe Agent | coding | SWE-bench run by claude-code natively inside the task container. | Eval software engineering capabilities on SWE-bench with any Gym agent. | - | - | - | <a href='responses_api_agents/anyswe_agent/configs/anyswe_claude_code.yaml'>anyswe_claude_code.yaml</a> | - |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README.md:182

Since the harness-agnostic SWE environment and the new user-selectable Docker sandbox provider (sandbox.provider=docker) are notable user-facing features, a fern/ docs-site page (or a note in the existing sandbox docs) would help discoverability beyond the in-tree READMEs. Reasonable as a follow-up.

adil-a pushed a commit to adil-a/Gym that referenced this pull request Jun 27, 2026
…ng tests)

- docker provider: configurable default exec timeout + bounded close/cp so a hung
  in-container command can't block a rollout; clear error when docker is missing
  (wrap FileNotFoundError); classify only rc 125 (not 126/127 user-command codes)
  as an infra failure.
- verify_task: mask SandboxCreateError / image-pull failures (error_kind=sandbox)
  so infra hiccups on the docker pull-on-demand path don't depress resolve rate.
- swe_rebench: empty-required guard (consistent with compute_resolved) so a
  degenerate row no longer inflates to reward 1.0.
- anyswe _setup_params: accept instance_dict as str OR dict (mirror _build_swetask).
- shlex.quote dataset values interpolated into bash -c (nv_internal/harness/
  swe_bench_ext).
- r2egym docstring: missing eval_script is unmasked reward-0, not an eval-error mask.
- openclaw config: select the apptainer provider for its .sif formatter; drop dead
  apptainer_memory_limit_mb/skip_eval fields.
- tests: add test_parsing_frameworks.py (jest/maven/cargo/pytest/unknown); docker
  regression tests (rc125-only, 126/127 user-errors, default timeout, missing-docker);
  split verify_task into sandbox-masked vs generic-unmasked; swe_rebench empty-required.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Codex <codex@openai.com>
kajalj22
kajalj22 previously approved these changes Jun 28, 2026
…ine) (#1802)

Addresses @ffrujeri's review on #1572 (all 13 inline comments) plus the
reward-profiling baseline. Targets `cmunley1/anyswe`.

> Note: the swe_env `README.md` / `ng_test_all` CI-unblock landed
separately via #1796 (merged). This PR is the review-comment fixes +
docs + gold baseline on top of that.

## Review-comment fixes (@ffrujeri)
| # | Comment | Fix |
|---|---|---|
| 1 | truncation mask never fires | inner agents (hermes/claude_code)
now emit top-level `status="incomplete"` on internal
max-turns/context/timeout; anyswe masks `resolved AND incomplete`
(verified the status round-trips through `response.json`). openclaw
exposes no internal signal → documented. |
| 2 | openclaw config not runnable | select the apptainer provider for
its `.sif` formatter; drop dead `apptainer_memory_limit_mb`/`skip_eval`
fields. |
| 3 | docker timeout gaps | configurable default `exec` timeout +
bounded `close`/`cp` so a hung in-container command can't block a
rollout. |
| 4 | reward-profiling evidence | gold-patch baseline in the swe_env
README (below). |
| 5 | `_setup_params` assumes str `instance_dict` | accept str **or**
dict (mirror `_build_swetask`). |
| 6 | r2egym docstring | missing `eval_script` is unmasked reward-0, not
an eval-error mask. |
| 7 | missing docker binary | wrap `FileNotFoundError` → clear
`SandboxCreateError`. |
| 8 | unquoted shell interpolation | `shlex.quote` dataset values in
`bash -c` (nv_internal/harness/swe_bench_ext). |
| 9 | rc 125/126/127 infra-classify | narrow to **rc 125** only (126/127
are legit user-command codes). |
| 10 | swe_rebench inflates on empty-required | empty-required guard
(consistent with `compute_resolved`). |
| 11 | unmasked infra failure on docker grade | mask
`SandboxCreateError`/image-pull (`error_kind="sandbox"`). |
| 12 | per-framework parsers untested | add `test_parsing_frameworks.py`
(the 7 reviewer cases). |
| 13 | docs discoverability | add a fern SWE-Environment page. |

## Gold-patch baseline (review #4) — at parity with the nested reference
A full 500-instance SWE-bench Verified **gold-patch census on docker**
resolves **493/500** (`patch_exists` 500/500, 0 infra errors), matching
the apptainer/`.sif` *nested* reference **492/500** to within
environment noise; empty patch **0/500**. (docker and apptainer both use
the host-side flat grader here, so they're 1-1; the `.sif` figure is
swebench's nested `run_evaluation`.)

The census surfaced + fixed two real flat↔nested **reconstruction** gaps
(445 → 486 → 493), verified with **0 regressions**:
- **`PYTEST_ADDOPTS=-rA`**: swebench 4.1.0's eval for some families
(sphinx via `tox`, several sklearn) runs pytest without `-rA`, so
passing tests print only as dots and the host-side parser saw zero
passes → ~45 unresolved even for gold (445→486).
- **drop `GIT_CONFIG_GLOBAL=/dev/null`**: older images' git can't parse
`/dev/null`, so the eval's `git checkout` + test-patch `git apply`
failed → required tests "absent" (486→493).

The remaining 7 misses are a small symmetric difference with `.sif` (4
shared genuine env-flaky: astropy-7606/8707/8872, django-10097; 3
docker-only sphinx instance-specific quirks; docker also resolves 4 that
`.sif` misses).

## Validation
- 258 unit tests pass (swe_env + docker provider + anyswe +
claude_code/openclaw); ruff clean; `fern check` clean.
- The `#1` truncation logic was adversarially verified end-to-end
(top-level `status` survives the container→host round-trip; the mask is
live).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@adil-a
adil-a dismissed stale reviews from kajalj22 and ffrujeri via 4aa34fc June 29, 2026 06:46
Resolve the sole conflict (claude_code_agent/tests/test_app.py): keep main's new tests
(test_skills_staged_and_bare_dropped + test_bad_skills_path_does_not_leak_config_dir) AND
the anyswe truncation test, named test_timeout_emits_incomplete_result to match its body
(asserts usage['incomplete'] is True) and the merged app.py's emit-incomplete-on-timeout
behavior. anyswe (19) + claude_code (44) tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Codex <codex@openai.com>
@github-actions

Copy link
Copy Markdown
Contributor

🌿 Preview your docs: https://nvidia-preview-cmunley1-anyswe.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

codex and others added 2 commits June 29, 2026 06:58
The cmd_timeouts param pushed __init__ over 119 cols; ruff-format wraps it. (Surfaced by
CI's ruff-format hook once the merge made #1572 mergeable and the full suite ran.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Codex <codex@openai.com>
…(CI fix)

Server-suite shard 6 failed: test_flat_eval_script_forces_pytest_addopts did
monkeypatch.setattr('swebench.harness.test_spec.test_spec.make_test_spec', ...),
whose string target forces importing swebench to resolve it -> ModuleNotFoundError
in the swe_env unit-test venv (which intentionally does NOT pin swebench; the grader's
swebench paths are covered via fixtures + the GraderDependencyError test). Stub the
swebench module chain in sys.modules instead, so the -rA assertion still runs in CI
without the heavy dep. Verified passing both with swebench installed and with it blocked.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Codex <codex@openai.com>
@anwithk

anwithk commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

To be merged after release branch is cut

# AnySWE: folder-structure & config fixes to align with AnyTerminal /
Harbor

## Summary

Small follow-up fixes on top of the AnySWE work to bring it in line with
the
AnyTerminal setup and remove duplication between the two. Three things:

1. **Fix the SWE Hermes config** to match the Harbor setup we already
use in AnyTerminal.
2. **Refactor the AnySWE folder structure** (deps + results) to match
AnyTerminal so it's easier to use.
3. **Move agent dependency scripts into their respective agent folders**
so AnyTerminal and AnySWE no longer keep duplicate copies.

## Changes

### 1. Hermes config aligned with Harbor (`configs/anyswe_hermes.yaml`)

Replaced the ad-hoc agent kwargs with the same knobs Harbor/AnyTerminal
use:

- `max_turns` 30 → 90
- Dropped `temperature`, `enabled_toolsets`, and the inline
`system_prompt` override (use the agent defaults)
- Added `terminal_timeout: 180`, `compression_enabled: true`,
`compression_threshold: 0.85`

### 2. Folder structure refactor (`app.py`, `.gitignore`)

- Agent deps now install under `deps/anyswe_<agent>_deps` instead of a
flat `anyswe_<agent>_deps` at the parent level.
- Setup scripts are resolved from the agent's own folder
(`responses_api_agents/<agent>/scripts/`) and requirements from the
agent's `requirements.txt`, rather than local copies.
- `_portable_python.sh` is passed to the dep script via
`PORTABLE_PYTHON_SH` so the script no longer assumes its own location.
- Added a `results_dir` config option: when set, results are written
there (session id derived from the dir name); otherwise falls back to
the previous auto-generated `anyswe_results_<session_id>` workspace dir.
- `.gitignore` updated for the new `results/` and `deps/` dirs.

### 3. De-duplicated agent dependency scripts

Dep scripts moved out of `anyswe_agent/setup_scripts/` into each agent's
own folder:

- `hermes_agent/scripts/hermes_agent_deps.sh`
- `openclaw_agent/scripts/openclaw_agent_deps.sh`
- `claude_code_agent/scripts/claude_code_agent_deps.sh` (removed the
duplicate under AnySWE)

AnySWE and AnyTerminal now both consume these from a single source
instead of maintaining their own copies.

## Tests

- Updated `anyswe_agent/tests/test_app.py`,
`swe_env/tests/test_swe_env.py`, and `swe_env/tests/test_swebench.py`
for the new paths.

## Notes

- `.gitignore` currently lists `results/` twice — harmless, worth
squashing if we touch it again.

---------

Signed-off-by: Elisa Martinez Abad <elisam@nvidia.com>
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Jul 17, 2026
cmunley1 added a commit that referenced this pull request Jul 30, 2026
# anyswe_agent

Update for #1572 to use sandbox api 

Runs any Gym agent inside a SWE-style task container and evaluates the
resulting
`git diff HEAD` patch with the dataset harness. Works with
`hermes_agent`,
`claude_code_agent`, or another compatible Gym agent.

# Quickstart

From the repo root, create `env.yaml` for the policy model server:

```yaml
policy_base_url: http://localhost:10240/v1
policy_api_key: EMPTY
policy_model_name: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
```

Prepare 5 examples (tasks+images):

```bash
python responses_api_agents/anyswe_agent/prepare.py --limit 5
```

Start the environment:

```bash
ng_run "+config_paths=[responses_api_agents/anyswe_agent/configs/anyswe_hermes.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]" \
  ++anyswe_hermes.responses_api_agents.anyswe_agent.container_formatter='responses_api_agents/anyswe_agent/data/sifs/{instance_id}.sif'
```

Collect rollouts:

```bash
ng_collect_rollouts \
  +agent_name=anyswe_hermes \
  +input_jsonl_fpath=responses_api_agents/anyswe_agent/data/swebench_verified.jsonl \
  +output_jsonl_fpath=results/anyswe_rollouts.jsonl \
  +limit=5
```

Each rollout row contains `reward`, the full trajectory, and
`mask_sample` for
timeouts or unreliable rewards.

# Agent wiring

Point the config at the Gym agent server:

```yaml
agent_server_module: responses_api_agents.hermes_agent.app
agent_server_class: HermesAgent
agent_config_class: HermesAgentConfig
agent_kwargs: {max_turns: 100, terminal_backend: local}
```

Agent dependencies install once at startup into a portable prefix
mounted inside
the task container. Add `setup_scripts/<agent_dir>_deps.sh` for new
agents.

# Dataset and images

`prepare.py` writes `data/swebench_verified.jsonl` and builds
`data/sifs/{instance_id}.sif`.

```bash
python responses_api_agents/anyswe_agent/prepare.py
```

Image builds require `apptainer`, network access to the SWE-bench
registry, and
substantial disk space. Use `--limit` and `--jobs N` while iterating.
Dataset
prep requires `pip install datasets`.

Supported datasets: SWE-bench, SWE-bench Multilingual, R2E-Gym.

---------

Signed-off-by: Christian Munley <cmunley@nvidia.com>
OlegSudakov pushed a commit to OlegSudakov/Gym that referenced this pull request Aug 7, 2026
# anyswe_agent

Update for NVIDIA-NeMo#1572 to use sandbox api 

Runs any Gym agent inside a SWE-style task container and evaluates the
resulting
`git diff HEAD` patch with the dataset harness. Works with
`hermes_agent`,
`claude_code_agent`, or another compatible Gym agent.

# Quickstart

From the repo root, create `env.yaml` for the policy model server:

```yaml
policy_base_url: http://localhost:10240/v1
policy_api_key: EMPTY
policy_model_name: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
```

Prepare 5 examples (tasks+images):

```bash
python responses_api_agents/anyswe_agent/prepare.py --limit 5
```

Start the environment:

```bash
ng_run "+config_paths=[responses_api_agents/anyswe_agent/configs/anyswe_hermes.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]" \
  ++anyswe_hermes.responses_api_agents.anyswe_agent.container_formatter='responses_api_agents/anyswe_agent/data/sifs/{instance_id}.sif'
```

Collect rollouts:

```bash
ng_collect_rollouts \
  +agent_name=anyswe_hermes \
  +input_jsonl_fpath=responses_api_agents/anyswe_agent/data/swebench_verified.jsonl \
  +output_jsonl_fpath=results/anyswe_rollouts.jsonl \
  +limit=5
```

Each rollout row contains `reward`, the full trajectory, and
`mask_sample` for
timeouts or unreliable rewards.

# Agent wiring

Point the config at the Gym agent server:

```yaml
agent_server_module: responses_api_agents.hermes_agent.app
agent_server_class: HermesAgent
agent_config_class: HermesAgentConfig
agent_kwargs: {max_turns: 100, terminal_backend: local}
```

Agent dependencies install once at startup into a portable prefix
mounted inside
the task container. Add `setup_scripts/<agent_dir>_deps.sh` for new
agents.

# Dataset and images

`prepare.py` writes `data/swebench_verified.jsonl` and builds
`data/sifs/{instance_id}.sif`.

```bash
python responses_api_agents/anyswe_agent/prepare.py
```

Image builds require `apptainer`, network access to the SWE-bench
registry, and
substantial disk space. Use `--limit` and `--jobs N` while iterating.
Dataset
prep requires `pip install datasets`.

Supported datasets: SWE-bench, SWE-bench Multilingual, R2E-Gym.

---------

Signed-off-by: Christian Munley <cmunley@nvidia.com>
@cmunley1 cmunley1 closed this Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Decouple SWE environment infrastructure from agent harnesses

7 participants