Skip to content

fix(anyswe_agent): make R2E-Gym grading resolve - #4019

Merged
cmunley1 merged 9 commits into
NVIDIA-NeMo:mainfrom
linj-glitch:linj/anyswe-r2e-grading
Oct 29, 2026
Merged

cmunley1 merged 9 commits into
NVIDIA-NeMo:mainfrom
linj-glitch:linj/anyswe-r2e-grading

Conversation

@linj-glitch

@linj-glitch linj-glitch commented Oct 4, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

Fix R2E-Gym grading for colored pytest output, unittest-style names, verbose results, and rows using expected_output_json. Previously, patched runs could be incorrectly unresolved; ambiguous test names or status words inside failures could also produce incorrect rewards. No separate issue is needed for these focused reproductions.

Explicit test lists preserve the requested class/file identity and parameter IDs. A shortened name can match one concrete test or multiple tests with the same status; mixed statuses remain ambiguous, including PASSED mixed with XFAIL. A duplicate SKIPPED record cannot overwrite a passing result; FAILED and ERROR dominate. Failure diagnostics match complete observed names before removing suffixes, including teardown errors after passing bodies with arbitrary parameter IDs.

Rows without test lists require a nonempty exact comparison with expected_output_json. Parsing uses the leading summary status and consistent ANSI normalization. Expected maps retain the upstream key encoding and final-phase behavior, including expected teardown errors; identities already lost by that legacy encoding cannot be recovered here. Main's byte-safe patch upload and merged #4030 hidden-test linking are preserved.

Compatibility and benchmark impact

  • Test-list mode now accepts XFAIL as passing, whereas main requires PASSED. This is an intentional reward/benchmark-result change, endorsed in the author review. Matching shortened names with unanimous statuses can also change unresolved results to resolved; mixed statuses still fail closed.
  • Expected-map mode compares the entire observed status map with the dataset's expected map, including expected FAILED/ERROR outcomes. It does not apply test-list mode's XFAIL rule.
  • save_eval_output defaults to false. Set it to true to retain the last 200,000 characters in eval_output.txt for either R2E-Gym or SWE-bench. Both verifiers always receive the full log. SWE-bench reward semantics are unchanged.

Validation

Validated head: d3022c4. Python 3.13.14; imports were asserted to resolve to the reviewed checkout.

  • python -m pytest -q -o addopts= --disable-warnings responses_api_agents/anyswe_agent/tests: 128 passed.
  • The reviewer-requested alias and default transcript regressions produced 10 failures / 34 passes before the fix. The final suite covers unanimous/mixed aliases, omitted/false/true transcript settings for both dataset families, the 200,000-character retention cap, and complete verifier input.
  • pre-commit run --all-files: passed. Complete PR diff, description, and reply guardword checks and diff whitespace checks passed. Additive commits carry DCO sign-off.
  • Existing real-pytest regressions cover colorized output and passing bodies followed by failing teardown, including unmatched brackets, diagnostic separators, and reversed duplicate-result order. Expected-map encoding and byte-preserving patch tests also pass.

Actual-row grading replay

Regraded the unmodified published old/new commit execution logs from R2E-Gym/R2E-Gym-Subset. Selection was the first eligible row for each of five distinct repositories in sorted cached-shard/stored-row order, without filtering by verdict. Baseline: main c155b77c8420283982ec30347a46b92ae6ee1af5. Each row has an expected map and no test lists.

Dataset repository / commit Old-commit log, main → PR New-commit log, main → PR
orange3 / 2d9617bd0cb1f0ba61771258410ab8fae8e7e24d unresolved → unresolved unresolved → resolved
coveragepy / c1bfa7352368b63f3a9b30c02f242408d07a7ab2 unresolved → unresolved unresolved → resolved
numpy / 14445500bdf67600f926c6426bad55977441dca0 unresolved → unresolved unresolved → resolved
datalad / f5e1d276ab51aefcf5e48e6f7bd9833b19ef7f90 unresolved → unresolved unresolved → resolved
pyramid / fbbb20c7953370c86f999e865b1a9d682690eb70 unresolved → unresolved unresolved → resolved

The numpy and pyramid new-commit executions have nonzero exit codes but match their full expected maps, as required by the dataset's grading rule. This is a grading replay of actual published execution transcripts, not a fresh model/sandbox rollout or a resolve-rate measurement. It does not measure how often the separate test-list XFAIL change affects benchmark results.

Rollout limitation

Historical author-reported Claude Code evaluation and GRPO use found patched R2E-Gym runs incorrectly unresolved; the grading fixes were subsequently used over several thousand training rollouts.

A fresh bounded model → patch → verifier run now covers one real coveragepy row at this head. The model received the original issue and repository tools, without the dataset patch or published execution logs. Qwen3-8B (b968826d9c46dd6066d109eabc6255188de91218) produced a one-line source change that materializes info before its two iterations. A fresh Apptainer verifier applied that patch: the unpatched image had 6 passed / 1 failed; the model patch had 7 passed. On these same newly generated logs, main f70bf75c3c3eef60484044fc0cd992ff97f12029 grades both unresolved, while this PR grades unpatched unresolved / patched resolved.

Provenance: R2E-Gym-Subset revision 2e8108ff942f24fcb5686badfaf7f9a8808566d5, coveragepy row c1bfa7352368b63f3a9b30c02f242408d07a7ab2, image namanjain12/coveragepy_final:c1bfa7352368b63f3a9b30c02f242408d07a7ab2 at digest sha256:36e1de348436d3f8c11a3637674813e04322cafc4f9356d72c3bba5334a45851. The patch SHA-256 is 885127c984d5f0ea6cb83cd3f489d48d2e70a92539ba8eed0d2af4ae6052384c; verifier-log SHA-256 is 6ebc5762aef12fbff4e6229d9489604ab5f4bfca83b9f7c855ee04cbee614b68.

The agent reached its configured 12-turn limit after producing the patch, so agent_error_kind=max_iteration and mask_sample=true remain set despite resolved=true. Together, the five actual dataset-row log replays and this one fresh model/sandbox row provide before/after grading evidence for the reward-impact smoke request. A broader fresh benchmark evaluation and test-list XFAIL frequency were not measured; these are validation limits, not additional reviewer requirements. The XFAIL-as-pass benchmark-result change is explicitly documented above. Current-head CI and maintainer review remain merge gates.

@copy-pr-bot

copy-pr-bot Bot commented Oct 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33 yaoyu-33 added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working complexity:medium Single-domain change with interacting parts or a moderate review surface needs-review PR is ready for code review and waiting on a reviewer labels Oct 5, 2026
@linj-glitch
linj-glitch marked this pull request as draft October 5, 2026 06:11
@linj-glitch
linj-glitch force-pushed the linj/anyswe-r2e-grading branch from 86e9d27 to 0e3da61 Compare October 8, 2026 00:15
R2E-Gym grading could not mark a task resolved in several common cases:

- Rows with empty FAIL_TO_PASS / PASS_TO_PASS (R2E-Gym-Subset ships them
  empty and carries the verdict in `expected_output_json`) always scored
  0. Grade those rows with R2E-Gym's own `_calculate_reward_r2e` rule: the
  status map from the short test summary must equal `expected_output_json`
  exactly, FAILED/ERROR entries included. Explicit lists still take
  precedence.
- pytest `-rA` prints node ids (`r2e_tests/test_core.py::TestX::test_y`),
  while R2E lists use unittest-style names (`TestX.test_y`). Match on the
  node id, the dir-less node id, `Class.method` and the bare method name.
- R2E images run pytest with color on, so no line started with a status
  word. Some rows also list required tests wrapped in ANSI bold escapes.
  Strip escapes from log lines and test names.
- Also read pytest's verbose `<node id> STATUS [ x%]` lines, for runners
  that replace the short summary, and count XFAIL as passing like
  swebench's grading does.

Persist the grading transcript as `eval_output.txt` next to the rollout
artifacts, so a reward of 0 with a real patch can be told apart from a
runner that printed no per-test results.

Signed-off-by: Lin Jia <linj@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linj-glitch
linj-glitch force-pushed the linj/anyswe-r2e-grading branch from 0e3da61 to 890a022 Compare October 8, 2026 00:20
Merge current main and reject ambiguous test aliases, mismatched class or file requirements, and malformed empty-name expected-output matches. Preserve parameter IDs and failing duplicate statuses; update regression coverage and bytes-based patch grading.

Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch
linj-glitch marked this pull request as ready for review October 13, 2026 23:20
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 14, 2026
@cmunley1

Copy link
Copy Markdown
Contributor

/ok to test fb9bd32

@cmunley1

Copy link
Copy Markdown
Contributor

/ok to test 860cecc

@cmunley1

Copy link
Copy Markdown
Contributor

/review

@linj-glitch

Copy link
Copy Markdown
Collaborator Author

/ok to test 50331c6

…ng R2E test

pytest prints one node id twice when a data-driven case exists in two
files (mypy's suites): once SKIPPED and once PASSED, in either order.
Whichever came second stuck, so a passing patch graded 0. SKIPPED is now
the weakest record for an identity, while a later FAILED or ERROR still
wins over PASSED/XFAIL.

Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch

Copy link
Copy Markdown
Collaborator Author

Pushed 969f85d on top of the rebased branch: one more R2E grading defect found while training on mypy rows.

pytest prints a node id twice when a data-driven case exists in two .test files, once SKIPPED and once PASSED, in either order. With the previous rule whichever record came second stuck, so a passing patch graded 0. SKIPPED is now the weakest record for an identity; a later FAILED/ERROR still wins over PASSED/XFAIL. test_a_duplicate_skipped_record_does_not_hide_a_pass_but_a_failure_still_wins covers both orders and the failure cases. All 117 anyswe tests and pre-commit pass locally.

@linj-glitch

Copy link
Copy Markdown
Collaborator Author

/ok to test 969f85d

Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch

Copy link
Copy Markdown
Collaborator Author

/ok to test 57cff0d

@marcromeyn

Copy link
Copy Markdown
Contributor

Thanks, this is a careful fix with strong regression coverage. A few things before approval:

  1. R2E-Gym rewards change here (stricter name matching, the expected_output_json exact-map rule, and XFAIL now counting as passing in list mode). Could you add a small smoke run on a handful of R2E-Gym rows at this head, showing resolved/unresolved before vs. after, so the reward impact is visible?
  2. Please call out the XFAIL-as-pass change explicitly in the PR description as a benchmark-result change.
  3. eval_output.txt is now written for SWE-bench rollouts as well (up to 200 KB each). Is that intended for training-scale runs, or should it be opt-in?

@cmunley1, since you wrote the original R2E grading in #2011, could you sign off on the identity-matching rules?

@cmunley1 cmunley1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes looks good to me @marcromeyn - exact matching with unique aliases, SKIPPED weakest with FAILED/ERROR winning, XFAIL as pass, and R2E-Gym's exact map for rows. Small comment, if every match of an ambiguous alias has the same status, status_of could return it, since today a Class.method in two test files grades 0 even when both pass. Also could eval_output.txt become opt-in, as 200 KB per rollout is ~400 MB per training step.

@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 26, 2026
@linj-glitch

Copy link
Copy Markdown
Collaborator Author

Addressed the grading and transcript suggestions in d3022c4:

  • A shortened required name now resolves when all matching concrete tests report the same status. Mixed statuses remain ambiguous, including PASSED mixed with XFAIL; file/class identity is still enforced.
  • save_eval_output: true opts into retaining eval_output.txt; the default is false for both R2E-Gym and SWE-bench. The cap is 200,000 characters, and grading always uses the full output.
  • The description now explicitly calls out the benchmark-result change: test-list mode accepts XFAIL, while main requires PASSED. Expected-map mode continues to require exact map equality.

Validation: 128 AnySWE tests passed with source paths checked, plus pre-commit run --all-files. The new regressions failed in 10 cases before the change.

For actual-row evidence, I replayed published old/new commit logs for five distinct repositories from R2E-Gym-Subset revision 2e8108ff942f24fcb5686badfaf7f9a8808566d5, without changing their contents. Compared with main c155b77c, all five old-commit logs remain unresolved; all five new-commit logs change from unresolved to resolved through their expected status maps. The PR description records the exact row commit IDs and selection rule.

This is published execution-log replay, not a fresh agent rollout. The requested current-head model → patch → verifier smoke remains outstanding, and this sample does not measure test-list XFAIL frequency. I am leaving that validation gap open.

@linj-glitch

Copy link
Copy Markdown
Collaborator Author

/ok to test d3022c4

@linj-glitch

linj-glitch commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator Author

Fresh evidence at d3022c4c48d8b20dce059488c0e5867c9b41a134: one real coveragepy R2E row now ran through Qwen3-8B → source patch → a fresh Apptainer verifier. The original issue was the task prompt; no dataset patch or published execution log was supplied to the model. The unpatched image had 6 passed / 1 failed; the model-produced info = list(info) change applied cleanly and yielded 7 passed. Regrading these newly produced logs: main f70bf75c gives unresolved/unresolved, this PR gives unresolved/resolved.

The body records exact model/dataset/image revisions and patch/verifier hashes. The agent hit its 12-turn cap, so the resolved sample correctly retains mask_sample=true. Together, the five actual dataset-row log replays and this one fresh model/sandbox row provide before/after grading evidence for the reward-impact smoke request. The body explicitly calls out XFAIL-as-pass as a benchmark-result change. Broader fresh benchmark evaluation and test-list XFAIL frequency were not measured; these are validation limits, not additional reviewer requirements. Maintainer review remains the approval gate. No code changed for this smoke.

@cmunley1
cmunley1 disabled auto-merge October 29, 2026 05:31
@cmunley1
cmunley1 merged commit 376154d into NVIDIA-NeMo:main Oct 29, 2026
52 of 55 checks passed

This branch was previously deployed

1 inactive deployment
public — d3022c4c Deployed Oct 28, 2026 by copy-pr-bot[bot] via release / finalize / notify #8000
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working complexity:medium Single-domain change with interacting parts or a moderate review surface needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants