Repository navigation
fix(anyswe_agent): make R2E-Gym grading resolve - #4019
Conversation
86e9d27 to
0e3da61
Compare
R2E-Gym grading could not mark a task resolved in several common cases: - Rows with empty FAIL_TO_PASS / PASS_TO_PASS (R2E-Gym-Subset ships them empty and carries the verdict in `expected_output_json`) always scored 0. Grade those rows with R2E-Gym's own `_calculate_reward_r2e` rule: the status map from the short test summary must equal `expected_output_json` exactly, FAILED/ERROR entries included. Explicit lists still take precedence. - pytest `-rA` prints node ids (`r2e_tests/test_core.py::TestX::test_y`), while R2E lists use unittest-style names (`TestX.test_y`). Match on the node id, the dir-less node id, `Class.method` and the bare method name. - R2E images run pytest with color on, so no line started with a status word. Some rows also list required tests wrapped in ANSI bold escapes. Strip escapes from log lines and test names. - Also read pytest's verbose `<node id> STATUS [ x%]` lines, for runners that replace the short summary, and count XFAIL as passing like swebench's grading does. Persist the grading transcript as `eval_output.txt` next to the rollout artifacts, so a reward of 0 with a real patch can be told apart from a runner that printed no per-test results. Signed-off-by: Lin Jia <linj@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0e3da61 to
890a022
Compare
Merge current main and reject ambiguous test aliases, mismatched class or file requirements, and malformed empty-name expected-output matches. Preserve parameter IDs and failing duplicate statuses; update regression coverage and bytes-based patch grading. Signed-off-by: Lin Jia <linj@nvidia.com>
|
/ok to test fb9bd32 |
|
/ok to test 860cecc |
|
/review |
Signed-off-by: Lin Jia <linj@nvidia.com>
|
/ok to test 50331c6 |
…ng R2E test pytest prints one node id twice when a data-driven case exists in two files (mypy's suites): once SKIPPED and once PASSED, in either order. Whichever came second stuck, so a passing patch graded 0. SKIPPED is now the weakest record for an identity, while a later FAILED or ERROR still wins over PASSED/XFAIL. Signed-off-by: Lin Jia <linj@nvidia.com>
|
Pushed 969f85d on top of the rebased branch: one more R2E grading defect found while training on mypy rows. pytest prints a node id twice when a data-driven case exists in two |
|
/ok to test 969f85d |
Signed-off-by: Lin Jia <linj@nvidia.com>
|
/ok to test 57cff0d |
|
Thanks, this is a careful fix with strong regression coverage. A few things before approval:
@cmunley1, since you wrote the original R2E grading in #2011, could you sign off on the identity-matching rules? |
cmunley1
left a comment
There was a problem hiding this comment.
Yes looks good to me @marcromeyn - exact matching with unique aliases, SKIPPED weakest with FAILED/ERROR winning, XFAIL as pass, and R2E-Gym's exact map for rows. Small comment, if every match of an ambiguous alias has the same status, status_of could return it, since today a Class.method in two test files grades 0 even when both pass. Also could eval_output.txt become opt-in, as 200 KB per rollout is ~400 MB per training step.
Signed-off-by: Lin Jia <linj@nvidia.com>
|
Addressed the grading and transcript suggestions in d3022c4:
Validation: 128 AnySWE tests passed with source paths checked, plus For actual-row evidence, I replayed published old/new commit logs for five distinct repositories from R2E-Gym-Subset revision This is published execution-log replay, not a fresh agent rollout. The requested current-head model → patch → verifier smoke remains outstanding, and this sample does not measure test-list |
|
/ok to test d3022c4 |
|
Fresh evidence at The body records exact model/dataset/image revisions and patch/verifier hashes. The agent hit its 12-turn cap, so the resolved sample correctly retains |
What and why
Fix R2E-Gym grading for colored pytest output, unittest-style names, verbose results, and rows using
expected_output_json. Previously, patched runs could be incorrectly unresolved; ambiguous test names or status words inside failures could also produce incorrect rewards. No separate issue is needed for these focused reproductions.Explicit test lists preserve the requested class/file identity and parameter IDs. A shortened name can match one concrete test or multiple tests with the same status; mixed statuses remain ambiguous, including
PASSEDmixed withXFAIL. A duplicateSKIPPEDrecord cannot overwrite a passing result;FAILEDandERRORdominate. Failure diagnostics match complete observed names before removing suffixes, including teardown errors after passing bodies with arbitrary parameter IDs.Rows without test lists require a nonempty exact comparison with
expected_output_json. Parsing uses the leading summary status and consistent ANSI normalization. Expected maps retain the upstream key encoding and final-phase behavior, including expected teardown errors; identities already lost by that legacy encoding cannot be recovered here. Main's byte-safe patch upload and merged #4030 hidden-test linking are preserved.Compatibility and benchmark impact
XFAILas passing, whereas main requiresPASSED. This is an intentional reward/benchmark-result change, endorsed in the author review. Matching shortened names with unanimous statuses can also change unresolved results to resolved; mixed statuses still fail closed.FAILED/ERRORoutcomes. It does not apply test-list mode'sXFAILrule.save_eval_outputdefaults tofalse. Set it totrueto retain the last 200,000 characters ineval_output.txtfor either R2E-Gym or SWE-bench. Both verifiers always receive the full log. SWE-bench reward semantics are unchanged.Validation
Validated head: d3022c4. Python 3.13.14; imports were asserted to resolve to the reviewed checkout.
python -m pytest -q -o addopts= --disable-warnings responses_api_agents/anyswe_agent/tests: 128 passed.pre-commit run --all-files: passed. Complete PR diff, description, and reply guardword checks and diff whitespace checks passed. Additive commits carry DCO sign-off.Actual-row grading replay
Regraded the unmodified published old/new commit execution logs from R2E-Gym/R2E-Gym-Subset. Selection was the first eligible row for each of five distinct repositories in sorted cached-shard/stored-row order, without filtering by verdict. Baseline: main
c155b77c8420283982ec30347a46b92ae6ee1af5. Each row has an expected map and no test lists.2d9617bd0cb1f0ba61771258410ab8fae8e7e24dc1bfa7352368b63f3a9b30c02f242408d07a7ab214445500bdf67600f926c6426bad55977441dca0f5e1d276ab51aefcf5e48e6f7bd9833b19ef7f90fbbb20c7953370c86f999e865b1a9d682690eb70The numpy and pyramid new-commit executions have nonzero exit codes but match their full expected maps, as required by the dataset's grading rule. This is a grading replay of actual published execution transcripts, not a fresh model/sandbox rollout or a resolve-rate measurement. It does not measure how often the separate test-list
XFAILchange affects benchmark results.Rollout limitation
Historical author-reported Claude Code evaluation and GRPO use found patched R2E-Gym runs incorrectly unresolved; the grading fixes were subsequently used over several thousand training rollouts.
A fresh bounded model → patch → verifier run now covers one real coveragepy row at this head. The model received the original issue and repository tools, without the dataset patch or published execution logs. Qwen3-8B (
b968826d9c46dd6066d109eabc6255188de91218) produced a one-line source change that materializesinfobefore its two iterations. A fresh Apptainer verifier applied that patch: the unpatched image had 6 passed / 1 failed; the model patch had 7 passed. On these same newly generated logs, mainf70bf75c3c3eef60484044fc0cd992ff97f12029grades both unresolved, while this PR grades unpatched unresolved / patched resolved.Provenance: R2E-Gym-Subset revision
2e8108ff942f24fcb5686badfaf7f9a8808566d5, coveragepy rowc1bfa7352368b63f3a9b30c02f242408d07a7ab2, imagenamanjain12/coveragepy_final:c1bfa7352368b63f3a9b30c02f242408d07a7ab2at digestsha256:36e1de348436d3f8c11a3637674813e04322cafc4f9356d72c3bba5334a45851. The patch SHA-256 is885127c984d5f0ea6cb83cd3f489d48d2e70a92539ba8eed0d2af4ae6052384c; verifier-log SHA-256 is6ebc5762aef12fbff4e6229d9489604ab5f4bfca83b9f7c855ee04cbee614b68.The agent reached its configured 12-turn limit after producing the patch, so
agent_error_kind=max_iterationandmask_sample=trueremain set despiteresolved=true. Together, the five actual dataset-row log replays and this one fresh model/sandbox row provide before/after grading evidence for the reward-impact smoke request. A broader fresh benchmark evaluation and test-list XFAIL frequency were not measured; these are validation limits, not additional reviewer requirements. The XFAIL-as-pass benchmark-result change is explicitly documented above. Current-head CI and maintainer review remain merge gates.