Skip to content

fix(swebench): preserve per-test grading across pytest formats - #4475

Closed
giuliolovisotto wants to merge 1 commit into
mainfrom
glovisotto/swebench-grading-compat
Closed

giuliolovisotto wants to merge 1 commit into
mainfrom
glovisotto/swebench-grading-compat

Conversation

@giuliolovisotto

Copy link
Copy Markdown
Contributor

SWE-bench Verified can mark passing patches unresolved when the test runner's output differs from the expected format. This fixes two reproduced cases:

  • For Sphinx, pass pytest's -rA through tox so the grader receives named pass/fail results even when tox.ini lacks the image-time reporting edit. Tox's own -v does not enable pytest reporting. Only the generated test command changes; the embedded test patch remains intact.
  • For astropy__astropy-7606, match the expected test_compose_roundtrip[] to the observed test_compose_roundtrip[unit0] only when the original ID is absent. Preserve the observed status, raw log, and caller's test specification. Apply the same correction to live verification and rewrite_reports.

Missing tests, failed tests, invalid logs and unrelated instances retain their existing grading behavior. No migration is required. Scores can increase for affected passing patches: archived Astropy output can be regraded; Sphinx output without named results requires a verifier replay. No pass is inferred from dot output or an aggregate pass count.

No separate issue is needed for these focused, reproduced verifier defects.

Validation:

  • gym env test --resources-server swebench +should_validate_data=true ++skip_venv_if_present=true: 45 passed; example-data validation passed.
  • 17 grading regression cases, including actual pytest subprocess output, missing/failing tests, conflicting IDs, invalid logs, live verification and report rewriting. Disabling both fixes makes five regression cases fail.
  • pre-commit run --all-files and git diff --cached --check: passed.
  • Saved model-output replay: Astropy's known missing-ID false zeros become resolved; the saved Sphinx log remains unresolved. Re-executing a saved Sphinx model patch in a fresh SWE-bench image through Gym's verifier completes and resolves with named test results.
  • Fresh model smoke validation is in progress; retaining draft status until it completes.

Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
@giuliolovisotto giuliolovisotto added bug Something isn't working area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers labels Oct 12, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 12, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

khazic pushed a commit to khazic/Gym-lao that referenced this pull request Oct 15, 2026
## Summary
- Add the SA-V tracking verifier and its server configuration.
- Declare the SA-V environment servers used by video RL.
- Add portable five-row example data, schema, metrics, and checked
rollout artifacts.

## Why
This is the Gym-side verifier foundation for mixed SA-V tracking and
CapRL video training in NeMo-RL. No separate issue is needed; this is
part of the Super 3.5 video RL upstreaming stack.

## Validation
- Runtime evidence: the [healthy 16-node async `r18` W&B
run](https://wandb.ai/adlr/Nemotron-omni-RL-debug/runs/oci-hsg-cs-001_n3post_main_1d76d243b_vllm029_super35_video_step120_async16n_trainmb45056_expandable_uncapped_r18_20261003)
shows repeated GRPO cycles, stable TMPE, and increasing reward on the
equivalent video implementation.
- `gym dataset collate ... +mode=example_validation`: 5 examples;
generated `data/example_metrics.json`.
- Focused validation: 25 verifier tests and all 5 example rows against
`task_data.py` passed.
- Ruff format and lint passed for the changed Python files.
- Both Gym stack ranges pass `git diff --check`.

## Rollouts
`data/example_rollouts.jsonl` contains one checked rollout for each of
the five example rows. Representative task checks:

- `pt`:
`<answer>{"tracks":[{"point":1,"frame":1,"xy":[500,500],"visible":true}]}</answer>`
— `format_ok=true`, reward `1.0`, visible-frame mask membership `true`.
- `box`:
`<answer>{"tracks":[{"box":1,"frame":1,"bbox":[250,250,750,750],"visible":true}]}</answer>`
— `format_ok=true`, reward `1.0`, frame IoU `1.0`.
- `ref`:
`<answer>{"tracks":[{"id":7,"frame":1,"xy":[500,500]}]}</answer>` —
`format_ok=true`, reward `1.0`, predicted id `7` assigned to the GT
object and mask membership `true`.

The artifact also contains visible/invisible `pt` coverage and an
all-absent `ref` negative case, preserving the verifier's current
unmatched-ID behavior.

## Compatibility
Adds a new verifier/configuration and does not change unrelated
environments. Example media is embedded as a small inline JPEG, so no
employee or cluster-local paths are required.

## Stack
Gym PR 1 of 2. Consolidates and supersedes the implementation scope of
NVIDIA-NeMo#3504 after review history is preserved.

## Consolidated stack links

### Gym
1. [Gym NVIDIA-NeMo#3999 — SA-V tracking
verifier](NVIDIA-NeMo#3999)
2. [Gym NVIDIA-NeMo#4000 — R3 routing replay and rollout
identity](NVIDIA-NeMo#4000)

### NeMo-RL
1. [NeMo-RL NVIDIA-NeMo#4472 — Super video data and model
loading](NVIDIA-NeMo/RL#4472)
2. [NeMo-RL NVIDIA-NeMo#4473 — RL/MOPD video frame
contract](NVIDIA-NeMo/RL#4473)
3. [NeMo-RL NVIDIA-NeMo#4474 — sync/async Gym routing
runtime](NVIDIA-NeMo/RL#4474)
4. [NeMo-RL NVIDIA-NeMo#4475 — SA-V pin and CI
fixtures](NVIDIA-NeMo/RL#4475)
5. [NeMo-RL NVIDIA-NeMo#4476 — final dependency
pins](NVIDIA-NeMo/RL#4476)

### Child dependency refs
- [MCore integration
ref](https://github.com/ehosseiniasl/Megatron-LM/tree/ehsan/video-r3-mtp-gradient-integration-20261004),
adapting [MCore #7278](NVIDIA/Megatron-LM#7278)
and [MCore #7279](NVIDIA/Megatron-LM#7279) to
the pinned base.
- [MBridge integration
ref](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/ehsan/video-r3-mcore-pin-integration-20261004).

---------

Signed-off-by: ehosseiniasl <ehsan.hosseiniasl@gmail.com>
Co-authored-by: Pulkit Kumar <pulkitk@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant