Skip to content

swe_gym: pandas and dask images keep a release of the project in the conda package cache; for 411 tasks it already contains the fix, and the current anti-cheat cleanup does not remove it #5744

Open

Description

@Rogerffff

Use cases, pain points, and background

What the current cleanup does

  • resources_servers/swebench/anti_cheat_setup.sh only cleans git: it points HEAD at a new branch, deletes every other ref, expires the reflog, and runs git gc --prune=now. It is called by apply_anti_cheat_setup() in anti_cheat.py, which 12 servers share.
  • The anti_cheat.py module docstring gives the reason: without it, an agent could read the answer with git log --all or git show <future-commit>. A release in the conda package cache that already contains the fix is the same kind of path, but the current cleanup does not touch it.
  • The swe_gym module docstring says "Nothing an agent can read or run in its sandbox reveals the fix or the hidden tests". For the conda package cache in SWE-Gym images, this does not hold.

Scale (swe_gym/data/supported_instance_ids.txt lists 2,278 tasks; we checked 2,201 of them; 53 hydra tasks and 24 bokeh tasks were not checked)

  • The images of all 656 pandas tasks and all 140 dask tasks keep one unpacked release of the project under /opt/miniconda3/pkgs/; none of the other images we checked does. For 418 tasks (pandas 298, dask 120), the release contains fragments of the task's gold patch.
  • For 411 of them, the release contains the task's fix: checked with GitHub compare, the fix PR's merge commit is in the history of the release tag (345 tasks), or the fix came in through a Backport PR #<n> on the release branch (66 tasks). The other 7 (all pandas) do not contain it. The per-task list is in the appendix.
  • How much of the fix is readable: we downloaded the 6 packages with the same names from conda-forge (their file counts and metadata entry counts match the images) and counted, for each task, how many fix lines (lines the gold patch adds) appear verbatim in the file at the same path in the package ("fix lines at the same path" below). We count only lines added to non-test .py files and Cython sources (.pyx, .pxd, .pxi, .pxi.in), and only those that are at least 20 characters long after removing whitespace, are not comments, and are not already in that file at the base commit; a file that was moved or renamed counts as not readable.
    • For 395/411 tasks, at least half of these lines are readable; for 317 of them, all are.
    • Only 1 task (pandas-dev__pandas-55929) has no lines to count under this rule, but its fix is in the cached package as well. The fix deletes one comment line and adds one comment line plus one statement that already exists elsewhere in the same file, so the rule filters all of them out. The function in the cached package is identical, line by line, to the fixed version, so comparing it with the working directory (/testbed) shows the fix.
  • Much of the hidden-test code is in the package too: the cached release ships the project's own test directories (pandas/tests, dask/**/tests). Counting, with the same rule, the lines the hidden-test patch adds to test files: for 378/411 tasks, at least half are readable; for 318 of them, all are.
  • Reproduction (no model involved): we ran the current cleanup exactly as apply_anti_cheat_setup does, with PATH from the swe_gym testbed environment.
    • dask__dask-8801: after the cleanup, /opt/miniconda3/pkgs/dask-core-2024.8.0-*/ and its .conda are still there; dask/config.py at the same path in the cached release contains 12 of the task's 13 fix lines.
    • pandas-dev__pandas-48058: the file at the same path in the cached pandas-1.5.3-* contains 2/2 fix lines.
  • The cached files are mode 0644 and their directories 0755, so any user can read them. With the docker provider, the swe_gym agent runs as root anyway: the SandboxSpec that swe_gym builds sets no user, the docker provider adds --user only when a user is given, and the images set no USER.

The current cleanup is best-effort

  • Per the apply_anti_cheat_setup docstring, a failure is only logged, not raised, and the caller still hands the sandbox to the agent.
  • The open draft PR feat(swe): repository-task servers on the sandbox owner base #5331 adds an anti_cheat_required mode (raise on failure and do not hand out the sandbox), but it judges success only by the exit status, and it is currently enabled only in the swe_rebench config.

Possibly also affected: swe_rebench (we have not run this NeMo server)

  • swe_rebench uses images from nebius/SWE-rebench-V2. We reported the same kind of problem in those images on the dataset's discussion board: https://huggingface.co/datasets/nebius/SWE-rebench-V2/discussions/7
  • The maintainers replied on 2026-11-05: they have published a cleaned version, SWE-rebench V2.1 RL, on Harbor, which fixes issues including newer copies of the project leaking the fix; they are checking the pip cache path for their next update (their reply said next week). We have not checked V2.1 RL.

Precedent in the repository

  • resources_servers/denovo_swe/_denovoswe_clean.sh already removes other copies of the project outside git (the pip cache, and the project directory and metadata in site-packages).

Description:

After the git step of the current cleanup, add a separate cache step that deletes only the project's own packages from the conda package cache. Enable it only for swe_gym at first.

Design:

  1. A new script, run as its own step: add resources_servers/swebench/anti_cheat_cache_setup.sh. In apply_anti_cheat_setup, run it as root with a separate sandbox.exec after the git step.
    • It is not appended to the git script, because the git script returns early with exit 0 when the working directory is missing or is not a git repository, which would skip anything appended; the cache step itself does not depend on git.
  2. What it deletes:
    • First, it finds the project's package names: in the working directory (/testbed) and in its src/, it takes the immediate subdirectories that have an __init__.py and drops names such as tests, docs and benchmarks; the remaining directory names are the package names (for example dask, pandas). This step only reads; nothing in the working directory is deleted.
    • Then, in pkgs/ of each conda installation (/opt/miniconda3, /opt/conda, etc., resolved to real paths), it deletes the cached packages that contain site-packages/<package name>/__init__.py: the unpacked directory, together with its .conda and .tar.bz2.
  3. What it does not delete:
    • It does not run conda clean --all. First, that command would not remove these unpacked packages: conda 23.11.0 keeps a package's unpacked directory when any of its files has more than one hard link (conda/cli/main_clean.py:116), and in these images the project files have a link count of 2 although only one path to them can be found on the file system, most likely because the environment's copy was deleted in a later image layer. Second, it would delete the archives of every other package and the index cache, which an agent would need to install dependencies from the cache when the sandbox has no network access.
  4. Output and exit status: it prints one line,
    CACHE_CLEANUP conda_roots=... project=... removed=<n> project_copies_left=<n> elapsed_s=<n>.
    • Exit 0: there is no conda installation, or the project was identified and nothing is left after deletion.
    • Exit 1: anything else, including a conda package cache when the project cannot be identified (project=unknown; nothing is deleted in that case).
    • So it can feed the anti_cheat_required mode in feat(swe): repository-task servers on the sandbox owner base #5331 directly; without that mode, like the git step, it logs only on failure.
  5. Switch: we have only validated SWE-Gym images, so we suggest enabling it only for swe_gym at first: add a parameter to apply_anti_cheat_setup that is off by default (for example clean_conda_pkg_cache: bool = False), have swe_gym pass True, and leave the other 11 servers unchanged. On other images, if the project cannot be identified, the cache step exits 1 and each seed logs one more failure; those servers can turn it on after validating it on their own images.

Validation

  • 4 tasks: for each task we started fresh containers from the original image, in two arms: the current cleanup only; the current cleanup plus the cache step. We then scanned each container for fix lines at the same path, and evaluated the gold patch and an empty patch with the SWE-bench procedure.

    Task Current cleanup only: fix lines at the same path After the cache step Gold patch / empty patch, both arms
    dask-8801 12/13 0/13 pass / fail
    pandas-48058 2/2 0/2 pass / fail
    pandas-47841 2/2 0/2 pass / fail
    pandas-47804 0/5 (in the cached release the directory of the fixed file was renamed from core/exchange/ to core/interchange/, so the same-path count finds nothing) 0/5 (copy deleted) pass / fail
    • On all 4 tasks, the cache step deleted 2 items (the unpacked directory and the .conda), reported project_copies_left=0, and exited 0.
    • The F2P/P2P pass counts are identical in both arms.
    • A read-only check shows no project copy left in the conda package cache after the cache step; the pip cache is still there and contains only dependencies.
  • Timing: we started 3 more fresh containers per task and arm and timed only the cleanup steps (wall-clock time of docker exec on the host, with the images already local).

    Task Git step Cache step
    dask-8801 0.89–0.91 s 0.32 s
    pandas (3 tasks) 4.65–4.71 s 0.63–0.64 s
  • No wrong deletions: for one task from each of the other 9 repositories (MONAI, modin, moto, mypy, dvc, pydantic, conan, hydra, bokeh), the cache step identified each project's packages (for hydra and bokeh it also picked up a helper directory; see the next item). These images have no project copy in the conda package cache; the cache step deleted nothing (the pkgs/ listing is identical before and after), exited 0, and took 0.07–0.13 s.

  • Package name identification: the rule finds all of the project's packages for all 11 repositories in the task list; for 2 of them it also picks up one helper directory, which caused no wrong deletion in our runs.

  • NeMo server end to end: we ran swe_gym locally with the docker provider (configs unchanged), with anti_cheat.py replaced by a version that runs the cache step unconditionally after the git step (on swe_gym this is the same as the switch being on). Tasks: dask-8801, pandas-47804, pandas-53958.

    • After every seed, the cache step had deleted 2 items, with project_copies_left=0.
    • Fix lines at the same path in the agent sandbox: dask-8801 from 12/13 to 0/13, pandas-53958 from 2/2 to 0/2.
    • /verify gives reward 1 for the gold patch and 0 for an empty patch, with the same F2P/P2P counts as without the change.

Out of scope:

  • The grading sandbox: grading already happens in a separate, uncleaned sandbox, and this proposal does not change it.
  • The other 11 servers: we have not validated their images; the switch is off by default (see Design item 5).
  • The pip cache: in the 14 swe_gym images we ran, the pip cache, where there is one, holds only dependencies, not the project.
  • Other installs of the project in an image (for example in another Python environment's site-packages): when the project is itself a tool such as conda or setuptools, that install is the copy the environment runs on, and deleting it at runtime would break the environment.
  • Changes to the git step itself, the network as a way to get the fix, and other problems in the images; whether anti_cheat_required should be on by default is for the maintainers to decide.

Acceptance Criteria:

  • After the swe_gym cleanup, the conda package cache has no project copy: the cache step reports project_copies_left=0, and the fix lines at the same path drop to 0.
  • The cache step exits 1 when a copy is left or the project cannot be identified.
  • The gold patch still passes, an empty patch still fails, and the reward does not change.
  • Servers that do not turn on the switch behave as before.

Questions for the maintainers

  1. Enable it per server (our proposal), or run it by default for every server that calls apply_anti_cheat_setup?
  2. Should the cache step's exit status count toward the anti_cheat_required mode in feat(swe): repository-task servers on the sandbox owner base #5331?
  3. Does any server need the project's own cached package at runtime?

If the direction looks right, we will open a PR with the script, the switch in anti_cheat.py, the corresponding update to resources_servers/swebench/tests/test_anti_cheat.py, and, per the contribution guidelines, one representative model rollout.

Appendix: affected tasks in swe_gym/data/supported_instance_ids.txt (418)

Each image keeps an unpacked release of the project under /opt/miniconda3/pkgs/ that contains fragments of the task's gold patch. "Contains the fix" means the fix PR's merge commit, or its backport PR on the release branch, is in the release tag's history (GitHub compare). Task IDs are pandas-dev__pandas-<n> and dask__dask-<n>; the numbers below are <n>.

pandas 2.2.2 (tag v2.2.2; 212 tasks)

  • contains the fix, merge commit in tag history (173): 50056, 52042, 52532, 52788, 52952, 52992, 53044, 53049, 53060, 53102, 53118, 53122, 53161, 53189, 53194, 53207, 53213, 53215, 53216, 53220, 53237, 53272, 53288, 53296, 53298, 53301, 53344, 53360, 53391, 53397, 53418, 53419, 53443, 53458, 53494, 53495, 53505, 53519, 53520, 53571, 53607, 53641, 53651, 53652, 53654, 53671, 53697, 53710, 53736, 53764, 53769, 53787, 53805, 53817, 53830, 53844, 53855, 53856, 53877, 53958, 53962, 53975, 54001, 54002, 54015, 54025, 54027, 54068, 54074, 54087, 54098, 54099, 54115, 54129, 54130, 54133, 54159, 54166, 54170, 54179, 54186, 54189, 54226, 54234, 54246, 54272, 54306, 54341, 54372, 54428, 54451, 54460, 54643, 55058, 55077, 55079, 55115, 55173, 55227, 55249, 55257, 55258, 55259, 55270, 55283, 55314, 55340, 55352, 55370, 55439, 55448, 55534, 55568, 55580, 55586, 55633, 55678, 55690, 55719, 55734, 55754, 55761, 55768, 55777, 55807, 55841, 55868, 55895, 55927, 55929, 55933, 55963, 55968, 56016, 56053, 56054, 56055, 56120, 56149, 56194, 56220, 56228, 56237, 56244, 56250, 56290, 56291, 56294, 56321, 56332, 56334, 56341, 56350, 56352, 56365, 56368, 56371, 56412, 56421, 56426, 56427, 56459, 56487, 56488, 56490, 56522, 56523, 56531, 56538, 56558, 56580, 56585, 57485
  • contains the fix via a backport PR (38): 56059, 56167, 56370, 56613, 56616, 56647, 56650, 56677, 56757, 56766, 56767, 56769, 56771, 56772, 56788, 56906, 57018, 57058, 57089, 57101, 57102, 57139, 57157, 57169, 57173, 57174, 57175, 57250, 57323, 57329, 57402, 57439, 57474, 57488, 57553, 57665, 57758, 57764
  • does not contain the fix (1): 57375

pandas 1.5.3 (tag v1.5.3; 86 tasks)

  • contains the fix, merge commit in tag history (52): 47157, 47222, 47231, 47273, 47361, 47371, 47446, 47451, 47493, 47528, 47547, 47550, 47557, 47581, 47586, 47605, 47619, 47623, 47633, 47643, 47716, 47731, 47747, 47754, 47757, 47761, 47771, 47780, 47804, 47810, 47840, 47841, 47848, 47860, 47881, 47894, 47927, 47972, 47987, 48008, 48057, 48058, 48073, 48094, 48147, 48149, 48164, 48167, 48169, 48176, 49453, 50627
  • contains the fix via a backport PR (28): 48334, 48555, 48579, 48587, 48619, 48686, 48693, 48696, 48697, 48711, 48713, 48734, 48736, 48760, 48782, 48797, 48820, 48824, 48890, 48995, 49080, 49284, 49377, 49579, 49610, 49864, 50682, 50685
  • does not contain the fix (6): 49967, 50331, 51042, 51960, 51976, 52459

dask 2024.8.1 (tag 2024.8.1; 62 tasks)

  • contains the fix, merge commit in tag history (62): 7688, 8865, 8895, 8920, 8925, 8945, 8954, 9027, 9087, 9240, 9250, 9272, 9285, 9342, 9345, 9349, 9350, 9378, 9531, 9627, 9653, 9739, 9760, 9885, 9981, 10009, 10027, 10042, 10054, 10064, 10128, 10149, 10159, 10211, 10212, 10294, 10342, 10380, 10395, 10404, 10422, 10441, 10454, 10506, 10509, 10521, 10556, 10616, 10695, 10734, 10750, 10754, 10765, 10785, 10796, 10797, 10803, 10827, 10846, 10921, 10972, 11023

dask 2024.8.0 (tag 2024.8.0; 30 tasks)

  • contains the fix, merge commit in tag history (30): 7636, 7656, 7894, 7973, 8040, 8106, 8166, 8185, 8253, 8261, 8283, 8316, 8375, 8462, 8501, 8525, 8538, 8590, 8601, 8610, 8686, 8687, 8689, 8703, 8785, 8792, 8801, 8805, 8820, 8860

dask 2021.7.2 (tag 2021.07.2; 26 tasks)

  • contains the fix, merge commit in tag history (26): 6608, 6626, 6632, 6660, 6682, 6742, 6749, 6776, 6779, 6801, 6809, 6815, 6854, 6862, 6887, 6893, 6939, 6960, 6978, 6980, 6992, 7056, 7073, 7092, 7138, 7145

dask 2022.11.0 (tag 2022.11.0; 2 tasks)

  • contains the fix, merge commit in tag history (2): 7125, 7230
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    community-requestIssue reported or requested by someone from the communitywaiting-on-maintainersWaiting on maintainers to respond

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions