Repository navigation
Description
Use cases, pain points, and background
What the current cleanup does
resources_servers/swebench/anti_cheat_setup.shonly cleans git: it points HEAD at a new branch, deletes every other ref, expires the reflog, and runsgit gc --prune=now. It is called byapply_anti_cheat_setup()inanti_cheat.py, which 12 servers share.- The
anti_cheat.pymodule docstring gives the reason: without it, an agent could read the answer withgit log --allorgit show <future-commit>. A release in the conda package cache that already contains the fix is the same kind of path, but the current cleanup does not touch it. - The
swe_gymmodule docstring says "Nothing an agent can read or run in its sandbox reveals the fix or the hidden tests". For the conda package cache in SWE-Gym images, this does not hold.
Scale (swe_gym/data/supported_instance_ids.txt lists 2,278 tasks; we checked 2,201 of them; 53 hydra tasks and 24 bokeh tasks were not checked)
- The images of all 656 pandas tasks and all 140 dask tasks keep one unpacked release of the project under
/opt/miniconda3/pkgs/; none of the other images we checked does. For 418 tasks (pandas 298, dask 120), the release contains fragments of the task's gold patch. - For 411 of them, the release contains the task's fix: checked with GitHub compare, the fix PR's merge commit is in the history of the release tag (345 tasks), or the fix came in through a
Backport PR #<n>on the release branch (66 tasks). The other 7 (all pandas) do not contain it. The per-task list is in the appendix. - How much of the fix is readable: we downloaded the 6 packages with the same names from conda-forge (their file counts and metadata entry counts match the images) and counted, for each task, how many fix lines (lines the gold patch adds) appear verbatim in the file at the same path in the package ("fix lines at the same path" below). We count only lines added to non-test
.pyfiles and Cython sources (.pyx,.pxd,.pxi,.pxi.in), and only those that are at least 20 characters long after removing whitespace, are not comments, and are not already in that file at the base commit; a file that was moved or renamed counts as not readable.- For 395/411 tasks, at least half of these lines are readable; for 317 of them, all are.
- Only 1 task (
pandas-dev__pandas-55929) has no lines to count under this rule, but its fix is in the cached package as well. The fix deletes one comment line and adds one comment line plus one statement that already exists elsewhere in the same file, so the rule filters all of them out. The function in the cached package is identical, line by line, to the fixed version, so comparing it with the working directory (/testbed) shows the fix.
- Much of the hidden-test code is in the package too: the cached release ships the project's own test directories (
pandas/tests,dask/**/tests). Counting, with the same rule, the lines the hidden-test patch adds to test files: for 378/411 tasks, at least half are readable; for 318 of them, all are. - Reproduction (no model involved): we ran the current cleanup exactly as
apply_anti_cheat_setupdoes, with PATH from the swe_gym testbed environment.dask__dask-8801: after the cleanup,/opt/miniconda3/pkgs/dask-core-2024.8.0-*/and its.condaare still there;dask/config.pyat the same path in the cached release contains 12 of the task's 13 fix lines.pandas-dev__pandas-48058: the file at the same path in the cachedpandas-1.5.3-*contains 2/2 fix lines.
- The cached files are mode 0644 and their directories 0755, so any user can read them. With the docker provider, the
swe_gymagent runs as root anyway: theSandboxSpecthatswe_gymbuilds sets nouser, the docker provider adds--useronly when a user is given, and the images set noUSER.
The current cleanup is best-effort
- Per the
apply_anti_cheat_setupdocstring, a failure is only logged, not raised, and the caller still hands the sandbox to the agent. - The open draft PR feat(swe): repository-task servers on the sandbox owner base #5331 adds an
anti_cheat_requiredmode (raise on failure and do not hand out the sandbox), but it judges success only by the exit status, and it is currently enabled only in theswe_rebenchconfig.
Possibly also affected: swe_rebench (we have not run this NeMo server)
swe_rebenchuses images fromnebius/SWE-rebench-V2. We reported the same kind of problem in those images on the dataset's discussion board: https://huggingface.co/datasets/nebius/SWE-rebench-V2/discussions/7- The maintainers replied on 2026-11-05: they have published a cleaned version, SWE-rebench V2.1 RL, on Harbor, which fixes issues including newer copies of the project leaking the fix; they are checking the pip cache path for their next update (their reply said next week). We have not checked V2.1 RL.
Precedent in the repository
resources_servers/denovo_swe/_denovoswe_clean.shalready removes other copies of the project outside git (the pip cache, and the project directory and metadata insite-packages).
Description:
After the git step of the current cleanup, add a separate cache step that deletes only the project's own packages from the conda package cache. Enable it only for swe_gym at first.
Design:
- A new script, run as its own step: add
resources_servers/swebench/anti_cheat_cache_setup.sh. Inapply_anti_cheat_setup, run it as root with a separatesandbox.execafter the git step.- It is not appended to the git script, because the git script returns early with
exit 0when the working directory is missing or is not a git repository, which would skip anything appended; the cache step itself does not depend on git.
- It is not appended to the git script, because the git script returns early with
- What it deletes:
- First, it finds the project's package names: in the working directory (
/testbed) and in itssrc/, it takes the immediate subdirectories that have an__init__.pyand drops names such astests,docsandbenchmarks; the remaining directory names are the package names (for exampledask,pandas). This step only reads; nothing in the working directory is deleted. - Then, in
pkgs/of each conda installation (/opt/miniconda3,/opt/conda, etc., resolved to real paths), it deletes the cached packages that containsite-packages/<package name>/__init__.py: the unpacked directory, together with its.condaand.tar.bz2.
- First, it finds the project's package names: in the working directory (
- What it does not delete:
- It does not run
conda clean --all. First, that command would not remove these unpacked packages: conda 23.11.0 keeps a package's unpacked directory when any of its files has more than one hard link (conda/cli/main_clean.py:116), and in these images the project files have a link count of 2 although only one path to them can be found on the file system, most likely because the environment's copy was deleted in a later image layer. Second, it would delete the archives of every other package and the index cache, which an agent would need to install dependencies from the cache when the sandbox has no network access.
- It does not run
- Output and exit status: it prints one line,
CACHE_CLEANUP conda_roots=... project=... removed=<n> project_copies_left=<n> elapsed_s=<n>.- Exit 0: there is no conda installation, or the project was identified and nothing is left after deletion.
- Exit 1: anything else, including a conda package cache when the project cannot be identified (
project=unknown; nothing is deleted in that case). - So it can feed the
anti_cheat_requiredmode in feat(swe): repository-task servers on the sandbox owner base #5331 directly; without that mode, like the git step, it logs only on failure.
- Switch: we have only validated SWE-Gym images, so we suggest enabling it only for
swe_gymat first: add a parameter toapply_anti_cheat_setupthat is off by default (for exampleclean_conda_pkg_cache: bool = False), haveswe_gympassTrue, and leave the other 11 servers unchanged. On other images, if the project cannot be identified, the cache step exits 1 and each seed logs one more failure; those servers can turn it on after validating it on their own images.
Validation
-
4 tasks: for each task we started fresh containers from the original image, in two arms: the current cleanup only; the current cleanup plus the cache step. We then scanned each container for fix lines at the same path, and evaluated the gold patch and an empty patch with the SWE-bench procedure.
Task Current cleanup only: fix lines at the same path After the cache step Gold patch / empty patch, both arms dask-8801 12/13 0/13 pass / fail pandas-48058 2/2 0/2 pass / fail pandas-47841 2/2 0/2 pass / fail pandas-47804 0/5 (in the cached release the directory of the fixed file was renamed from core/exchange/tocore/interchange/, so the same-path count finds nothing)0/5 (copy deleted) pass / fail - On all 4 tasks, the cache step deleted 2 items (the unpacked directory and the
.conda), reportedproject_copies_left=0, and exited 0. - The F2P/P2P pass counts are identical in both arms.
- A read-only check shows no project copy left in the conda package cache after the cache step; the pip cache is still there and contains only dependencies.
- On all 4 tasks, the cache step deleted 2 items (the unpacked directory and the
-
Timing: we started 3 more fresh containers per task and arm and timed only the cleanup steps (wall-clock time of
docker execon the host, with the images already local).Task Git step Cache step dask-8801 0.89–0.91 s 0.32 s pandas (3 tasks) 4.65–4.71 s 0.63–0.64 s -
No wrong deletions: for one task from each of the other 9 repositories (MONAI, modin, moto, mypy, dvc, pydantic, conan, hydra, bokeh), the cache step identified each project's packages (for hydra and bokeh it also picked up a helper directory; see the next item). These images have no project copy in the conda package cache; the cache step deleted nothing (the
pkgs/listing is identical before and after), exited 0, and took 0.07–0.13 s. -
Package name identification: the rule finds all of the project's packages for all 11 repositories in the task list; for 2 of them it also picks up one helper directory, which caused no wrong deletion in our runs.
-
NeMo server end to end: we ran
swe_gymlocally with the docker provider (configs unchanged), withanti_cheat.pyreplaced by a version that runs the cache step unconditionally after the git step (on swe_gym this is the same as the switch being on). Tasks: dask-8801, pandas-47804, pandas-53958.- After every seed, the cache step had deleted 2 items, with
project_copies_left=0. - Fix lines at the same path in the agent sandbox: dask-8801 from 12/13 to 0/13, pandas-53958 from 2/2 to 0/2.
/verifygives reward 1 for the gold patch and 0 for an empty patch, with the same F2P/P2P counts as without the change.
- After every seed, the cache step had deleted 2 items, with
Out of scope:
- The grading sandbox: grading already happens in a separate, uncleaned sandbox, and this proposal does not change it.
- The other 11 servers: we have not validated their images; the switch is off by default (see Design item 5).
- The pip cache: in the 14 swe_gym images we ran, the pip cache, where there is one, holds only dependencies, not the project.
- Other installs of the project in an image (for example in another Python environment's
site-packages): when the project is itself a tool such as conda or setuptools, that install is the copy the environment runs on, and deleting it at runtime would break the environment. - Changes to the git step itself, the network as a way to get the fix, and other problems in the images; whether
anti_cheat_requiredshould be on by default is for the maintainers to decide.
Acceptance Criteria:
- After the
swe_gymcleanup, the conda package cache has no project copy: the cache step reportsproject_copies_left=0, and the fix lines at the same path drop to 0. - The cache step exits 1 when a copy is left or the project cannot be identified.
- The gold patch still passes, an empty patch still fails, and the reward does not change.
- Servers that do not turn on the switch behave as before.
Questions for the maintainers
- Enable it per server (our proposal), or run it by default for every server that calls
apply_anti_cheat_setup? - Should the cache step's exit status count toward the
anti_cheat_requiredmode in feat(swe): repository-task servers on the sandbox owner base #5331? - Does any server need the project's own cached package at runtime?
If the direction looks right, we will open a PR with the script, the switch in anti_cheat.py, the corresponding update to resources_servers/swebench/tests/test_anti_cheat.py, and, per the contribution guidelines, one representative model rollout.
Appendix: affected tasks in swe_gym/data/supported_instance_ids.txt (418)
Each image keeps an unpacked release of the project under /opt/miniconda3/pkgs/ that contains fragments of the task's gold patch. "Contains the fix" means the fix PR's merge commit, or its backport PR on the release branch, is in the release tag's history (GitHub compare). Task IDs are pandas-dev__pandas-<n> and dask__dask-<n>; the numbers below are <n>.
pandas 2.2.2 (tag v2.2.2; 212 tasks)
- contains the fix, merge commit in tag history (173): 50056, 52042, 52532, 52788, 52952, 52992, 53044, 53049, 53060, 53102, 53118, 53122, 53161, 53189, 53194, 53207, 53213, 53215, 53216, 53220, 53237, 53272, 53288, 53296, 53298, 53301, 53344, 53360, 53391, 53397, 53418, 53419, 53443, 53458, 53494, 53495, 53505, 53519, 53520, 53571, 53607, 53641, 53651, 53652, 53654, 53671, 53697, 53710, 53736, 53764, 53769, 53787, 53805, 53817, 53830, 53844, 53855, 53856, 53877, 53958, 53962, 53975, 54001, 54002, 54015, 54025, 54027, 54068, 54074, 54087, 54098, 54099, 54115, 54129, 54130, 54133, 54159, 54166, 54170, 54179, 54186, 54189, 54226, 54234, 54246, 54272, 54306, 54341, 54372, 54428, 54451, 54460, 54643, 55058, 55077, 55079, 55115, 55173, 55227, 55249, 55257, 55258, 55259, 55270, 55283, 55314, 55340, 55352, 55370, 55439, 55448, 55534, 55568, 55580, 55586, 55633, 55678, 55690, 55719, 55734, 55754, 55761, 55768, 55777, 55807, 55841, 55868, 55895, 55927, 55929, 55933, 55963, 55968, 56016, 56053, 56054, 56055, 56120, 56149, 56194, 56220, 56228, 56237, 56244, 56250, 56290, 56291, 56294, 56321, 56332, 56334, 56341, 56350, 56352, 56365, 56368, 56371, 56412, 56421, 56426, 56427, 56459, 56487, 56488, 56490, 56522, 56523, 56531, 56538, 56558, 56580, 56585, 57485
- contains the fix via a backport PR (38): 56059, 56167, 56370, 56613, 56616, 56647, 56650, 56677, 56757, 56766, 56767, 56769, 56771, 56772, 56788, 56906, 57018, 57058, 57089, 57101, 57102, 57139, 57157, 57169, 57173, 57174, 57175, 57250, 57323, 57329, 57402, 57439, 57474, 57488, 57553, 57665, 57758, 57764
- does not contain the fix (1): 57375
pandas 1.5.3 (tag v1.5.3; 86 tasks)
- contains the fix, merge commit in tag history (52): 47157, 47222, 47231, 47273, 47361, 47371, 47446, 47451, 47493, 47528, 47547, 47550, 47557, 47581, 47586, 47605, 47619, 47623, 47633, 47643, 47716, 47731, 47747, 47754, 47757, 47761, 47771, 47780, 47804, 47810, 47840, 47841, 47848, 47860, 47881, 47894, 47927, 47972, 47987, 48008, 48057, 48058, 48073, 48094, 48147, 48149, 48164, 48167, 48169, 48176, 49453, 50627
- contains the fix via a backport PR (28): 48334, 48555, 48579, 48587, 48619, 48686, 48693, 48696, 48697, 48711, 48713, 48734, 48736, 48760, 48782, 48797, 48820, 48824, 48890, 48995, 49080, 49284, 49377, 49579, 49610, 49864, 50682, 50685
- does not contain the fix (6): 49967, 50331, 51042, 51960, 51976, 52459
dask 2024.8.1 (tag 2024.8.1; 62 tasks)
- contains the fix, merge commit in tag history (62): 7688, 8865, 8895, 8920, 8925, 8945, 8954, 9027, 9087, 9240, 9250, 9272, 9285, 9342, 9345, 9349, 9350, 9378, 9531, 9627, 9653, 9739, 9760, 9885, 9981, 10009, 10027, 10042, 10054, 10064, 10128, 10149, 10159, 10211, 10212, 10294, 10342, 10380, 10395, 10404, 10422, 10441, 10454, 10506, 10509, 10521, 10556, 10616, 10695, 10734, 10750, 10754, 10765, 10785, 10796, 10797, 10803, 10827, 10846, 10921, 10972, 11023
dask 2024.8.0 (tag 2024.8.0; 30 tasks)
- contains the fix, merge commit in tag history (30): 7636, 7656, 7894, 7973, 8040, 8106, 8166, 8185, 8253, 8261, 8283, 8316, 8375, 8462, 8501, 8525, 8538, 8590, 8601, 8610, 8686, 8687, 8689, 8703, 8785, 8792, 8801, 8805, 8820, 8860
dask 2021.7.2 (tag 2021.07.2; 26 tasks)
- contains the fix, merge commit in tag history (26): 6608, 6626, 6632, 6660, 6682, 6742, 6749, 6776, 6779, 6801, 6809, 6815, 6854, 6862, 6887, 6893, 6939, 6960, 6978, 6980, 6992, 7056, 7073, 7092, 7138, 7145
dask 2022.11.0 (tag 2022.11.0; 2 tasks)
- contains the fix, merge commit in tag history (2): 7125, 7230
Activity