Repository navigation
Conversation
…heat git scrub anti_cheat_setup.sh deleted every tag, including the tags on ancestors of the task's base commit. Projects that take their version from git describe then report 0+untagged..., which in SWE-Gym's pandas 2.1 tasks breaks pandas/conftest.py and stops the agent from running tests in its sandbox. Record the tags merged into the base before deleting refs, and recreate them before git gc. Ancestry is computed with --no-replace-objects, since replace refs are deleted too and could otherwise make a later commit look like an ancestor. Tags that are not on an ancestor, other branches, remotes and reflogs are still removed. Add a regression test that runs the script on throwaway repositories, and update the four docs that said the scrub drops all tags. Closes NVIDIA-NeMo#5745 Signed-off-by: Rogerffff <143265077+Rogerffff@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Closes #5745 by implementing the proposal there.
anti_cheat_setup.shdeletes every tag, including the tags on ancestors of the task's base commit. Projects that compute their version fromgit describeat runtime then report0+untagged.... In SWE-Gym's pandas 2.1 tasks,pandas/conftest.pythen fails to import, so the agent cannot run any test (#5745: 113 of 114 tested 2.1 tasks, plus 21 tasks on 2.0 and 1.5 where some tests fail). Grading is not affected, because/verifyruns in an unscrubbed sandbox.anti_cheat_setup.sh: record the tags on ancestors of HEAD (git --no-replace-objects for-each-ref --merged=<base> refs/tags) and recreate them after the refs are deleted, beforegit gc. Everything else is still removed: later and side-branch tags, branches, remotes, reflogs and unreachable objects.--no-replace-objectskeeps a replace ref from passing a later commit off as an ancestor. TheGIT_CLEANUPline gainstags_kept=N.tests/test_anti_cheat_setup_script.py(10 tests) runs the real script on throwaway repositories, withHOMEandGIT_CONFIG_GLOBALintmp_path.On the questions in #5745: I found no recorded reason for deleting ancestor tags (#2579, #3327). They point only at commits the agent can already see, though annotated tags keep their message. The change is in the shared script, so all 14 servers that run it get it (not 12, as #5745 said); I checked images from five of them (see Validation).
Validation
On Linux (Debian 12 x86_64, git 2.39.5, bash 5.2, Python 3.13.14): swebench 177, swe_gym 105 and r2e_gym 64 passed, data validation passes, and pre-commit passes (guardwords skipped without the private catalog). The new tests on unpatched main: 8 failed, 2 passed.
Image controls on Linux: fresh containers per published image (no scrub, current script, this PR; scrubbed as
apply_anti_cheat_setupdoes), then the task's test command at the base, with no patch:e0c5b878f9bed20btest_numeric.pyfails to collect (int('0+untagged')in hypothesis) / 1321 passed, 51 failedsbt "show version"works / sbt cannot loadbuild.sbt(git describeexits 128) / workspre_commandsremove them firstNothing after the base stays reachable with either script. The swe_gym rows and the rollouts ran the previous revision of this branch, without
--no-replace-objects, which changes nothing on those images (they have no replace refs); the other rows ran this revision.Rollouts
One smoke rollout per side on
pandas-dev__pandas-53958:deepseek-flashthroughinference_provider, OpenCode throughsingle_agent_turn, Docker sandbox, one sample each.pandas 0+untagged... is installed)/verifyOn main the agent read the conftest failure as an environment problem ("pytest was blocked by an unrelated pyarrow/pandas version warning in this environment", in its final message) and checked its change with hand-written
python -csnippets instead. The equal rewards say nothing about this change: both agents copied the fix from the newer pandas in the image's conda package cache (#5744).Command
CFG=(--config resources_servers/swe_gym/configs/swe_gym_opencode.yaml --config nemo_gym/sandbox/providers/docker/configs/docker.yaml --model-type inference_provider) OVR=(++sandbox.docker.create.network=host ++swe_gym_opencode_agent.responses_api_agents.opencode_agent.token_id_capture=false ++swe_gym_opencode_agent.responses_api_agents.opencode_agent.timeout=2400 ++swe_gym_opencode_agent.responses_api_agents.opencode_agent.context_window=131072) gym env start "${CFG[@]}" "${OVR[@]}" gym eval run --no-serve "${CFG[@]}" "${OVR[@]}" --input pandas_53958_task.jsonl --output rollouts.jsonl --limit 1network=hostlets OpenCode in the sandbox reach the model server on127.0.0.1. On main the agent ran a wrong node ID (TestPDApi::test_api_typing), but pytest failed earlier, at conftest import.Compatibility and benchmark impact
GIT_CLEANUPline.git describecome from a real tag again instead of0+untagged.... Where the missing tags broke the agent's own test runs, those runs now work; elsewhere tests behave the same (see the image controls). Estimated tasks this fixes: swe_gym, the 134 measured inanti_cheat_setup.shalso deletes the tags on ancestors of the base commit, so for someswe_gympandas tasks, tests that used to pass fail in the agent sandbox #5745 plus about 77 untested pandas 2.1 tasks, of 2,278; r2e_gym, about 245 of 4,578 (pandas from February 2023 on) plus some numpy tasks that importhypothesis.extra.numpy; swe_rebench, the 42 scalameta tasks of 22,663 (static-scan estimates for r2e_gym and swe_rebench). Rollouts on those tasks may differ from runs made before this change.git update-refper kept tag, 0.05 to 0.6 s more on the four SWE-Gym images and 3.6 s for 2,000 tags. I am happy to switch to a singlegit update-ref --stdin.anti_cheat.pybut not this script, and merge without conflicts.Checklist
pre-commit run --all-files) (so CI lint/format checks pass).git commit -s) (so the DCO check passes).Proposed labels (I cannot apply labels):
bug,area:environment,needs-review.