Repository navigation
Conversation
d1901c1 to
90142af
Compare
1105fdf to
7ee8313
Compare
| raise AntiCheatSetupError(f"{instance_id}: anti-cheat setup raised: {error!r}") from error | ||
| print(f"[{log_prefix}] {instance_id}: anti-cheat setup raised (non-fatal)", format_exc(), file=sys.stderr) | ||
| return | ||
| if result.return_code == 0: |
There was a problem hiding this comment.
anti_cheat_required treats a zero exit as a finished scrub. anti_cheat_setup.sh exits 0 when it skips, and also when non_ancestor_left is not 0 (gc_rc is only printed). For swe_rebench, can the agent still be handed a history that contains the fix? Should a required scrub fail the seed in those cases?
| started = time() | ||
| try: | ||
| sandbox = await self._create_sandbox(body, files=files) | ||
| sandbox = await self.start_grading_sandbox(body, files) |
There was a problem hiding this comment.
The finally below stops this sandbox with an unshielded await, and except Exception does not catch cancellation, so a cancelled verify cancels the stop too. SWE-bench shields that stop. Should this server, and the same finally in scale_swe, swe_next, swemer_v1, and swemer_v2, do the same? A cancel inside start_grading_sandbox, before sandbox is assigned, also looks like it would miss the finally.
7ee8313 to
4987302
Compare
4987302 to
679a601
Compare
679a601 to
d38da2d
Compare
swe_next now derives from SandboxSessionResourcesServer, in the shape #4304 gave swe_rebench, swemer_v1 and swemer_v2: an Environment Server can seed an episode session, hand the agent the task sandbox through sandbox_access, verify once and close. An agent's own /run still seeds with the flat row. With the shape come its siblings' rules: the patch is captured once per session and the agent's sandbox stopped after it; a capture the sandbox could not run gives a masked, incomplete verdict instead of grading an empty patch; a grading sandbox is registered with the session, so a close stops it; a test command the provider interrupted is incomplete, and one that timed out is a completed 0; per-task logs are written only when verification_log_dir is set (they used to go under the server's source folder). Signed-off-by: Pranav Thombre <pthombre@nvidia.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…vers swe_rebench, swemer_v1, swemer_v2 and swe_next keep the seed and verify they have, and now derive from RepoTaskSessionResourcesServer (resources_servers/swebench/repo_task_sessions.py), a thin layer on CheckpointedSandboxSessionResourcesServer, so their episodes take part in partial-rollout checkpoints: the agent's sandbox is snapshotted with its work and forked on a restore. - The agent's sandbox is created with new_task_sandbox, on the provider every session shares; grading sandboxes stay on a provider of their own and are not session state. - The files the checkout held untracked before the agent started, and the commit the patch is taken against where the seed records one (swemer_v1, swemer_v2, swe_next), are exported and restored with the session. - The captured patch is saved as a JSON value (saved_capture / loaded_capture), with its extraction error and whether the sandbox could not run the capture, so a commit can store it and a /verify after a restore grades the saved patch. checkpoint_verify = "replay". - Golden-patch servers run restart_only. - swe_next's allowed_agents adds opencode_agent. scale_swe keeps its own episode-session implementation. Signed-off-by: Pranav Thombre <pthombre@nvidia.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The swe_rebench, scale_swe, swemer_v1, swemer_v2 and swe_next graders parse only the region between their start and end markers, from the command's stdout followed by its stderr. A provider that returns a command's stdout and stderr apart (E2B) puts every line a test runner writes on stderr after the end marker, so tests that report there (jest does) grade as missing. Each eval script now sends its stderr to stdout, in order, before anything runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
…po-task server require it On AgentENV at 16-node scale, 16 scrubs per run hit a gateway 502 after the sandbox had answered; the best-effort scrub then handed the agent a git history that still held the task's fix. The scrub and the committer-identity step are idempotent and are now retried on a provider error (3 attempts, 2 s then 4 s) for every server that calls them. A repo-task server (swe_rebench, swemer_v1, swemer_v2, swe_next) with `anti_cheat_required` raises AntiCheatSetupError instead of continuing past a scrub that did not complete, so the seed fails and the rollout is retried; swe_rebench's resources config turns it on. The repo-task servers also take `agent_network_policy`: a provider-neutral egress rule applied to the agent's sandbox only (grading sandboxes keep the spec's network). It is recorded with the sandbox, so a restore's fork gets the same rule; an address range stays valid across jobs. Signed-off-by: Pranav Thombre <pthombre@nvidia.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…istine files cannot be listed A sandbox that was just created or restored can refuse a connection for a few seconds (a gateway 502). The repository-task seed's `git rev-parse HEAD` then failed the seed, and the pristine untracked-file listing silently became empty (job 8539361: 6 sandboxes). An empty list makes the patch capture count the image's own untracked files as the agent's new files, so the rollout could be graded on a wrong patch instead of being retried. Both reads are read-only and now go through `exec_idempotent`. The listing moves from each server into RepoTaskSessionResourcesServer. A listing that still cannot run, or that the provider interrupted, fails the seed, so the row is dispatched again; a listing that runs and fails (the workdir is not a git repository) still gives the empty set with a warning. Signed-off-by: Pranav Thombre <pthombre@nvidia.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… scrub that did not scrub, and retry the first commands Found in a review of the repository-task servers' checkpoints and hardening. - `agent_network_policy` was written into the agent sandbox's provider options whether or not the provider reads them, so on a provider that does not the agent kept the spec's network with nothing to show for it. A provider that applies `provider_options.network_policy` now says so (`applies_network_policy`, set on OpenSandbox), and a repository-task server with the rule refuses to start on one that does not. The field comment no longer calls the rule provider-neutral: it is in OpenSandbox's shape. - `anti_cheat_required` only read the scrub's exit code, and `anti_cheat_setup.sh` exits 0 when it skipped the repository and when refs still reach commits after the base. A required scrub now also fails when the command was not run to its end, when the script printed no summary, when it skipped, and when it counted commits that are no ancestor of the base. A `git gc` that failed after the refs were removed is logged, since no name reaches those commits; it is tried again only when an earlier attempt raised, because that attempt's gc may still be running. A scrub that is not required reads the exit code alone, as before. - A listing of the pristine untracked files that git refuses now fails the seed, like one the provider could not run. It used to give the empty set, and the patch capture then counted the image's own untracked files as the agent's. - The commands that really run first in a new sandbox are retried when the provider raises: swe_next's link of the workdir and its removal of the grading artifacts, and the swemer servers' check for a repository. A check the provider reports as not run now fails the seed instead of reading as "no repository", which initialised a new one on top of a repository that may exist. - Tests: a restore keeps the outcome of a capture that failed (git refused it, or the sandbox could not run it); a seed that reads no base commit fails and stops its sandbox; the committer-identity retry; the shipped swe_rebench configs require the scrub; the network rule is added to the spec's other provider options. The eval-script tests run in a temporary directory and refuse to run anything but the script's redirect, so a reworded script can no longer run a real checkout reset where pytest was started. The commit-during-grading test bounds its waits, and no anti-cheat test waits for the retry delay any more. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
d38da2d to
cdc148e
Compare
What changed and why
The stack makes SWE-bench Verified checkpointable (#5267), but the SWE training blend runs on other resources servers. On
main,swe_rebench,swemer_v1,swemer_v2andswe_nextkeep each session's sandbox, base commit and pristine untracked files in memory under the session cookie and serve no episode sessions, so the Environment Server cannot seed them and a checkpoint cannot export them.This PR puts the four servers on the checkpointing session base of #5266, through one shared base for their common session shape.
RepoTaskSessionResourcesServer, beside the patch-capture and anti-cheat helpers the servers already share:task_datamust validate as the server's row and the task id must name it, else the seed is refused. A restored session's seed gets the fork without setup.new_task_sandboxand prepares the repository. The commit the patch is taken against and the files the checkout held untracked before the agent started are exported and restored with the session, so a restore captures against the same base.checkpoint_verify = "replay": verify captures the agent's patch once, stops the agent's sandbox, and grades the patch in a fresh sandbox. A verify that a checkpoint replays, or that runs again after a restore, grades the saved patch. A capture that fails grades an empty patch, which is saved like any other. An episode session replays its first verdict to an identical retry.restart_only.allowed_agentsaddsopencode_agent.scale_sweis not included. Onmainit has its own episode sessions, which differ from the shared session base, so it keeps them and its episodes start over after a crash (restart_only). Checkpointing it is a follow-up. Of the changes below it gets the eval stderr fix and the scrub retries.Eval stderr (second commit)
The graders parse only the region between their start and end markers. A provider that returns a command's stdout and stderr apart (E2B) put every line a test runner wrote on stderr after the end marker, so tests that report there graded as missing. Each eval script of the four servers and of
scale_swenow sends its stderr to its stdout, in order.Scrub hardening and agent network isolation (third commit)
Found in the AgentENV training runs. The change is provider-neutral.
anti_cheat_required(repo-task servers, on forswe_rebench): a scrub that still fails raises an error, the seed fails, and the rollout is retried instead of running against the answer.agent_network_policy(repo-task servers, off by default): a provider-neutral egress rule applied to the agent's sandbox only; grading sandboxes keep the spec's network. The rule is recorded with the sandbox, so a restore's fork gets it too. A typical rule denies all egress except an allowed address range, for example the model servers', so agents reach the model and nothing else.Setup reads on a new sandbox (last commit)
How it works
Where this PR sits in the overall flow
The highlighted part is what this PR changes.
flowchart LR C["Controller<br/>NeMo RL, or rollout collection"] CO["Coordination<br/>prepare, commit, resume, restore,<br/>retire, forget, release"] E["Environment server<br/>bookmarks, re-seed after a restore"] A["Harness agent, e.g. OpenCode<br/>run until parked, re-bind, resume"] S["Process supervisor in the task sandbox<br/>pause, continue, stop"] R["Resources servers<br/>SWE-bench, and now swe_rebench,<br/>swemer_v1, swemer_v2, swe_next"] OB["Session base with checkpoints<br/>export, restore, release"] K["Sandbox checkpointer<br/>snapshot index, release, gc"] P["Sandbox provider, e.g. OpenSandbox<br/>snapshot, fork, delete, list"] M["Policy model<br/>held calls, capture ledger"] D[("Checkpoint directory<br/>and state_dir")] C --> CO --> E & A & R & M E -- "seed, re-seed" --> R E -- "seed, re-seed, activation" --> A R -. "sandbox_access" .-> A A --> S R --- OB --> K --> P A -- "model calls" --> M CO --> D classDef this fill:khaki,stroke:darkgoldenrod,stroke-width:2px,color:black class R thisThis PR
flowchart TD T["Episode seed from the Environment Server"] --> V{"task_data is the server's row<br/>and the task id names it?"} V -- "no" --> X["seed refused, no sandbox"] V -- "yes" --> SU F["Flat seed from an agent's own run"] --> SU SU["Start the task sandbox, prepare the repository,<br/>keep the base commit and pristine untracked files"] SU --> W["Agent works in the lent sandbox<br/>a checkpoint snapshots it"] W --> VE["verify"] --> CO{"Patch already captured?"} CO -- "yes" --> G CO -- "no" --> CA["Capture the patch against the base commit,<br/>drop pristine untracked files"] CA -- "ok" --> ST["Save the capture, stop the agent's sandbox"] --> G CA -- "fails" --> E0["Save an empty patch, stop the agent's sandbox"] --> G G["Grade in fresh sandboxes,<br/>inconclusive runs retried as before"]Where this sits in the stack
This is one PR in a stack for checkpointing environment-server episodes whose agent harness runs in a sandbox the resources server lends (first pair: OpenCode and SWE-bench on OpenSandbox; #5410 to #5413 add AgentENV, where OpenCode resumes in place after a restore). #5264 has merged, and the rest of the stack is rebased onto
mainafter it. Each PR is based on the one before it, so each diff shows only that PR's change. The stack runs parallel to thelegacy_agentpath of #4220 to #4936, which it does not change or depend on.Issue
No tracking issue. This extends the environment-server checkpoint plan from SWE-bench Verified to four servers of the SWE training blend, so a training run can checkpoint them.
Validation
check_sandbox_session_checkpoints, and episode and flat seeds, the replayable verify, failed captures, and restore into a fork behave as described above.anti_cheat_required, the network rule, and the setup-read retries.scale_swe's included, pass; lint is clean.mainafter feat(checkpoint): protocol for harnesses that borrow a sandbox #5264 merged: the unit, server and process-level checkpoint suites fail only wheremainitself does, apart from intermittent failures in tests that pass when run alone or that fail the same way onmain.Compatibility
scale_swe: sessions unchanged; with checkpointing on, its episodes start over after a crash.anti_cheat_required, on forswe_rebench). The network rule is opt-in.allowed_agentsaddsopencode_agent.checkpoint.state_dirwhen their provider can snapshot.🤖 Generated with Claude Code