Skip to content

feat(swe): repository-task servers on the sandbox owner base - #5331

Draft
pthombre wants to merge 6 commits into
pranav/envckpt-5-e2e-docsfrom
pranav/envckpt-6-repo-task-servers
Draft

pthombre wants to merge 6 commits into
pranav/envckpt-5-e2e-docsfrom
pranav/envckpt-6-repo-task-servers

Conversation

@pthombre

@pthombre pthombre commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

What changed and why

The stack makes SWE-bench Verified checkpointable (#5267), but the SWE training blend runs on other resources servers. On main, swe_rebench, swemer_v1, swemer_v2 and swe_next keep each session's sandbox, base commit and pristine untracked files in memory under the session cookie and serve no episode sessions, so the Environment Server cannot seed them and a checkpoint cannot export them.

This PR puts the four servers on the checkpointing session base of #5266, through one shared base for their common session shape.

  • RepoTaskSessionResourcesServer, beside the patch-capture and anti-cheat helpers the servers already share:
    • Episode seed (Environment Server): task_data must validate as the server's row and the task id must name it, else the seed is refused. A restored session's seed gets the fork without setup.
    • An agent's own run still seeds with the flat row under its session cookie, with the same request and response.
    • One setup for both. It starts the task sandbox through new_task_sandbox and prepares the repository. The commit the patch is taken against and the files the checkout held untracked before the agent started are exported and restored with the session, so a restore captures against the same base.
    • Replayable verify. checkpoint_verify = "replay": verify captures the agent's patch once, stops the agent's sandbox, and grades the patch in a fresh sandbox. A verify that a checkpoint replays, or that runs again after a restore, grades the saved patch. A capture that fails grades an empty patch, which is saved like any other. An episode session replays its first verdict to an identical retry.
    • Grading sandboxes are not session state: each grading run starts its own.
    • Golden-patch servers run restart_only.
  • Per server, only what is specific stays: the row model, the sandbox spec, the agent's working directory and the repository preparation.
  • Configs: each server's allowed_agents adds opencode_agent.
  • scale_swe is not included. On main it has its own episode sessions, which differ from the shared session base, so it keeps them and its episodes start over after a crash (restart_only). Checkpointing it is a follow-up. Of the changes below it gets the eval stderr fix and the scrub retries.

Eval stderr (second commit)

The graders parse only the region between their start and end markers. A provider that returns a command's stdout and stderr apart (E2B) put every line a test runner wrote on stderr after the end marker, so tests that report there graded as missing. Each eval script of the four servers and of scale_swe now sends its stderr to its stdout, in order.

Scrub hardening and agent network isolation (third commit)

Found in the AgentENV training runs. The change is provider-neutral.

  • The anti-cheat scrub is retried on provider errors. At scale, a sandbox gateway can briefly return an error, and the best-effort scrub then handed the agent a git history that still held the task's fix. The scrub and the committer-identity step are idempotent, so a provider error is now retried a few times with backoff. This applies to every server that calls the shared helpers.
  • anti_cheat_required (repo-task servers, on for swe_rebench): a scrub that still fails raises an error, the seed fails, and the rollout is retried instead of running against the answer.
  • agent_network_policy (repo-task servers, off by default): a provider-neutral egress rule applied to the agent's sandbox only; grading sandboxes keep the spec's network. The rule is recorded with the sandbox, so a restore's fork gets it too. A typical rule denies all egress except an allowed address range, for example the model servers', so agents reach the model and nothing else.

Setup reads on a new sandbox (last commit)

  • The base-commit and pristine-file reads are retried when the provider fails to run them: a new sandbox can refuse its first commands for a few seconds.
  • A pristine-file listing that still cannot run fails the seed instead of returning an empty list, which would let the patch capture count the image's own untracked files as the agent's. A listing that runs and fails (the working directory is not a git repository) still gives the empty list, with a warning.

How it works

Where this PR sits in the overall flow

The highlighted part is what this PR changes.

flowchart LR
  C["Controller<br/>NeMo RL, or rollout collection"]
  CO["Coordination<br/>prepare, commit, resume, restore,<br/>retire, forget, release"]
  E["Environment server<br/>bookmarks, re-seed after a restore"]
  A["Harness agent, e.g. OpenCode<br/>run until parked, re-bind, resume"]
  S["Process supervisor in the task sandbox<br/>pause, continue, stop"]
  R["Resources servers<br/>SWE-bench, and now swe_rebench,<br/>swemer_v1, swemer_v2, swe_next"]
  OB["Session base with checkpoints<br/>export, restore, release"]
  K["Sandbox checkpointer<br/>snapshot index, release, gc"]
  P["Sandbox provider, e.g. OpenSandbox<br/>snapshot, fork, delete, list"]
  M["Policy model<br/>held calls, capture ledger"]
  D[("Checkpoint directory<br/>and state_dir")]
  C --> CO --> E & A & R & M
  E -- "seed, re-seed" --> R
  E -- "seed, re-seed, activation" --> A
  R -. "sandbox_access" .-> A
  A --> S
  R --- OB --> K --> P
  A -- "model calls" --> M
  CO --> D
  classDef this fill:khaki,stroke:darkgoldenrod,stroke-width:2px,color:black
  class R this
Loading

This PR

flowchart TD
  T["Episode seed from the Environment Server"] --> V{"task_data is the server's row<br/>and the task id names it?"}
  V -- "no" --> X["seed refused, no sandbox"]
  V -- "yes" --> SU
  F["Flat seed from an agent's own run"] --> SU
  SU["Start the task sandbox, prepare the repository,<br/>keep the base commit and pristine untracked files"]
  SU --> W["Agent works in the lent sandbox<br/>a checkpoint snapshots it"]
  W --> VE["verify"] --> CO{"Patch already captured?"}
  CO -- "yes" --> G
  CO -- "no" --> CA["Capture the patch against the base commit,<br/>drop pristine untracked files"]
  CA -- "ok" --> ST["Save the capture, stop the agent's sandbox"] --> G
  CA -- "fails" --> E0["Save an empty patch, stop the agent's sandbox"] --> G
  G["Grade in fresh sandboxes,<br/>inconclusive runs retried as before"]
Loading

Where this sits in the stack

This is one PR in a stack for checkpointing environment-server episodes whose agent harness runs in a sandbox the resources server lends (first pair: OpenCode and SWE-bench on OpenSandbox; #5410 to #5413 add AgentENV, where OpenCode resumes in place after a restore). #5264 has merged, and the rest of the stack is rebased onto main after it. Each PR is based on the one before it, so each diff shows only that PR's change. The stack runs parallel to the legacy_agent path of #4220 to #4936, which it does not change or depend on.

  1. feat(checkpoint): protocol for harnesses that borrow a sandbox #5264: feat(checkpoint): protocol for harnesses that borrow a sandbox (merged)
  2. feat(sandbox): snapshot, fork and delete for OpenSandbox, with a checkpointer #5265: feat(sandbox): snapshot, fork and delete for OpenSandbox, with a checkpointer
  3. feat(swebench): checkpoint the task sandboxes of episode sessions #5266: feat(resources): sandbox owner base for resources servers
  4. feat(swebench): typed sessions on the owner base with capture-once verify #5267: feat(swebench): typed sessions on the owner base with capture-once verify
  5. feat(opencode): pause at checkpoints and resume after a crash #5268: feat(opencode): pause at checkpoints and resume after a crash
  6. test(checkpoint): end-to-end harness scenarios, SWE-bench recipe, docs #5269: test(checkpoint): end-to-end harness scenarios, SWE-bench recipe, docs
  7. feat(swe): repository-task servers on the sandbox owner base #5331: feat(swe): repository-task servers on the sandbox owner base (this PR)
  8. feat(sandbox): AgentENV snapshots, lease and kill-at-timeout for the E2B provider #5410: feat(sandbox): AgentENV snapshots, lease and kill-at-timeout for the E2B provider
  9. feat(checkpoint): hold-pause a harness when the sandbox snapshot keeps its processes #5411: feat(checkpoint): hold-pause a harness when the sandbox snapshot keeps its processes
  10. feat(agent-utils): resume a supervised harness in place after a restore that kept its processes #5412: feat(agent-utils): resume a supervised harness in place after a restore that kept its processes
  11. feat(opencode): resume OpenCode in place on sandboxes whose snapshots keep processes #5413: feat(opencode): resume OpenCode in place on sandboxes whose snapshots keep processes

Issue

No tracking issue. This extends the environment-server checkpoint plan from SWE-bench Verified to four servers of the SWE training blend, so a training run can checkpoint them.

Validation

  • New unit tests run the four servers against a fake provider with snapshots: each passes check_sandbox_session_checkpoints, and episode and flat seeds, the replayable verify, failed captures, and restore into a fork behave as described above.
  • Unit tests for the eval scripts with stdout and stderr apart, the scrub retries, anti_cheat_required, the network rule, and the setup-read retries.
  • The servers' existing suites, scale_swe's included, pass; lint is clean.
  • Rebased onto main after feat(checkpoint): protocol for harnesses that borrow a sandbox #5264 merged: the unit, server and process-level checkpoint suites fail only where main itself does, apart from intermittent failures in tests that pass when run alone or that fail the same way on main.
  • No training run yet on the stack as it is now. The two 16-node AgentENV RL training runs of this stack, a checkpoint run and a restore from its checkpoint, were made on its previous version, which had its own sandbox owner base and set the clock back inside the sandbox. They committed every checkpoint with no sandbox loss, and the restore resumed every held session in place. In those runs, which had all five servers on the previous owner base, every scrub that hit a transient provider error succeeded on retry, and no seed failed on the setup reads, where a run before that fix lost some and once got an empty pristine list.

Compatibility

  • scale_swe: sessions unchanged; with checkpointing on, its episodes start over after a crash.
  • Scrub: retries apply to every caller of the shared helpers; failing closed is opt-in (anti_cheat_required, on for swe_rebench). The network rule is opt-in.
  • Behavior without checkpoints: the same setup, patch capture and grading. The agent's sandbox now stops right after the capture, including a capture that failed.
  • Flat seeds keep their request and response; one that would replace an episode session is refused.
  • Config: the four configs gain optional sandbox checkpoint settings; their golden-patch, patch-capture-mode and sandbox fields move to a shared repo-task config unchanged. allowed_agents adds opencode_agent.
  • Eval output: stderr is joined to stdout. On a provider that returns them apart, tests that report on stderr are now graded instead of counted as missing.
  • Seeds fail, and the caller retries them, when the pristine-file listing cannot run.
  • Checkpointing on: the four servers need checkpoint.state_dir when their provider can snapshot.

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Oct 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@pthombre
pthombre added this pull request to stack #5270 October 29, 2026 19:26
@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers feature New capabilities, enhancements, or enablement work labels Oct 29, 2026
@pthombre
pthombre force-pushed the pranav/envckpt-6-repo-task-servers branch from d1901c1 to 90142af Compare October 31, 2026 18:07
@pthombre
pthombre force-pushed the pranav/envckpt-6-repo-task-servers branch from 1105fdf to 7ee8313 Compare November 3, 2026 03:22
raise AntiCheatSetupError(f"{instance_id}: anti-cheat setup raised: {error!r}") from error
print(f"[{log_prefix}] {instance_id}: anti-cheat setup raised (non-fatal)", format_exc(), file=sys.stderr)
return
if result.return_code == 0:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

anti_cheat_required treats a zero exit as a finished scrub. anti_cheat_setup.sh exits 0 when it skips, and also when non_ancestor_left is not 0 (gc_rc is only printed). For swe_rebench, can the agent still be handed a history that contains the fix? Should a required scrub fail the seed in those cases?

Comment thread resources_servers/swe_rebench/app.py Outdated
started = time()
try:
sandbox = await self._create_sandbox(body, files=files)
sandbox = await self.start_grading_sandbox(body, files)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The finally below stops this sandbox with an unshielded await, and except Exception does not catch cancellation, so a cancelled verify cancels the stop too. SWE-bench shields that stop. Should this server, and the same finally in scale_swe, swe_next, swemer_v1, and swemer_v2, do the same? A cancel inside start_grading_sandbox, before sandbox is assigned, also looks like it would miss the finally.

@pthombre

pthombre commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Being rebased onto current main, after #5264 and #5265 (already rebased). Until this branch is pushed, its diff still shows their old commits. Not ready for review yet.

@pthombre
pthombre force-pushed the pranav/envckpt-6-repo-task-servers branch from 679a601 to d38da2d Compare November 12, 2026 20:35
pthombre and others added 6 commits November 15, 2026 05:55
swe_next now derives from SandboxSessionResourcesServer, in the shape #4304 gave swe_rebench, swemer_v1 and
swemer_v2: an Environment Server can seed an episode session, hand the agent the task sandbox through
sandbox_access, verify once and close. An agent's own /run still seeds with the flat row.

With the shape come its siblings' rules: the patch is captured once per session and the agent's sandbox stopped
after it; a capture the sandbox could not run gives a masked, incomplete verdict instead of grading an empty
patch; a grading sandbox is registered with the session, so a close stops it; a test command the provider
interrupted is incomplete, and one that timed out is a completed 0; per-task logs are written only when
verification_log_dir is set (they used to go under the server's source folder).

Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…vers

swe_rebench, swemer_v1, swemer_v2 and swe_next keep the seed and verify they have, and now derive from
RepoTaskSessionResourcesServer (resources_servers/swebench/repo_task_sessions.py), a thin layer on
CheckpointedSandboxSessionResourcesServer, so their episodes take part in partial-rollout checkpoints: the
agent's sandbox is snapshotted with its work and forked on a restore.

- The agent's sandbox is created with new_task_sandbox, on the provider every session shares; grading sandboxes
  stay on a provider of their own and are not session state.
- The files the checkout held untracked before the agent started, and the commit the patch is taken against
  where the seed records one (swemer_v1, swemer_v2, swe_next), are exported and restored with the session.
- The captured patch is saved as a JSON value (saved_capture / loaded_capture), with its extraction error and
  whether the sandbox could not run the capture, so a commit can store it and a /verify after a restore grades
  the saved patch. checkpoint_verify = "replay".
- Golden-patch servers run restart_only.
- swe_next's allowed_agents adds opencode_agent.

scale_swe keeps its own episode-session implementation.

Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The swe_rebench, scale_swe, swemer_v1, swemer_v2 and swe_next graders parse only the region between their start
and end markers, from the command's stdout followed by its stderr. A provider that returns a command's stdout and
stderr apart (E2B) puts every line a test runner writes on stderr after the end marker, so tests that report there
(jest does) grade as missing. Each eval script now sends its stderr to stdout, in order, before anything runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
…po-task server require it

On AgentENV at 16-node scale, 16 scrubs per run hit a gateway 502 after the sandbox had answered; the
best-effort scrub then handed the agent a git history that still held the task's fix. The scrub and the
committer-identity step are idempotent and are now retried on a provider error (3 attempts, 2 s then 4 s)
for every server that calls them. A repo-task server (swe_rebench, swemer_v1, swemer_v2, swe_next) with
`anti_cheat_required` raises AntiCheatSetupError instead of continuing past a scrub that did not complete,
so the seed fails and the rollout is retried; swe_rebench's resources config turns it on.

The repo-task servers also take `agent_network_policy`: a provider-neutral egress rule applied to the
agent's sandbox only (grading sandboxes keep the spec's network). It is recorded with the sandbox, so a
restore's fork gets the same rule; an address range stays valid across jobs.

Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…istine files cannot be listed

A sandbox that was just created or restored can refuse a connection for a few seconds (a gateway 502). The
repository-task seed's `git rev-parse HEAD` then failed the seed, and the pristine untracked-file listing silently
became empty (job 8539361: 6 sandboxes). An empty list makes the patch capture count the image's own untracked files
as the agent's new files, so the rollout could be graded on a wrong patch instead of being retried.

Both reads are read-only and now go through `exec_idempotent`. The listing moves from each server into
RepoTaskSessionResourcesServer. A listing that still cannot run, or that the provider interrupted, fails the seed,
so the row is dispatched again; a listing that runs and fails (the workdir is not a git repository) still gives the
empty set with a warning.

Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… scrub that did not scrub, and retry the first commands

Found in a review of the repository-task servers' checkpoints and hardening.

- `agent_network_policy` was written into the agent sandbox's provider options whether or not the provider reads
  them, so on a provider that does not the agent kept the spec's network with nothing to show for it. A provider
  that applies `provider_options.network_policy` now says so (`applies_network_policy`, set on OpenSandbox), and a
  repository-task server with the rule refuses to start on one that does not. The field comment no longer calls
  the rule provider-neutral: it is in OpenSandbox's shape.
- `anti_cheat_required` only read the scrub's exit code, and `anti_cheat_setup.sh` exits 0 when it skipped the
  repository and when refs still reach commits after the base. A required scrub now also fails when the command
  was not run to its end, when the script printed no summary, when it skipped, and when it counted commits that
  are no ancestor of the base. A `git gc` that failed after the refs were removed is logged, since no name
  reaches those commits; it is tried again only when an earlier attempt raised, because that attempt's gc may
  still be running. A scrub that is not required reads the exit code alone, as before.
- A listing of the pristine untracked files that git refuses now fails the seed, like one the provider could not
  run. It used to give the empty set, and the patch capture then counted the image's own untracked files as the
  agent's.
- The commands that really run first in a new sandbox are retried when the provider raises: swe_next's link of
  the workdir and its removal of the grading artifacts, and the swemer servers' check for a repository. A check
  the provider reports as not run now fails the seed instead of reading as "no repository", which initialised a
  new one on top of a repository that may exist.
- Tests: a restore keeps the outcome of a capture that failed (git refused it, or the sandbox could not run it);
  a seed that reads no base commit fails and stops its sandbox; the committer-identity retry; the shipped
  swe_rebench configs require the scrub; the network rule is added to the spec's other provider options. The
  eval-script tests run in a temporary directory and refuse to run anything but the script's redirect, so a
  reworded script can no longer run a real checkout reset where pytest was started. The commit-during-grading
  test bounds its waits, and no anti-cheat test waits for the retry delay any more.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Pranav Thombre <pthombre@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers feature New capabilities, enhancements, or enablement work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants