Skip to content

feat(agent-utils): resume a supervised harness in place after a restore that kept its processes - #5412

Draft
pthombre wants to merge 7 commits into
pranav/envckpt-8-hold-pausefrom
pranav/envckpt-9-resume-in-place
Draft

pthombre wants to merge 7 commits into
pranav/envckpt-8-hold-pausefrom
pranav/envckpt-9-resume-in-place

Conversation

@pthombre

@pthombre pthombre commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

What changed and why

After a crash, the stack restores a sandbox from its snapshot and relaunches the harness from its saved conversation (#5268). On a provider whose snapshots keep processes (AgentENV, #5410), the restored sandbox is still running the harness: the supervisor holds the checkpoint's hold pause (#5411), and the harness is mid-task with its tools, background servers and in-memory state intact. Relaunching would throw that away and start a second harness beside the first.

Two things stand in the way of simply continuing it:

  1. The model endpoint is gone. The call the harness was waiting on was cut, and the next job serves the model at a different URL.
  2. Gym must know the launch really survived: the same supervisor, holding the same pause, with no result written yet.

This PR adds the agent-utils pieces that solve both, for any harness under SandboxSession. #5413 wires them into OpenCode.

1. Model relay

The relay runs beside the harness inside the sandbox and forwards its model calls to the upstream named in an endpoint file, so the restoring job can re-point it by rewriting that file. It is JavaScript using only Node's standard modules, so a harness built on Bun runs it on its own binary (BUN_BE_BUN=1), and Node runs it too: the relay needs nothing from the task image.

  • Hold during a checkpoint. While a pause is in effect, new calls wait unsent.
  • Byte-identical re-send. A call whose response has not started is sent again, byte for byte, when the endpoint's generation changes. A response that has already started is cut with a reset.
  • Unreachable upstream is retried with backoff, then answered with 502.
  • Status file with its pid, port, generation and counters.

The harness runner starts the relay with the command the harness gives (MODEL_RELAY_COMMAND, with variables set for the relay's process alone), through a link with a neutral name that a task's pkill of python or of the harness does not match, and a watchdog restarts it on the same port. If the relay cannot start, the runner marks it unavailable and the harness calls the model server directly.

2. Taking over a restored launch

  • check_restored(...) checks in one command that the supervisor runs with the recorded identity, holds that pause in hold mode with no continue and no receipt, and optionally that the relay still runs. It returns RestoredLaunch, or NotResumable(reason) with one of supervisor_gone, identity_mismatch, pause_mismatch, launch_finished, relay_gone, relay_unavailable, check_failed.
  • SandboxSession.adopt(...) continues that pause, polls for the supervisor's receipt, then stops, collects and releases the launch as execute does. Later checkpoints pause the same supervisor from the next pause number.
  • abandon_launch(...) (and the supervisor's abandon mode) ends a launch that will not be taken over: KILL, never CONT, to the supervisor and everything it launched, then an abandoned receipt. A harness woken later would otherwise call a model server that no longer exists.

The wall clock is the provider's to keep. Nothing here sets a restored sandbox's clock. On AgentENV, snapshots.keep_guest_clock (#5410) makes a restored sandbox continue from the clock its snapshot was taken at.

3. Launches that lose their connection, and the relay's start wait

  • A launch whose connection drops after its supervisor started is finished from the supervisor's cleanup receipt, as a take-over after a restore is. The harness keeps running, and checkpoints still pause it. A launch whose supervisor never started still fails at once.
  • A receipt poll that raises counts as one failed poll instead of ending the wait, so one dropped poll during a take-over no longer relaunches the harness. A sandbox that is really gone still fails after five polls.
  • Relay start wait. The runner counts only its own running time against the relay's 10 s start wait. A checkpoint taken inside the wait stops the sandbox, and a restore that brought it back with the clock set forward used to exhaust the wait, even when the relay was already listening. The launch then ran without the relay and could not resume in place at the next restore.

4. A relay that runs on any image (last two commits)

The relay was first written in Python, so it ran only on task images with Python 3.6 or later; on others (PHP, Julia, Java and some JavaScript images) a launch ran without it and could not resume in place. It is now JavaScript, run by the harness's own binary, and the Python version is removed, so there is one implementation of the protocol. Its behavior and status file are unchanged, so the take-over below needed no change.

How it works

sequenceDiagram
  participant G as Restoring job (agent server)
  participant S as Supervisor (restored, holding pause N)
  participant R as Model relay (restored)
  participant H as Harness (restored, mid-call)
  participant M as New model server
  G->>S: check_restored(pause N, identity, relay)
  alt RestoredLaunch
    G->>R: rewrite endpoint file (new URL, generation plus 1)
    G->>S: continue N (adopt)
    R->>M: re-send the cut call, byte for byte
    M-->>R: response
    R-->>H: response (looks like a slow call)
    H-->>S: exits
    S-->>G: receipt, then collect and release
  else NotResumable(reason)
    G->>S: abandon launch (KILL, no CONT)
    Note over G: relaunch from the saved conversation, as before
  end
Loading

Where this sits in the stack

This stack checkpoints environment-server episodes whose agent harness runs in a sandbox the resources server lends. #5264 has merged, and the rest of the stack is rebased onto main after it. #5265 to #5331 add checkpoints for OpenCode and the SWE servers on OpenSandbox. #5410 to #5413 add AgentENV (E2B API, VM memory snapshots), where a restored sandbox still runs the harness's processes, so OpenCode continues in place instead of being relaunched. Each PR's diff shows only that PR's change.

  1. feat(checkpoint): protocol for harnesses that borrow a sandbox #5264: feat(checkpoint): protocol for harnesses that borrow a sandbox (merged)
  2. feat(sandbox): snapshot, fork and delete for OpenSandbox, with a checkpointer #5265: feat(sandbox): snapshot, fork and delete for OpenSandbox, with a checkpointer
  3. feat(swebench): checkpoint the task sandboxes of episode sessions #5266: feat(resources): sandbox owner base for resources servers
  4. feat(swebench): typed sessions on the owner base with capture-once verify #5267: feat(swebench): typed sessions on the owner base with capture-once verify
  5. feat(opencode): pause at checkpoints and resume after a crash #5268: feat(opencode): pause at checkpoints and resume after a crash
  6. test(checkpoint): end-to-end harness scenarios, SWE-bench recipe, docs #5269: test(checkpoint): end-to-end harness scenarios, SWE-bench recipe, docs
  7. feat(swe): repository-task servers on the sandbox owner base #5331: feat(swe): repository-task servers on the sandbox owner base
  8. feat(sandbox): AgentENV snapshots, lease and kill-at-timeout for the E2B provider #5410: feat(sandbox): AgentENV snapshots, lease and kill-at-timeout for the E2B provider
  9. feat(checkpoint): hold-pause a harness when the sandbox snapshot keeps its processes #5411: feat(checkpoint): hold-pause a harness when the sandbox snapshot keeps its processes
  10. feat(agent-utils): resume a supervised harness in place after a restore that kept its processes #5412: feat(agent-utils): resume a supervised harness in place after a restore that kept its processes (this PR)
  11. feat(opencode): resume OpenCode in place on sandboxes whose snapshots keep processes #5413: feat(opencode): resume OpenCode in place on sandboxes whose snapshots keep processes

Issue

No tracking issue exists. This is the core of AgentENV in-place resume; #5413 applies it to OpenCode.

Validation

  • Unit tests for the model relay (run as a real process on a JavaScript runtime against fake upstreams), check_restored for every reason, adopt, abandon_launch, and the relay start and fallback; existing suites pass; lint is clean.
  • Live tests against AgentENV of in-place resume after a simulated crash, with real services and with real OpenCode through the relay: calls cut before and after headers, mid-tool, relay killed, dead supervisor. The relay's behavior checked on OpenCode's own binary in AgentENV sandboxes built from images with and without Python.
  • Rebased onto main after feat(checkpoint): protocol for harnesses that borrow a sandbox #5264 merged: the unit, server and process-level checkpoint suites fail only where main itself does, apart from intermittent failures in tests that pass when run alone or that fail the same way on main.
  • No training run yet on the stack as it is now. The results below are from RL training runs on AgentENV with the previous version of the stack, which set the clock back inside the sandbox where this version relies on the provider (feat(sandbox): AgentENV snapshots, lease and kill-at-timeout for the E2B provider #5410):
    • Checkpoint and resume at 8 and 16 nodes: checkpoints committed at scale, sessions resumed in place, no corrupted token history, and training continued.
    • Two 16-node runs, a checkpoint run and a restore from its checkpoint, committed every checkpoint with no sandbox loss; the restore resumed every held session in place, on images with or without Python.
    • No restored launch failed its take-over.
    • With the model relay on OpenCode's own binary, a 16-node run on the five-route SWE blend and a restore from its checkpoint resumed every restored launch in place, including those on images without Python; none fell back to a relaunch, and the first checkpoint after the restore committed as fast as before it.

Compatibility

  • Additive. Nothing here runs unless a harness asks for it (relay params, check_restored, adopt). OpenSandbox runs and existing harnesses are unchanged.
  • Wall clock: Gym does not change a restored sandbox's clock; see snapshots.keep_guest_clock in feat(sandbox): AgentENV snapshots, lease and kill-at-timeout for the E2B provider #5410 for AgentENV.
  • Any task image runs the relay: it needs only the harness's Bun binary, or Node. A harness that gives no relay command, or whose relay does not start, calls the model directly and falls back to a relaunch on restore, without losing rollouts.
  • The runner no longer looks for a Python: MODEL_RELAY_COMMAND replaces MODEL_RELAY_SCRIPT and MODEL_RELAY_PYTHONS.
  • A launch command that raises after its supervisor started now waits for the supervisor's receipt instead of failing; one whose supervisor never started fails as before.

🤖 Generated with Claude Code

@pthombre pthombre added feature New capabilities, enhancements, or enablement work area:agent Agent harnesses and Responses API agent behavior complexity:high Cross-component or shared-contract change with a large review surface labels Oct 31, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 31, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@pthombre

pthombre commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Being rebased onto current main, after #5264 and #5265 (already rebased). Until this branch is pushed, its diff still shows their old commits. Not ready for review yet.

@pthombre
pthombre force-pushed the pranav/envckpt-9-resume-in-place branch 2 times, most recently from e170dfe to bc5892f Compare November 12, 2026 20:35
pthombre and others added 2 commits November 15, 2026 05:55
…re that kept its processes

After a restore of a snapshot that kept the sandbox's processes, the launch the crashed agent server started may
still be running in the fork, with its supervisor holding the checkpoint's pause. Three pieces let an adapter use
it instead of relaunching:

- `check_restored(sandbox, session_dir, pause_seq=..., identity=..., relay_status_path=..., timeout=...)` checks,
  in one command, that the supervisor still runs as the identity the checkpoint recorded, holds that pause in
  hold mode with no continue yet, has written no receipt, and (optionally) that the model relay still runs. It
  returns `RestoredLaunch`, or `NotResumable` with one of a fixed set of reasons.
- `SandboxSession.adopt(pause_seq=..., collect=..., timeout=..., close_timeout=...)` continues that pause and
  waits for the supervisor's receipt, then stops, collects and releases as `execute` does; `run_until_parked`
  drives it like a launch, and later checkpoints pause the same supervisor from the next number. The process
  that started the launch died with the agent server, so the wait polls for the receipt in short commands rather
  than holding one command open across a provider's stream limits.
- `process_supervisor.sh --abandon PID.START` and `abandon_launch(...)` end such a launch for good when it will
  not be taken over: KILL, never CONT, to the supervisor, the shell that started it, every process carrying its
  marker and their descendants, then a stopped, `abandoned` receipt. A woken harness would otherwise resume
  against a model server that no longer exists.

`execute` and `adopt` share the wait-finish-collect path (`_run_launch`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
…cross a sandbox restore

A sandbox restored from a memory snapshot still runs the harness, whose model client points at the model server of
the job that took the snapshot. model_relay.py runs beside the harness and forwards its model calls to the upstream
named in an endpoint file, so the restoring job can re-point them by writing the file:

- a request for /v1/<rest> goes to <upstream>/<rest>, with the harness's method, headers and body;
- while the supervisor's pause request is ahead of its continue request, new calls wait unsent;
- response headers reach the harness only with the first body byte, so a call whose response has not started is
  sent again, byte for byte, when the endpoint's generation changes; a started response is cut with a reset;
- an unreachable upstream is retried with backoff until --hold-timeout-s, then answered with a 502;
- a status file names the relay's pid, start time, port and counters.

The relay needs only Python 3.6 and its standard library. harness_runner.sh starts it when the params ask for one
(MODEL_RELAY_PORT, _ENDPOINT, _STATUS, _SCRIPT), on the first Python 3.6+ through a neutrally named interpreter link
so python-matching kills skip it, restarts it on the same port if it dies, and exports NEMO_GYM_MODEL_RELAY=1.
Without a usable interpreter it writes "unavailable" beside the status file and exports MODEL_RELAY_FALLBACK_ENV
instead, so the harness can call the model server directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
pthombre and others added 5 commits November 15, 2026 05:55
…rt wait

harness_runner.sh waited for the relay with a wall-clock deadline (bash SECONDS) and checked it before the
status file. A checkpoint taken inside that wait stops the sandbox, and a sandbox restored from its snapshot can
come back with the wall clock set forward. A jump of more than 10 s made the runner give up as soon as it ran
again, even when the relay was already listening, and run the command on MODEL_RELAY_FALLBACK_ENV: the launch
could no longer be resumed in place at the next checkpoint. A runner stopped that long by a signal did the same.

The wait now counts the runner's own running time from /proc/uptime (SECONDS without it), a gap between two polls
counts as at most a second, and the status file is read before the wait is checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
…ervisor's receipt

The command that starts a supervised launch holds the provider's output stream open for the whole run (8490 s for
OpenCode with checkpoints). When that stream drops partway (job 8552180: 21 launches), the launch failed: the session
stopped a healthy harness and the rollout started over.

When the launch command raises and the supervisor has claimed the launch (its pid or launch claim exists), the
session now logs "launch connection lost ...; waiting for its cleanup receipt" and waits for the receipt, as `adopt`
does after a restore. The receipt's harness status is the launch's result, and checkpoints still pause and continue
it through the supervisor's files. A launch whose supervisor never started still fails at once.

A receipt poll whose command raises now counts as one failed poll, like a poll that returned an error: it used to
end the wait, so a restored launch was relaunched instead of resumed (`adopt_failed`). A sandbox that is really gone
still fails after five polls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
…l as on Python

model_relay.mjs implements model_relay.py's protocol with node: modules only: the
same arguments, endpoint and pause files, hold, re-send, first-byte gating,
backoff and 502, and the same status-file layout, which the runner and
supervisor read with patterns. A harness built on Bun runs it on its own binary,
so an image needs no Python for a launch to resume in place.

harness_runner.sh takes MODEL_RELAY_COMMAND, a program and its leading
arguments, to run the relay instead of the first Python 3.6+ found, and
MODEL_RELAY_COMMAND_ENV for the relay's process alone. The neutral link name,
the watchdog, the start wait and the fallback are unchanged.

The relay tests run every behavior against both implementations; the
JavaScript ones need node or bun (NEMO_GYM_TEST_JS_RUNTIME) and skip without.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
…p the Python one

The Python relay needed a Python 3.6+ in the task image; model_relay.mjs does the
same on the harness's own Bun binary, which every image has, so a second copy of
the protocol would only need keeping in step. harness_runner.sh now starts the
relay only with MODEL_RELAY_COMMAND; the interpreter search, MODEL_RELAY_SCRIPT
and MODEL_RELAY_PYTHONS go. The relay's tests run on Node or Bun, and the
runner's on a stand-in relay that writes the status layout, so they need neither.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
…l a failed check from a lost launch

Found in a review of resuming a harness in place.

- `check_restored` runs through `exec_idempotent` and returns `check_failed` when the command still raises: a
  sandbox that refuses its first commands after a restore no longer fails the restore or reads as a lost launch.
- `adopt` repeats an unconfirmed continue up to three times, under the control lock; the request is the same each
  time. It writes an expiry that was requested before the take-over, before it continues the pause.
- The receipt wait ends on failed polls only after five in a row that span a minute, and never while a checkpoint
  holds the harness: a gateway restart or a snapshot no longer ends a running launch. A poll that returns at once
  is followed by a pause. The wait returns what the launch command returns: 0 when the receipt confirms the
  cleanup, else 1.
- The relay races a call against the endpoint change through a promise without a value, so a long call's settled
  results are not kept until the endpoint changes. It logs the first failure of an upstream it retries, and a
  response that fails after it started aborts the client connection, with a warning, instead of ending it as if
  complete.
- Tests: the worker of the resume tests runs until told to finish, so no test depends on a three-second window;
  bounded waits in the relay tests, a kill after a stop that times out, a `NEMO_GYM_TEST_JS_RUNTIME` that names no
  executable fails. New: a launch restored a second time, a cancelled adopted launch, the relay's restore order,
  an upstream that resets or drops a response, the relay's status file read by `check_restored`, and each fix
  above.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
@pthombre
pthombre force-pushed the pranav/envckpt-9-resume-in-place branch from bc5892f to f1f9f47 Compare November 16, 2026 21:44

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant