Repository navigation
feat(agent-utils): resume a supervised harness in place after a restore that kept its processes - #5412
Draft
pthombre wants to merge 7 commits into
Draft
feat(agent-utils): resume a supervised harness in place after a restore that kept its processes#5412pthombre wants to merge 7 commits into
pthombre wants to merge 7 commits into
Conversation
This was referenced Oct 31, 2026
pthombre
force-pushed
the
pranav/envckpt-9-resume-in-place
branch
2 times, most recently
from
November 2, 2026 06:16
b7223f1 to
11ee927
Compare
pthombre
force-pushed
the
pranav/envckpt-9-resume-in-place
branch
3 times, most recently
from
November 5, 2026 20:07
2adfe2d to
e9b23c4
Compare
Contributor
Author
pthombre
force-pushed
the
pranav/envckpt-9-resume-in-place
branch
2 times, most recently
from
November 12, 2026 20:35
e170dfe to
bc5892f
Compare
…re that kept its processes After a restore of a snapshot that kept the sandbox's processes, the launch the crashed agent server started may still be running in the fork, with its supervisor holding the checkpoint's pause. Three pieces let an adapter use it instead of relaunching: - `check_restored(sandbox, session_dir, pause_seq=..., identity=..., relay_status_path=..., timeout=...)` checks, in one command, that the supervisor still runs as the identity the checkpoint recorded, holds that pause in hold mode with no continue yet, has written no receipt, and (optionally) that the model relay still runs. It returns `RestoredLaunch`, or `NotResumable` with one of a fixed set of reasons. - `SandboxSession.adopt(pause_seq=..., collect=..., timeout=..., close_timeout=...)` continues that pause and waits for the supervisor's receipt, then stops, collects and releases as `execute` does; `run_until_parked` drives it like a launch, and later checkpoints pause the same supervisor from the next number. The process that started the launch died with the agent server, so the wait polls for the receipt in short commands rather than holding one command open across a provider's stream limits. - `process_supervisor.sh --abandon PID.START` and `abandon_launch(...)` end such a launch for good when it will not be taken over: KILL, never CONT, to the supervisor, the shell that started it, every process carrying its marker and their descendants, then a stopped, `abandoned` receipt. A woken harness would otherwise resume against a model server that no longer exists. `execute` and `adopt` share the wait-finish-collect path (`_run_launch`). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
…cross a sandbox restore A sandbox restored from a memory snapshot still runs the harness, whose model client points at the model server of the job that took the snapshot. model_relay.py runs beside the harness and forwards its model calls to the upstream named in an endpoint file, so the restoring job can re-point them by writing the file: - a request for /v1/<rest> goes to <upstream>/<rest>, with the harness's method, headers and body; - while the supervisor's pause request is ahead of its continue request, new calls wait unsent; - response headers reach the harness only with the first body byte, so a call whose response has not started is sent again, byte for byte, when the endpoint's generation changes; a started response is cut with a reset; - an unreachable upstream is retried with backoff until --hold-timeout-s, then answered with a 502; - a status file names the relay's pid, start time, port and counters. The relay needs only Python 3.6 and its standard library. harness_runner.sh starts it when the params ask for one (MODEL_RELAY_PORT, _ENDPOINT, _STATUS, _SCRIPT), on the first Python 3.6+ through a neutrally named interpreter link so python-matching kills skip it, restarts it on the same port if it dies, and exports NEMO_GYM_MODEL_RELAY=1. Without a usable interpreter it writes "unavailable" beside the status file and exports MODEL_RELAY_FALLBACK_ENV instead, so the harness can call the model server directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
…rt wait harness_runner.sh waited for the relay with a wall-clock deadline (bash SECONDS) and checked it before the status file. A checkpoint taken inside that wait stops the sandbox, and a sandbox restored from its snapshot can come back with the wall clock set forward. A jump of more than 10 s made the runner give up as soon as it ran again, even when the relay was already listening, and run the command on MODEL_RELAY_FALLBACK_ENV: the launch could no longer be resumed in place at the next checkpoint. A runner stopped that long by a signal did the same. The wait now counts the runner's own running time from /proc/uptime (SECONDS without it), a gap between two polls counts as at most a second, and the status file is read before the wait is checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
…ervisor's receipt The command that starts a supervised launch holds the provider's output stream open for the whole run (8490 s for OpenCode with checkpoints). When that stream drops partway (job 8552180: 21 launches), the launch failed: the session stopped a healthy harness and the rollout started over. When the launch command raises and the supervisor has claimed the launch (its pid or launch claim exists), the session now logs "launch connection lost ...; waiting for its cleanup receipt" and waits for the receipt, as `adopt` does after a restore. The receipt's harness status is the launch's result, and checkpoints still pause and continue it through the supervisor's files. A launch whose supervisor never started still fails at once. A receipt poll whose command raises now counts as one failed poll, like a poll that returned an error: it used to end the wait, so a restored launch was relaunched instead of resumed (`adopt_failed`). A sandbox that is really gone still fails after five polls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
…l as on Python model_relay.mjs implements model_relay.py's protocol with node: modules only: the same arguments, endpoint and pause files, hold, re-send, first-byte gating, backoff and 502, and the same status-file layout, which the runner and supervisor read with patterns. A harness built on Bun runs it on its own binary, so an image needs no Python for a launch to resume in place. harness_runner.sh takes MODEL_RELAY_COMMAND, a program and its leading arguments, to run the relay instead of the first Python 3.6+ found, and MODEL_RELAY_COMMAND_ENV for the relay's process alone. The neutral link name, the watchdog, the start wait and the fallback are unchanged. The relay tests run every behavior against both implementations; the JavaScript ones need node or bun (NEMO_GYM_TEST_JS_RUNTIME) and skip without. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
…p the Python one The Python relay needed a Python 3.6+ in the task image; model_relay.mjs does the same on the harness's own Bun binary, which every image has, so a second copy of the protocol would only need keeping in step. harness_runner.sh now starts the relay only with MODEL_RELAY_COMMAND; the interpreter search, MODEL_RELAY_SCRIPT and MODEL_RELAY_PYTHONS go. The relay's tests run on Node or Bun, and the runner's on a stand-in relay that writes the status layout, so they need neither. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
…l a failed check from a lost launch Found in a review of resuming a harness in place. - `check_restored` runs through `exec_idempotent` and returns `check_failed` when the command still raises: a sandbox that refuses its first commands after a restore no longer fails the restore or reads as a lost launch. - `adopt` repeats an unconfirmed continue up to three times, under the control lock; the request is the same each time. It writes an expiry that was requested before the take-over, before it continues the pause. - The receipt wait ends on failed polls only after five in a row that span a minute, and never while a checkpoint holds the harness: a gateway restart or a snapshot no longer ends a running launch. A poll that returns at once is followed by a pause. The wait returns what the launch command returns: 0 when the receipt confirms the cleanup, else 1. - The relay races a call against the endpoint change through a promise without a value, so a long call's settled results are not kept until the endpoint changes. It logs the first failure of an upstream it retries, and a response that fails after it started aborts the client connection, with a warning, instead of ending it as if complete. - Tests: the worker of the resume tests runs until told to finish, so no test depends on a three-second window; bounded waits in the relay tests, a kill after a stop that times out, a `NEMO_GYM_TEST_JS_RUNTIME` that names no executable fails. New: a launch restored a second time, a cancelled adopted launch, the relay's restore order, an upstream that resets or drops a response, the relay's status file read by `check_restored`, and each fix above. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Pranav Thombre <pthombre@nvidia.com>
pthombre
force-pushed
the
pranav/envckpt-9-resume-in-place
branch
from
November 16, 2026 21:44
bc5892f to
f1f9f47
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed and why
After a crash, the stack restores a sandbox from its snapshot and relaunches the harness from its saved conversation (#5268). On a provider whose snapshots keep processes (AgentENV, #5410), the restored sandbox is still running the harness: the supervisor holds the checkpoint's hold pause (#5411), and the harness is mid-task with its tools, background servers and in-memory state intact. Relaunching would throw that away and start a second harness beside the first.
Two things stand in the way of simply continuing it:
This PR adds the agent-utils pieces that solve both, for any harness under
SandboxSession. #5413 wires them into OpenCode.1. Model relay
The relay runs beside the harness inside the sandbox and forwards its model calls to the upstream named in an endpoint file, so the restoring job can re-point it by rewriting that file. It is JavaScript using only Node's standard modules, so a harness built on Bun runs it on its own binary (
BUN_BE_BUN=1), and Node runs it too: the relay needs nothing from the task image.The harness runner starts the relay with the command the harness gives (
MODEL_RELAY_COMMAND, with variables set for the relay's process alone), through a link with a neutral name that a task'spkillof python or of the harness does not match, and a watchdog restarts it on the same port. If the relay cannot start, the runner marks it unavailable and the harness calls the model server directly.2. Taking over a restored launch
check_restored(...)checks in one command that the supervisor runs with the recorded identity, holds that pause in hold mode with no continue and no receipt, and optionally that the relay still runs. It returnsRestoredLaunch, orNotResumable(reason)with one ofsupervisor_gone,identity_mismatch,pause_mismatch,launch_finished,relay_gone,relay_unavailable,check_failed.SandboxSession.adopt(...)continues that pause, polls for the supervisor's receipt, then stops, collects and releases the launch asexecutedoes. Later checkpoints pause the same supervisor from the next pause number.abandon_launch(...)(and the supervisor's abandon mode) ends a launch that will not be taken over: KILL, never CONT, to the supervisor and everything it launched, then anabandonedreceipt. A harness woken later would otherwise call a model server that no longer exists.The wall clock is the provider's to keep. Nothing here sets a restored sandbox's clock. On AgentENV,
snapshots.keep_guest_clock(#5410) makes a restored sandbox continue from the clock its snapshot was taken at.3. Launches that lose their connection, and the relay's start wait
4. A relay that runs on any image (last two commits)
The relay was first written in Python, so it ran only on task images with Python 3.6 or later; on others (PHP, Julia, Java and some JavaScript images) a launch ran without it and could not resume in place. It is now JavaScript, run by the harness's own binary, and the Python version is removed, so there is one implementation of the protocol. Its behavior and status file are unchanged, so the take-over below needed no change.
How it works
sequenceDiagram participant G as Restoring job (agent server) participant S as Supervisor (restored, holding pause N) participant R as Model relay (restored) participant H as Harness (restored, mid-call) participant M as New model server G->>S: check_restored(pause N, identity, relay) alt RestoredLaunch G->>R: rewrite endpoint file (new URL, generation plus 1) G->>S: continue N (adopt) R->>M: re-send the cut call, byte for byte M-->>R: response R-->>H: response (looks like a slow call) H-->>S: exits S-->>G: receipt, then collect and release else NotResumable(reason) G->>S: abandon launch (KILL, no CONT) Note over G: relaunch from the saved conversation, as before endWhere this sits in the stack
This stack checkpoints environment-server episodes whose agent harness runs in a sandbox the resources server lends. #5264 has merged, and the rest of the stack is rebased onto
mainafter it. #5265 to #5331 add checkpoints for OpenCode and the SWE servers on OpenSandbox. #5410 to #5413 add AgentENV (E2B API, VM memory snapshots), where a restored sandbox still runs the harness's processes, so OpenCode continues in place instead of being relaunched. Each PR's diff shows only that PR's change.Issue
No tracking issue exists. This is the core of AgentENV in-place resume; #5413 applies it to OpenCode.
Validation
check_restoredfor every reason,adopt,abandon_launch, and the relay start and fallback; existing suites pass; lint is clean.mainafter feat(checkpoint): protocol for harnesses that borrow a sandbox #5264 merged: the unit, server and process-level checkpoint suites fail only wheremainitself does, apart from intermittent failures in tests that pass when run alone or that fail the same way onmain.Compatibility
check_restored,adopt). OpenSandbox runs and existing harnesses are unchanged.snapshots.keep_guest_clockin feat(sandbox): AgentENV snapshots, lease and kill-at-timeout for the E2B provider #5410 for AgentENV.MODEL_RELAY_COMMANDreplacesMODEL_RELAY_SCRIPTandMODEL_RELAY_PYTHONS.🤖 Generated with Claude Code