Skip to content
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
923b105
feat(agent): stop a running rollout as a timeout
gwarmstrong Oct 26, 2026
40b6b94
Merge main into georgea/gym-rollout-stop
gwarmstrong Oct 27, 2026
21744ea
feat(agent): mark a stopped rollout with ng_rollout_stop instead of a…
gwarmstrong Oct 27, 2026
d76b7ba
fix(agent): route a rollout stop to the worker running the rollout
gwarmstrong Oct 27, 2026
1622c37
fix(agent): write the expiry marker as a file within the stop timeout
gwarmstrong Oct 27, 2026
5eb5621
fix(codex): grade a run stopped before its worker wrote events
gwarmstrong Oct 27, 2026
ed95769
docs(agent): say a rollout stop needs no session cookie
gwarmstrong Oct 27, 2026
2fdb28c
fix(codex): keep the empty transcript for a stop only when none was w…
gwarmstrong Oct 27, 2026
c29a0af
docs(agent): scope a rollout stop to the current agent session
gwarmstrong Oct 27, 2026
c6302a9
fix(agent): keep a stop whose marker landed though its write failed
gwarmstrong Oct 27, 2026
6c07428
test(agent): bound the expiry tests' launch waits
gwarmstrong Oct 27, 2026
6ddb7cd
Merge main into georgea/gym-rollout-stop
gwarmstrong Oct 28, 2026
cad493c
fix(agent): bound the stop confirmation and await an unfinished marke…
gwarmstrong Oct 28, 2026
92dce12
Merge remote-tracking branch 'upstream/main' into georgea/gym-rollout…
gwarmstrong Oct 28, 2026
a093a69
fix(agent): keep a stop deliverable when its request disconnects
gwarmstrong Oct 29, 2026
2b71260
fix(agent): stop a running harness only where its upload cannot wait …
gwarmstrong Oct 29, 2026
f4b3839
Merge remote-tracking branch 'upstream/main' into georgea/gym-rollout…
gwarmstrong Oct 30, 2026
7f6535a
Merge remote-tracking branch 'upstream/main' into georgea/gym-rollout…
gwarmstrong Oct 30, 2026
417f7f1
Merge remote-tracking branch 'upstream/main' into georgea/gym-rollout…
gwarmstrong Oct 30, 2026
caacdbd
Merge remote-tracking branch 'upstream/main' into georgea/gym-rollout…
gwarmstrong Oct 30, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions environment_servers/single_agent_turn/tests/test_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -1523,6 +1523,30 @@ async def test_incomplete_context_limit_verifies_partial_work(action: str) -> No
assert verify["json"]["response"] == response


@pytest.mark.parametrize("action", ["fail", "retry"])
async def test_a_stopped_run_is_verified_and_its_result_keeps_the_stop_marker(action: str) -> None:
from nemo_gym.base_responses_api_agent import ROLLOUT_STOP_METADATA_KEY

environment, client = _environment_server()
environment.config.agent_outcome_policy = AgentOutcomePolicyConfig(default_action=action)
stopped = _agent_response().model_dump(mode="json")
stopped.update(status="incomplete", metadata={ROLLOUT_STOP_METADATA_KEY: '{"reason":"budget"}'})
client.responses[2] = _Response(stopped)
verification = orjson.loads(client.responses[4].body)
verification.update(response=stopped)
client.responses[4] = _Response(verification)

result = await environment.run_request(_request())

# A stop is not a failure: no failure policy applies, and the graded result carries the marker.
assert result.failure is None
assert result.result.reward == 1.0
verify = next(kwargs for _, path, kwargs in client.calls if path == "/verify")
assert verify["json"]["response"] == stopped
assert result.result.response.status == "incomplete"
assert result.result.response.metadata == {ROLLOUT_STOP_METADATA_KEY: '{"reason":"budget"}'}


@pytest.mark.parametrize("raw", ["not valid JSON", '{"failure_kind":"unregistered_error"}'])
async def test_invalid_outcome_is_retained_without_grading_or_retry(raw: str) -> None:
from nemo_gym.agent_outcome import AGENT_OUTCOME_METADATA_KEY
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,9 @@ The Environment Server drives an agent session through three HTTP endpoints. `Si
| `POST /v1/agent_sessions` | `AgentSeedSessionRequest` | `AgentSeedSessionResponse` and an `agent_session_id` session cookie | `_seed_agent_session_state()` |
| `POST /ng-rollout/<capture_key>[/training-token-capture]/v1/responses` | The task's `responses_create_params` | `NeMoGymResponse` | Session branch of `responses()` |
| `POST /v1/agent_sessions/close` | `AgentCloseSessionRequest` | `AgentCloseSessionResponse` | `_close_agent_session_state()` |
| `POST /rollouts/<capture_key>/stop` (optional) | `RolloutStopRequest` (an optional `reason`) | `RolloutStopResponse` | `_rollout_sandbox_session()` |

The stop route ends a running harness exactly as at its time limit, so the episode is still verified and graded as a timeout. It is for a caller that enforces its own budget (agent time, tokens, cost) or for an operator; cancelling the `/run` request instead discards the rollout. An agent whose harness runs in a `SandboxSession` supports it by returning that session from `_rollout_sandbox_session()` and passing its response through `report_rollout_stop()`, as Codex does; other agents answer HTTP 501. The stopped run's response keeps the timeout's `incomplete` status, which `single_agent_turn` verifies, and its metadata gains `ng_rollout_stop`, a JSON object with the caller's `reason` (or `null`). That marker tells a stop apart from a timeout or a cancellation. A stop is not a failure, so the response carries no `ng_agent_outcome` failure kind, and a rollout that was stopped and graded is an ordinary result. The caller keeps awaiting its `/run` request, which returns the graded result once verification finishes. Harness runners see the timeout path, not a cancellation: they find no `stop.request` marker. `stopped: true` means the stop was accepted; the run can still finish on its own before the stop takes effect. `stopped` is false when the run already ended or a stop was already requested. An unknown or closed rollout answers HTTP 404, and a stop that arrives while the session is still seeding takes effect when the harness would launch. Stopping a running harness writes a file its supervisor watches for, so it needs a sandbox provider whose uploads never wait for the capacity running commands hold (`SupportsSandboxIndependentUploads`): Local, and OpenSandbox with `operations.background_exec: true`. On other providers, such as Docker, Apptainer, Enroot and OpenShell, whose uploads share the command limit, and OpenSandbox in foreground mode, a stop of a running harness answers HTTP 501; a stop before launch works on every provider. With several server workers the request needs no session cookie: the worker running the rollout claims its `/rollouts/<capture_key>` path, and the others forward the stop to it. A stop applies to the rollout's current agent session. An environment that opens a new session for each turn, such as Harbor's multi-step tasks, still schedules the later turns, and a stop sent between sessions finds no open session and answers HTTP 404. A caller enforcing a budget for the whole episode must also keep the environment from starting further turns once the current one is graded.

`AgentSeedSessionRequest` carries `agent_session_id`, `episode_id`, `task_id`, `tool_accesses`, and an optional `sandbox_access`. `AgentCloseSessionResponse` returns `agent_observations`, an optional `trajectory` (the harness's own turns, which the Environment Server forwards as `ng_trajectory` on the result and on a failure row alike), and an optional `resources_cookies` jar.

Expand Down
19 changes: 16 additions & 3 deletions nemo_gym/agent_utils/process_supervisor.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,18 @@
# scans after dropping the marker from its environment. The receipt confirms that no reached
# process can still run; it is not a sandbox-empty check.
#
# usage: process_supervisor.sh --timeout SECONDS --cleanup-timeout SECONDS --receipt PATH --stop-file PATH -- COMMAND...
# The expire file, when given, moves the deadline to now: once it appears, the harness ends exactly
# as at its time limit and the receipt records timed_out. If it appears before launch, the harness
# never starts. The supervisor never deletes it.
#
# usage: process_supervisor.sh --timeout SECONDS --cleanup-timeout SECONDS --receipt PATH --stop-file PATH
# [--expire-file PATH] -- COMMAND...

# An inherited BASH_ENV or SHELLOPTS must not turn an expected failure into a missing receipt.
set +o errexit +o nounset +o pipefail

usage() {
echo "usage: process_supervisor.sh --timeout S --cleanup-timeout S --receipt PATH --stop-file PATH -- COMMAND..." >&2
echo "usage: process_supervisor.sh --timeout S --cleanup-timeout S --receipt PATH --stop-file PATH [--expire-file PATH] -- COMMAND..." >&2
exit 2
}

Expand Down Expand Up @@ -177,6 +182,8 @@ reached_alive() {

stopped() { ((stopping)) || [[ -e $stop_file ]]; }

expired() { [[ -n $expire_file && -e $expire_file ]]; }

write_receipt() {
printf '{"cleanup_confirmed": %s, "return_code": %s, "error": %s, "timed_out": %s}' \
"$1" "$return_code" "$2" "$timed_out" >"$receipt.tmp" && mv -f "$receipt.tmp" "$receipt"
Expand All @@ -193,6 +200,7 @@ while (($#)) && [[ $1 != -- ]]; do
--cleanup-timeout) cleanup=$(centiseconds "$2") || { echo "--cleanup-timeout must be finite and positive" >&2; exit 2; } ;;
--receipt) receipt=$2 ;;
--stop-file) stop_file=$2 ;;
--expire-file) expire_file=$2 ;;
*) echo "unexpected argument: $1" >&2; exit 2 ;;
esac
shift 2
Expand Down Expand Up @@ -223,6 +231,11 @@ if stopped; then
write_receipt true null # Nothing was launched.
exit 0
fi
if expired; then
timed_out=true
write_receipt true null # The deadline arrived before launch, so nothing was launched.
exit 0
fi
if ! now || ! stat_fields $$; then
write_receipt true '"Sandbox process supervision requires a readable /proc"' # Nothing was launched.
exit 1
Expand All @@ -240,7 +253,7 @@ stat_fields "$harness" && groups[harness]=${fields[19]} tracked[harness]=${field

now
deadline=$((NOW + timeout))
while kill -0 "$harness" 2>/dev/null && ! stopped && ((NOW < deadline)); do
while kill -0 "$harness" 2>/dev/null && ! stopped && ! expired && ((NOW < deadline)); do
sleep 0.05
now
done
Expand Down
122 changes: 121 additions & 1 deletion nemo_gym/agent_utils/sandbox_session.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from typing import cast

from nemo_gym.agent_utils.supervisor_client import (
EXPIRE_REQUEST_FILE,
OUTPUT_LOG_FILE,
STOP_REQUEST_FILE,
SUPERVISOR_FILE,
Expand All @@ -27,10 +28,12 @@
)
from nemo_gym.sandbox.api import AsyncSandbox
from nemo_gym.sandbox.providers.base import SandboxExecResult
from nemo_gym.sandbox.utils import read_text
from nemo_gym.sandbox.utils import read_text, upload_text


LOG = logging.getLogger(__name__)


# Shared runner uploaded by adapters; reads the bash source written by shell_assignments().
HARNESS_RUNNER_FILE = "harness_runner.sh"
HARNESS_RUNNER_SCRIPT = Path(__file__).with_name(HARNESS_RUNNER_FILE)
Expand Down Expand Up @@ -98,6 +101,10 @@ def word(item: object) -> str:
return "".join(f"{line}\n" for line in lines)


class ExpiryUnsupportedError(Exception):
"""``expire`` cannot stop this running harness: its sandbox's uploads can wait for capacity the harness holds."""


@dataclass(kw_only=True)
class SandboxSession[Artifacts]:
"""Execute, stop, capture, then release an owned or borrowed sandbox.
Expand Down Expand Up @@ -148,6 +155,13 @@ class SandboxSession[Artifacts]:
_close_task: asyncio.Task[None] | None = field(default=None, init=False)
_collect: Callable[[], Awaitable[Artifacts]] | None = field(default=None, init=False)
_capture_attempted: bool = field(default=False, init=False)
_expire_requested: bool = field(default=False, init=False)
# The caller's reason for the expiry ``expire`` requested, if it gave one.
expire_reason: str | None = field(default=None, init=False)
# An expiry whose marker write failed, possibly after the marker reached the sandbox, and its reason.
_unconfirmed_expiry: bool = field(default=False, init=False)
_unconfirmed_expire_reason: str | None = field(default=None, init=False)
_expire_write: asyncio.Task[None] | None = field(default=None, init=False)

@property
def closing(self) -> bool:
Expand All @@ -173,6 +187,16 @@ def stop_request_path(self) -> str:
"""Marker path an adapter may pass to its harness for cancellation diagnostics."""
return f"{self.session_dir}/{STOP_REQUEST_FILE}"

@property
def expired(self) -> bool:
"""Whether ``expire`` ended this activation; its receipt then records the timeout."""
return self._expire_requested and bool(self.cleanup and self.cleanup["timed_out"])

@property
def expired_before_launch(self) -> bool:
"""Whether ``expire`` kept the harness from starting, so it left no output to collect."""
return self.expired and self.cleanup["return_code"] is None and self.cleanup["error"] is None

async def read_output_log(self) -> str:
"""Read combined supervisor/harness diagnostics without masking the original failure."""
try:
Expand Down Expand Up @@ -209,6 +233,8 @@ async def execute(
try:
async with asyncio.timeout_at(deadline):
command = await asyncio.shield(self._stage_task)
if self._expire_requested:
await self._request_expiry(timeout=close_timeout)
if self.closing or self._finalize_task is not None:
raise RuntimeError(f"{self.harness} sandbox session closed before launch")
if deadline is not None:
Expand Down Expand Up @@ -274,6 +300,11 @@ async def begin_activation(self, *, session_dir: str, timeout: float) -> None:
raise RuntimeError(f"{self.harness} sandbox session is closing")
if not self.cleanup_confirmed:
raise RuntimeError(f"{self.harness} previous activation cleanup is unconfirmed")
# An expiry applies to the activation it was requested for; one requested before the first launch stays.
self._expire_requested = False
self.expire_reason = None
self._unconfirmed_expiry = False
self._expire_write = None
if self._session_root is None:
self._session_root = self.session_dir
self.session_dir = session_dir
Expand All @@ -287,6 +318,61 @@ async def begin_activation(self, *, session_dir: str, timeout: float) -> None:
self._collect = None
self._capture_attempted = False

async def expire(self, *, timeout: float, reason: str | None = None) -> bool:
"""End the harness as at its time limit, so its receipt records ``timed_out``.

Unlike ``stop_harness`` (a cancellation), the activation is then graded like a timeout.
A request made before launch ends the harness as it would launch. Returns False when the
activation already finished, the session is closing, or expiry was already requested.

Stopping a running harness takes an upload, which on a provider without
``SupportsSandboxIndependentUploads`` can wait behind the very commands it must stop, so there it
raises ``ExpiryUnsupportedError``. A request before launch works on every provider.
"""
if self.closing or self.sandbox_stopped or self._expire_requested or self.cleanup is not None:
return False
if self._exec_task is not None and self._exec_task.done():
return False
if self.launch_started and not self.sandbox.uploads_independent_of_execs:
raise ExpiryUnsupportedError(
f"the {self.harness} sandbox's provider cannot stop a running harness: its file uploads can wait "
"for the capacity running commands hold"
)
self._expire_requested = True
self.expire_reason = reason
if self.launch_started:
# A task, so that finalization can wait for a write still in flight before it checks the marker.
self._expire_write = asyncio.create_task(self._write_expiry(timeout=timeout, reason=reason))
# Shielded, so a caller that stops waiting (a disconnected stop request) cannot cancel the write before
# it starts and leave the stop requested but never delivered. The write is bounded by ``timeout`` and,
# if it fails, makes the stop retryable again.
await asyncio.shield(self._expire_write)
return True

async def _write_expiry(self, *, timeout: float, reason: str | None) -> None:
try:
await self._request_expiry(timeout=timeout)
except BaseException:
self._expire_requested = False
self.expire_reason = None
# The marker may have landed before the write failed; finalization keeps the stop only if it did.
self._unconfirmed_expiry, self._unconfirmed_expire_reason = True, reason
raise

async def _request_expiry(self, *, timeout: float) -> None:
"""Write the marker the supervisor watches, within ``timeout`` including any wait for the provider.

A file write, not an exec: a provider that bounds its concurrent execs (Local) counts the harness's
own launch against that bound for as long as it runs, and a stop must not wait for the harness to end.
"""
try:
async with asyncio.timeout(timeout):
await upload_text(self.sandbox, path=f"{self.session_dir}/{EXPIRE_REQUEST_FILE}", text="")
except TimeoutError as error:
raise RuntimeError(f"Could not request {self.harness} expiry within {timeout:g} s") from error
except Exception as error:
raise RuntimeError(f"Could not request {self.harness} expiry: {error}") from error

async def stop_harness(self, *, timeout: float) -> None:
"""Confirm harness termination before capture and transport release."""
if self.sandbox_stopped or not self.launch_started or self.cleanup is not None:
Expand Down Expand Up @@ -330,6 +416,7 @@ async def _finalize(self, *, timeout: float) -> None:
if self._stage_task is not None:
await self._cancel_and_wait(self._stage_task, timeout=timeout)
await self.stop_harness(timeout=timeout)
await self._confirm_unacknowledged_expiry(timeout=timeout)
if self.launch_started and not self._capture_attempted and self._collect is not None:
if self.sandbox_stopped:
self.capture_error = RuntimeError(f"{self.harness} sandbox stopped before artifact capture")
Expand All @@ -344,6 +431,39 @@ async def _finalize(self, *, timeout: float) -> None:
self._capture_attempted = True
await self._release_exec(timeout=timeout)

async def _confirm_unacknowledged_expiry(self, *, timeout: float) -> None:
"""Keep a stop whose marker write failed only if the marker is in place and the receipt timed out.

A write still in flight is awaited first, so a failure that arrives during finalization is checked too.
The whole check, including any wait for provider capacity, is bounded by ``timeout``; without a
confirmation the stop stays unattributed and finalization goes on. That includes a write still in flight
when the bound runs out: the harness may have ended at its own deadline, before any marker arrived.
"""
try:
async with asyncio.timeout(timeout):
if self._expire_write is not None and not self._expire_write.done():
await asyncio.wait({self._expire_write})
if (
not self._unconfirmed_expiry
or self._expire_requested
or not (self.cleanup and self.cleanup["timed_out"])
):
return
self._unconfirmed_expiry = False
marker = quote(f"{self.session_dir}/{EXPIRE_REQUEST_FILE}")
probe = await self.sandbox.exec(
f"[ -e {marker} ] && echo present", cwd=self.workdir, timeout_s=timeout
)
except Exception:
if self._expire_write is not None and not self._expire_write.done():
self._expire_requested, self.expire_reason = False, None
LOG.warning(
"Could not confirm the %s expiry marker; the stop stays unattributed", self.harness, exc_info=True
)
return
if probe.return_code == 0 and not probe.error_type and probe.stdout.strip() == "present":
self._expire_requested, self.expire_reason = True, self._unconfirmed_expire_reason

async def _release_exec(self, *, timeout: float) -> None:
if (
self.cleanup is not None or self.termination_error is not None or self.sandbox_stopped
Expand Down
3 changes: 3 additions & 0 deletions nemo_gym/agent_utils/supervisor_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
launch.claim Launch or stop shell Atomically claims launch or fences it
supervisor.pid Launch shell After claiming launch, before exec (see below)
stop.request Stop shell Polled by the supervisor to stop the harness
expire.request Session Polled by the supervisor to time out the harness
cleanup.json Supervisor or stop After cleanup, or when stop fences launch
output.log Launch shell Captures supervisor and harness diagnostics
===================== ==================== ===========================================
Expand Down Expand Up @@ -52,6 +53,7 @@
LAUNCH_CLAIM_FILE = "launch.claim"
SUPERVISOR_PID_FILE = "supervisor.pid"
STOP_REQUEST_FILE = "stop.request"
EXPIRE_REQUEST_FILE = "expire.request"
CLEANUP_RECEIPT_FILE = "cleanup.json"
OUTPUT_LOG_FILE = "output.log"

Expand Down Expand Up @@ -201,6 +203,7 @@ def supervised_launch_command(
f"exec bash {supervisor} "
f"--timeout {_seconds(timeout)} --cleanup-timeout {_seconds(cleanup_timeout)} "
f"--stop-file {quote(f'{session_dir}/{STOP_REQUEST_FILE}')} "
f"--expire-file {quote(f'{session_dir}/{EXPIRE_REQUEST_FILE}')} "
f"--receipt {quote(f'{session_dir}/{CLEANUP_RECEIPT_FILE}')} -- {harness} "
f">{quote(f'{session_dir}/{OUTPUT_LOG_FILE}')} 2>&1"
)
Expand Down
Loading
Loading