Skip to content

Commit 8d8809e

Browse files
committed
fix(sandbox): drain in-flight tools and validate harness startup
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
1 parent 187d39a commit 8d8809e

30 files changed

Lines changed: 729 additions & 134 deletions

‎benchmarks/terminal_bench_2_1/README.md‎

Lines changed: 18 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -156,13 +156,19 @@ a successful smoke, and one task does not establish a benchmark baseline.
156156

157157
## Tasks that start services
158158

159-
The supervisor stops and reaps the harness process group before capture or verification.
160-
Shell-tool commands can run in separate sessions, so a confirmed receipt does not prove
161-
that all foreground tools have stopped after CLI exit or timeout. Those commands may
162-
still change task files during verification. Processes outside the group, including
163-
services from completed tool calls, remain until Resources closes the sandbox after
164-
verification or episode failure. Unconfirmed harness termination blocks verification;
165-
a final failure receipt still lets agent close release its connection for owner teardown.
159+
The supervisor stops live processes in the harness group and reaps its children before
160+
capture or verification. On timeout or close, it first records its already adopted
161+
children, typically services from completed tool calls. It then stops the harness and
162+
repeatedly drains newly adopted children, including in-flight tools in separate sessions,
163+
within the cleanup deadline. Earlier adopted services outside the harness group remain
164+
for Resources to stop after verification or episode failure.
165+
166+
Natural CLI exit has no such cutoff: unfinished tools in separate sessions cannot be
167+
distinguished from services and may still change task files. Adoption races, PID reuse,
168+
and a preserved service orphaning more children during cleanup also limit this rule.
169+
A confirmed receipt is not a guarantee that the sandbox is idle or a security boundary.
170+
Unconfirmed termination blocks verification; a final failure receipt still lets agent
171+
close release its connection for owner teardown.
166172

167173
Tool shells retain the task HOME and XDG directories, so home-directory deliverables
168174
remain available to verification. Harness configuration and transcripts stay in a private
@@ -178,6 +184,8 @@ A file-producing smoke does not validate this service-dependent behavior.
178184
Resources retains completed close receipts for `session_close_retry_window_seconds`
179185
(default 300 seconds); retries do not extend the window. Pending setup or failed stops
180186
retain their handles. Close fences an in-progress setup before waiting for its lock,
181-
so setup finishing after a close timeout tears its sandbox down. Shutdown attempts
182-
session stops concurrently, but process termination can interrupt them; the configured
183-
provider TTL remains the backstop for abandoned sandboxes.
187+
so setup finishing after a close timeout attempts its own bounded teardown. If that
188+
attempt fails after the final close callback is cancelled, there is no durable automatic
189+
retry. The server retains the fence and handle for an explicit close retry or shutdown;
190+
provider TTL can stop the abandoned sandbox but does not purge the in-memory record.
191+
Shutdown attempts session stops concurrently, but process termination can interrupt them.

‎fern/versions/latest/pages/agent-server/integrate-existing-agents.mdx‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ For Environment Server execution, implement these `SimpleResponsesAPIAgent` hook
1616
an `AgentSessionState`. If setup fails and cleanup cannot finish, raise
1717
`AgentSessionSetupError(state, error=original_error)` so close can retry cleanup.
1818
- `_close_agent_session_state(state)`: stop the harness process group, collect observations,
19-
and release the session. Explicitly detached task services remain for the sandbox owner.
19+
and release the session. Preserved task services remain for the sandbox owner.
2020
Missing termination evidence remains retryable; a final negative receipt fails the activation
2121
and skips verification while close releases the borrowed connection.
2222

@@ -31,6 +31,14 @@ provider's exec request. Cancellation can kill the provider's entire process gro
3131
the supervisor when it is already the group leader (as with the local provider).
3232
`setsid()` alone does not guarantee cleanup; the close ordering does.
3333

34+
The shared supervisor stops live harness-group processes and reaps its children. Before
35+
signalling a timed-out or closing harness, it records already adopted children, typically
36+
services from completed tools. It preserves those outside the harness group and drains
37+
newly adopted children within the cleanup deadline, including tools in separate sessions.
38+
After natural CLI exit, unfinished tools in separate sessions cannot be distinguished
39+
from services. Adoption races, PID reuse, and later service forks also limit this cutoff;
40+
it is not a security boundary or a guarantee that task state cannot change.
41+
3442
## Tools in Agent vs. Resources Server
3543

3644
Existing agents may come with predefined tools, allowing you to leverage them directly and use the Resources server to supplement with any additional external tools. When building a new environment, prefer defining tools in the Resources server rather than the Agent server. This separation of concerns allows different agents to share the same Resources server without duplicating tool logic.

‎nemo_gym/agent_utils/process_supervisor.py‎

Lines changed: 63 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,11 @@
44
"""Standalone Linux process supervisor to upload beside a sandboxed harness.
55
66
Uses only the standard library; Gym need not be installed in the task sandbox.
7-
The receipt confirms termination of the harness process group independently of
8-
its result. Processes that leave that group, including detached task services,
9-
belong to the sandbox owner and survive until sandbox teardown. Process groups
10-
cannot distinguish an escaped tool command from an intentional task service.
7+
The receipt confirms termination independently of the harness result. On forced
8+
stop, newly adopted children are also drained, including tools in other sessions.
9+
Detached services adopted before stopping remain for verification. Natural exit
10+
only drains the harness group: orphaned services and unfinished detached tools
11+
cannot be distinguished without cooperation from the harness.
1112
"""
1213

1314
from __future__ import annotations
@@ -32,9 +33,9 @@
3233
class CleanupReceipt(TypedDict):
3334
"""Outcome after bounded harness termination.
3435
35-
``cleanup_confirmed`` means the harness group has stopped and been reaped,
36-
not that the task sandbox is empty. Detached services remain until the owner
37-
closes the sandbox, including after a forced harness stop.
36+
``cleanup_confirmed`` means no live harness-group members remain. Forced
37+
stops also drain newly adopted children. Previously adopted detached services
38+
remain until the owner closes the sandbox; this is not a sandbox-empty check.
3839
"""
3940

4041
cleanup_confirmed: bool
@@ -52,27 +53,61 @@ def exec_timeout(*, timeout: float, cleanup_timeout: float = DEFAULT_CLEANUP_TIM
5253
return timeout + CLEANUP_PHASE_COUNT * cleanup_timeout + 30
5354

5455

55-
def _reap_process_group(pgid: int, timeout: float) -> None:
56-
"""Stop and reap this group without signalling detached task services."""
56+
def _child_pids() -> set[int]:
57+
"""Read this single-threaded subreaper's current children."""
58+
return {int(pid) for pid in Path(f"/proc/self/task/{os.getpid()}/children").read_text().split()}
59+
60+
61+
def _group_has_live_processes(pgid: int) -> bool:
62+
# killpg(..., 0) also succeeds for zombies parented outside this subreaper.
63+
# They cannot execute or write, and their owner alone can reap them. Any
64+
# unreadable/invalid proc state propagates instead of confirming termination.
65+
for directory in Path("/proc").iterdir():
66+
if not directory.name.isdigit():
67+
continue
68+
try:
69+
fields = (directory / "stat").read_text().rsplit(")", 1)[1].split()
70+
except (FileNotFoundError, ProcessLookupError):
71+
continue
72+
if int(fields[2]) == pgid and fields[0] not in {"Z", "X"}:
73+
return True
74+
return False
75+
76+
77+
def _reap_process_group(pgid: int, timeout: float, *, preserved_children: set[int] | None = None) -> None:
78+
"""Drain the group and, on forced stop, newly adopted tools within one budget."""
5779
deadline = time.monotonic() + timeout
5880
while True:
5981
try:
6082
os.killpg(pgid, signal.SIGKILL)
6183
except ProcessLookupError:
62-
return
63-
# As a subreaper we adopt the wrapper's orphaned children. Reap only
64-
# this group: a detached service may still have useful work to do.
84+
pass
85+
group_children_pending = True
6586
try:
6687
while os.waitpid(-pgid, os.WNOHANG)[0]:
6788
pass
6889
except ChildProcessError:
69-
pass
90+
group_children_pending = False
91+
pending_children = set()
92+
if preserved_children is not None:
93+
# Killing one generation can orphan more children in other groups.
94+
# Re-read each round rather than taking only one post-stop snapshot.
95+
for pid in _child_pids() - preserved_children:
96+
try:
97+
os.kill(pid, signal.SIGKILL)
98+
os.waitpid(pid, os.WNOHANG)
99+
except ProcessLookupError:
100+
pass
101+
pending_children = _child_pids() - preserved_children
70102
try:
71103
os.killpg(pgid, 0)
104+
group_alive = _group_has_live_processes(pgid)
72105
except ProcessLookupError:
106+
group_alive = False
107+
if not group_alive and not group_children_pending and not pending_children:
73108
return
74109
if time.monotonic() >= deadline:
75-
raise TimeoutError("Harness process group has not terminated")
110+
raise TimeoutError("Harness process group or newly adopted tools have not terminated")
76111
time.sleep(0.01)
77112

78113

@@ -83,10 +118,12 @@ def _supervise(
83118
cleanup_timeout: float = DEFAULT_CLEANUP_TIMEOUT,
84119
stop_path: Path | None = None,
85120
) -> CleanupReceipt:
86-
"""Stop the harness group within its deadline; preserve escaped task services."""
121+
"""Stop the harness and forced-stop tools; preserve earlier detached services."""
87122
process = None
88123
launch_attempted = False
89124
stopping = False
125+
forced_stop = False
126+
preserved_children = None
90127
receipt: CleanupReceipt = {"cleanup_confirmed": False, "return_code": None, "error": None, "timed_out": False}
91128

92129
def interrupt(*_: object) -> None:
@@ -117,7 +154,11 @@ def interrupt(*_: object) -> None:
117154
while process.poll() is None and not stopping and time.monotonic() < deadline:
118155
time.sleep(0.05)
119156
if process.poll() is None:
157+
forced_stop = True
120158
receipt["timed_out"] = not stopping
159+
# Completed tool calls can leave services already adopted by us.
160+
# Preserve those, but drain children orphaned by this forced stop.
161+
preserved_children = _child_pids() - {process.pid}
121162
# Let the harness checkpoint before the bounded hard cleanup.
122163
process.send_signal(signal.SIGTERM)
123164
try:
@@ -130,17 +171,18 @@ def interrupt(*_: object) -> None:
130171
signal.signal(signal.SIGTERM, signal.SIG_IGN)
131172
try:
132173
if process is not None:
133-
# Stop remaining group members before capture or verification.
134-
# Harness shell tools can start their own sessions, so this does
135-
# not confirm termination of all in-flight tools. Services from
136-
# completed tool calls may also live outside this group; group
137-
# membership alone cannot distinguish them from unfinished tools.
174+
# Any services still in the harness group stop here too. Only
175+
# previously adopted children outside that group are preserved
176+
# on forced stop; audit service-launching task recipes.
177+
# Natural exit has no pre-stop boundary for detached processes.
138178
try:
139179
os.killpg(process.pid, signal.SIGKILL)
140180
except ProcessLookupError:
141181
pass
142182
receipt["return_code"] = process.wait(timeout=cleanup_timeout)
143-
_reap_process_group(process.pid, cleanup_timeout)
183+
_reap_process_group(process.pid, cleanup_timeout, preserved_children=preserved_children)
184+
if forced_stop and preserved_children is None:
185+
raise RuntimeError("Cannot identify pre-stop adopted children")
144186
receipt["cleanup_confirmed"] = True
145187
else:
146188
# Popen can fail after spawning but before returning a handle.

‎resources_servers/terminal_bench_2_1/tests/test_sessions.py‎

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -518,16 +518,20 @@ async def test_expired_record_waiter_joins_replacement_lock(setup, monkeypatch:
518518
await old.lock.acquire()
519519
waiting = asyncio.create_task(server.seed_session(request, seed))
520520
try:
521-
await asyncio.sleep(0)
522-
assert not waiting.done()
523-
now = 301
524-
current = await server.seed_session(request, seed)
521+
try:
522+
await asyncio.sleep(0)
523+
assert not waiting.done()
524+
now = 301
525+
current = await asyncio.wait_for(server.seed_session(request, seed), timeout=5)
526+
finally:
527+
old.lock.release()
528+
assert await asyncio.wait_for(waiting, timeout=5) == current
529+
assert server._session_id_to_state[seed.resources_session_id] is not old
530+
server._create_sandbox.assert_awaited_once()
525531
finally:
526-
old.lock.release()
527-
assert await waiting == current
528-
assert server._session_id_to_state[seed.resources_session_id] is not old
529-
server._create_sandbox.assert_awaited_once()
530-
await server.close_resources_session(close_body(seed))
532+
waiting.cancel()
533+
await asyncio.gather(waiting, return_exceptions=True)
534+
await server.close_resources_session(close_body(seed))
531535

532536

533537
async def test_failed_close_keeps_fence_and_owner_past_retry_window(setup, monkeypatch: pytest.MonkeyPatch) -> None:

‎responses_api_agents/codex_agent/README.md‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -138,14 +138,16 @@ Missing worker identity or exit diagnostics become `runtime_info_unavailable` or
138138
`worker_exit_code_unavailable` observation gaps. Positive cleanup confirmation remains
139139
required; the adapter validates the captured terminal events without inventing an exit code.
140140

141-
One shared Linux supervisor per activation fences delayed launches and bounds harness
142-
termination and reaps its process group. Shell-tool commands can run in separate sessions
143-
and survive CLI exit or timeout; a confirmed receipt does not prove that those commands
144-
have stopped changing task files. Processes outside the group, including services from
145-
completed tool calls, remain for the sandbox owner to stop: Resources after verification
146-
or episode failure, or the agent when closing its own sandbox. Ownership
147-
does not change process supervision. Services must not depend on the agent's temporary
148-
session files. Close confirms the receipt before cancelling provider execution,
141+
One shared Linux supervisor per activation fences delayed launches.
142+
The supervisor stops live harness-group processes and reaps its children. On timeout
143+
or close it preserves children already adopted before the stop, typically services from
144+
completed tools, and drains newly adopted children within the cleanup deadline, including
145+
tools in separate sessions. The sandbox owner stops the preserved services later.
146+
Natural CLI exit cannot distinguish unfinished tools in other sessions from services;
147+
those processes may still change task files. The stop-time cutoff also has adoption/PID
148+
races and cannot classify later service forks. It is not a security boundary.
149+
Services must use task-owned files rather than the temporary agent session directory.
150+
Close confirms the receipt before cancelling provider execution,
149151
then removes adapter-owned files and disconnects. Missing/negative termination evidence
150152
blocks verification. Missing evidence stays retryable; a final negative receipt releases
151153
the borrowed connection without making the failed activation gradable. The borrowing

‎responses_api_agents/codex_agent/tests/test_exec_transport.py‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -96,10 +96,17 @@ def payload(state, code, timeout=0.5):
9696
@pytest.mark.parametrize("ending", ["natural", "crash", "timeout", "cancel", "close"])
9797
async def test_exec_only_supervision_leaves_task_children_for_sandbox_owner(tmp_path, ending):
9898
state, provider, workdir = make_session(tmp_path)
99+
# Complete the tool before interruption so its detached service is already adopted.
100+
tool = (
101+
"import pathlib,subprocess,sys; "
102+
"p=subprocess.Popen([sys.executable,'-c','import time; time.sleep(60)'],"
103+
"start_new_session=True,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL); "
104+
"pathlib.Path('child.pid').write_text(str(p.pid))"
105+
)
99106
code = (
100-
"import subprocess,sys,time,pathlib; "
101-
"p=subprocess.Popen([sys.executable,'-c','import time; time.sleep(60)'],start_new_session=True); "
102-
"pathlib.Path('child.pid').write_text(str(p.pid)); "
107+
"import os,subprocess,sys,time,pathlib; "
108+
f"subprocess.run([sys.executable,'-c',{tool!r}],check=True); "
109+
"pathlib.Path('ready').touch(); "
103110
+ {"natural": "print('{}')", "crash": "raise SystemExit(7)"}.get(ending, "time.sleep(60)")
104111
)
105112
deadline = 0.5 if ending == "timeout" else 60
@@ -108,7 +115,7 @@ async def test_exec_only_supervision_leaves_task_children_for_sandbox_owner(tmp_
108115
)
109116
try:
110117
async with asyncio.timeout(5):
111-
while not (workdir / "child.pid").exists():
118+
while not (workdir / "ready").exists():
112119
await asyncio.sleep(0.01)
113120
if ending == "cancel":
114121
task.cancel()

‎responses_api_agents/codex_agent/tests/test_sandbox_runner.py‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -81,17 +81,22 @@ def test_capture_and_stdin(tmp_path):
8181
@pytest.mark.parametrize("ending", ["natural", "timeout", "cancel"])
8282
def test_task_descendants_survive_until_owner_teardown(tmp_path, ending):
8383
try:
84-
# The grandchild starts a new session, escaping process-group-only cleanup.
84+
# A completed tool leaves its detached service adopted before close.
85+
tool = (
86+
"import pathlib,subprocess,sys; "
87+
"p=subprocess.Popen([sys.executable,'-c','import time; time.sleep(60)'],"
88+
"start_new_session=True,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL); "
89+
"pathlib.Path('child.pid').write_text(str(p.pid))"
90+
)
8591
code = (
8692
"import subprocess,sys,time,pathlib; "
87-
"p=subprocess.Popen([sys.executable,'-c','import time; time.sleep(60)'],start_new_session=True); "
88-
"pathlib.Path('child.pid').write_text(str(p.pid)); "
89-
+ ("time.sleep(60)" if ending != "natural" else "pass")
93+
f"subprocess.run([sys.executable,'-c',{tool!r}],check=True); "
94+
"pathlib.Path('ready').touch(); " + ("time.sleep(60)" if ending != "natural" else "pass")
9095
)
9196
process = launch(tmp_path, code, timeout=1 if ending == "timeout" else 3)
9297
if ending == "cancel":
9398
for _ in range(200):
94-
if (tmp_path / "child.pid").exists():
99+
if (tmp_path / "ready").exists():
95100
break
96101
time.sleep(0.01)
97102
process.send_signal(signal.SIGTERM)

0 commit comments

Comments
 (0)