Skip to content

fix(agent): stop waiting for a stalled harness after a rollout stop - #6452

Open
gwarmstrong wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
gwarmstrong:georgea/stop-deadline-upstream
Open

gwarmstrong wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
gwarmstrong:georgea/stop-deadline-upstream

Conversation

@gwarmstrong

Copy link
Copy Markdown
Contributor

What does this PR do?

This follows up #5145 (rollout stop). A rollout stop only writes the supervisor's expire.request marker and returns. The activation then waits on the supervised command until the provider's execution deadline, which is the agent timeout plus the cleanup reserve. When memory pressure stalls the sandbox, the supervisor never acts on the marker, and Gym keeps polling the command for hours. In a TB-Science run, a Codex rollout was stopped at its 8 h budget and came back 292 minutes later. Its model replicas were held the whole time.

With this change, once the expiry marker is in the sandbox, the activation waits for the command only as long as the supervisor's own cleanup bound. That bound is the supervisor's three cleanup phases plus 30 s: session_close_timeout_seconds plus 30 s, which is 90 s by default and 330 s for TB-Science. It is the same reserve the provider deadline keeps after the harness timeout. An agent can set rollout_stop_grace_seconds to use a different value. Once the bound passes, the session logs one warning that names the rollout. It then reads the supervisor's receipt within the close timeout. It does not send a stop request, because a stop request would turn the expiry into a cancellation if the supervisor resumed.

  • If the receipt confirms termination (the command's status hung after the supervisor finished), the activation captures and is graded as a stop, as before.
  • Otherwise termination is unconfirmed, as after a negative receipt. The session records harness_termination_unconfirmed, stops polling the command and raises the failure, so the rollout fails cleanly instead of hanging. Close works as before: it releases a borrowed sandbox for the Resources Server's teardown, or stops an owned one.

Changes:

  • supervisor_client: cleanup_bound() is split out of exec_timeout(). The new read_cleanup_receipt() reads the receipt without asking anything of the supervisor. stop_and_confirm_cleanup now uses it, and its behavior is unchanged.
  • SandboxSession: expire() takes an optional grace. New _await_exec() waits for the command with that bound once an expiry is delivered, and new _stop_overdue_harness() settles a harness that outlives it.
  • BaseResponsesAPIAgentConfig.rollout_stop_grace_seconds is optional. Unset means the cleanup bound. It is passed to expire() both from the stop route and for a stop that arrives during the seed.
  • The HARNESS_TERMINATION_UNCONFIRMED comment now covers the new cause, and the harness guide (add-or-update-a-harness.mdx) documents the bound.
  • The session-routing tests' fake session accepts the new grace argument.

No issue was filed. The failure is the TB-Science run described above.

Validation

All runs are on upstream main at 155b27d plus this branch:

  • pytest tests/unit_tests/agents tests/unit_tests/server_runtime/test_session_routing.py: 735 passed, 2 skipped.
  • pytest responses_api_agents/codex_agent/tests tests/unit_tests/core: 1257 passed, 6 skipped and 1 failed. The failure is test_opensandbox_sdk_is_available_in_agent_environment, which checks the Codex agent's own venv for the OpenSandbox SDK. It fails because I ran from the root venv. It is unrelated to this change.
  • pre-commit run --files <the 9 changed files>: all hooks pass.
  • Coverage of the lines this branch adds, measured over the agent, routing and Codex session suites:
    • supervisor_client.py: 15/15.
    • sandbox_session.py: 53/55 (97.4% for the whole file). The 2 missed lines are the warning logged when releasing the overdue command itself fails.
  • New tests:
    • test_sandbox_session.py: a stopped harness that outlives its supervisor fails within the grace (borrowed and owned sandboxes, stop before and after launch, receipt missing or hanging). A hung command status is graded from a confirming receipt. A negative receipt found after the grace is the failure. A supervisor that ends the harness promptly is waited for without a warning. The default grace is the cleanup bound.
    • Codex test_sandbox_sessions.py: a stalled Codex rollout fails cleanly after the grace, and one whose command status hangs is still graded.

Rollouts

This was a TB-Science smoke with the Codex agent on OpenSandbox, run locally through the EFB runner against a multiplexer. It used Gym a631623d4a, which is this fix plus unrelated branch work, before the rebase onto main. It ran 2 tasks with session_close_timeout_seconds: 300.

One rollout's sandbox processes were frozen with SIGSTOP (the supervisor, runner, node and codex), and the rollout was then stopped:

  • The agent logged the Codex harness did not end within 330 s of its stop; no longer waiting for its command.
  • It found no receipt and failed the rollout with harness_termination_unconfirmed (it did not end within 330 s of its stop, and no receipt confirms that it ended).
  • Rollout collection recorded it as a failed rollout and finished.
  • The other task's rollout ran normally and was graded (reward 0).

Without the fix, the same stall kept the rollout until the provider deadline, 292 minutes after the stop in the original run.

Compatibility and benchmark impact

  • New optional agent config field rollout_stop_grace_seconds. Leaving it unset gives the cleanup bound. Existing configs need no change.
  • Behavior changes only for a stopped rollout whose harness is still running after the cleanup bound. Such a rollout now fails with harness_termination_unconfirmed and no longer waits until the provider deadline. A supervisor that works as designed ends its harness within the bound, so those results do not change. Runs that do not use the stop route are not affected.
  • SandboxSession.expire() takes a new optional keyword, grace. Test doubles that replace expire must accept it, as the routing tests' fake now does.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Documentation added or updated, or N/A with justification.
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass). Run on the changed files only; see Validation.
  • New source files include the required Apache-2.0 SPDX header. (N/A: no new files.)
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

A rollout stop (NVIDIA-NeMo#5145) only writes the supervisor's expire.request and
returns; the activation then waits on the supervised command until the
provider's execution deadline, the agent timeout plus the cleanup
reserve. When memory pressure stalls the sandbox, the supervisor does
not act on the marker, and Gym keeps polling the command for hours. In
a TB-Science run, a Codex rollout stopped at its 8 h budget came back
292 minutes later, with its model replicas held all along.

Once the expire marker is in the sandbox, the activation now waits for
the command only for the supervisor's own cleanup bound: its three
cleanup phases plus 30 s (cleanup_bound, the same reserve the provider
deadline keeps after the harness timeout; session_close_timeout_seconds
plus 30 s, 90 s by default and 330 s for TB-Science). Agents can set
rollout_stop_grace_seconds instead. Past it, the session logs one
warning naming the rollout and reads the supervisor's receipt within
the close timeout, without a stop request, which would turn the expiry
into a cancellation if the supervisor resumed:

- a receipt that confirms termination (the command's status hung after
  the supervisor finished) lets the activation capture and be graded as
  a stop, as before;
- otherwise termination is unconfirmed, as after a negative receipt:
  the session records harness_termination_unconfirmed, stops polling
  the command, and raises it, so the rollout fails cleanly instead of
  hanging. Close releases a borrowed sandbox for the Resources Server's
  teardown, or stops an owned one, unchanged.

stop_and_confirm_cleanup now reads the receipt through the new
read_cleanup_receipt; its behavior is unchanged.

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
SimpleResponsesAPIAgent.stop_rollout now passes grace to
SandboxSession.expire; the session-routing tests' stand-in session
must accept it.

Signed-off-by: gwarmstrong <gwarmstrong@users.noreply.github.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@gwarmstrong

Copy link
Copy Markdown
Contributor Author

/ok to test fe71308

@nemo-automation-bot
nemo-automation-bot Bot requested review from a team November 19, 2026 06:05
@github-actions

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the risk:high Autobot PR risk: high label Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Autobot PR risk: high

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants