Skip to content

fix(harness_agent): release the resources session of a failed run - #5085

Open
cmunley1 wants to merge 5 commits into
mainfrom
fix/harness-close-session
Open

cmunley1 wants to merge 5 commits into
mainfrom
fix/harness-close-session

Conversation

@cmunley1

@cmunley1 cmunley1 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

What

A failed harness run never reaches /verify, so the resources server keeps that rollout's state until its own session TTL. When the seed reply names its session (resources_session_id), harness_agent now posts /close_session for that id with the seed cookies, after it closes the box.

  • Session routing refuses a close whose body and cookie name different sessions, so a server that names no session is left alone.
  • Best effort: a failed close is logged and the run's original error still propagates. The close has a 30 s deadline, since the server client retries without one.

Why

Servers with heavy per-rollout workers (for example rna_design's tool workers, #4931) otherwise keep a dead rollout's worker alive for the full TTL. Today only rna_design names its session in the seed reply, so this takes effect with #4931 and other servers are unaffected.

Tests

responses_api_agents/harness_agent/tests: 40 passed, including test_failed_run_closes_the_session_the_seed_names with and without a named session and test_a_close_that_never_answers_does_not_hide_the_run_error. ruff check and format clean.

A failed run never reaches /verify, so the resources server kept its per-rollout state until its own TTL. When the
seed reply names its session (resources_session_id), harness_agent now posts /close_session for that id with the seed
cookies after it closes the box. Session routing refuses a close whose body and cookie name different sessions, so a
server that names no session is left alone. Best effort: a failed close is logged and the run's error propagates.
Tested with and without a named session.

Signed-off-by: Christian Munley <cmunley@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 26, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@cmunley1

Copy link
Copy Markdown
Contributor Author

/ok to test 93b5928

@cmunley1

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:agent Agent harnesses and Responses API agent behavior bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer labels Oct 26, 2026
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 26, 2026 16:16

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD — one BLOCKER: best-effort session cleanup can indefinitely delay propagation of a failed run's error.

Reviewed both implementation and tests, including affected sandbox lifecycle, session routing, request contracts, and HTTP retries. The test/import topic was inspected directly after its delegated report was unavailable. No other qualifying findings.

Static review only; tests were not executed. The description's referenced rna_design server is absent from this snapshot, so its worker teardown could not be verified directly.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

The server client retries without a deadline, so a resources server that never answers /close_session would keep a failed run from re-raising its error and hold a semaphore slot. Give the close a 30 s overall deadline; a timeout is logged like any other close failure.

Signed-off-by: Christian Munley <cmunley@nvidia.com>
@cmunley1

Copy link
Copy Markdown
Contributor Author

/ok to test 318924f

Resolve harness-agent imports and retain both bounded failed-run session cleanup and MCP forwarding.

Signed-off-by: Christian Munley <cmunley@nvidia.com>
@cmunley1

Copy link
Copy Markdown
Contributor Author

/ok to test c8638dc

@cmunley1

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP — the prior finding is resolved; no qualifying new findings.

The cleanup deadline now bounds both the HTTP request, including internal retries, and status/error-body handling. Timeout reaches the warning handler so the original run error can propagate. Sandbox-first ordering, named-session filtering, and merged cookies are retained. The new regression test checks that a never-answering close yields the original RuntimeError.

This follow-up compared subsequent changes against the previously reviewed head, including merged MCP/model-routing paths and relevant lifecycle tests. No unresolved prior findings or introduced regressions above the historical moderate-impact ceiling were identified.

Static review only; tests and real rollouts were not executed. Two delegated reports were unavailable; their cleanup and lifecycle-test topics were inspected directly.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

Main moved the agent's imports to the layered packages and added an
except Exception branch ahead of except BaseException, so the failed-run
session close now runs from that branch too (also for masked sandbox
failures, which never reach /verify).

Signed-off-by: Christian Munley <cmunley@nvidia.com>
@cmunley1

cmunley1 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test f875a72

Resolve conflicts with main.

Signed-off-by: Christian Munley <cmunley@nvidia.com>
@cmunley1

Copy link
Copy Markdown
Contributor Author

/ok to test 0c475a3

@cmunley1

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP — no unresolved prior findings or qualifying new regressions.

The earlier cleanup-deadline fix remains intact. The 30-second deadline still covers the HTTP request, internal retries, and status/error-body handling. Both failure handlers close the sandbox before attempting resources-session cleanup. The regression tests retain the named/unnamed-session, cookie, and original-error assertions.

I compared subsequent changes against the previously reviewed head across lifecycle handling, layered imports and request contracts, successful model/MCP routing, and regression tests. All three delegated inspections completed. No introduced regression above the historical moderate-impact ceiling was identified.

Static review only; tests and real rollouts weren't executed.

Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.

@cmunley1
cmunley1 requested a review from yaoyu-33 November 11, 2026 21:18
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Nov 12, 2026

@yaoyu-33 yaoyu-33 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Thanks for this, @cmunley1: a failed run now gives its rna_design worker back right away instead of after session_ttl_s (4200 s).

What I checked at 0c475a3:

  • responses_api_agents/harness_agent/tests: 63 passed. With app.py reverted to the merge base, both new tests fail. With only the 30 s deadline removed, test_a_close_that_never_answers_does_not_hide_the_run_error fails. So they test the fix, not a pass-through.
  • I added scratch probes (not pushed) and all pass. A masked sandbox-infra failure still returns mask_sample=True / evaluation_completed=False, and it posts one close. A cancelled run posts one close and still ends in CancelledError, including when the close never answers.
  • Fences and retries: harness_agent seeds with a cookie seed. Rollout collection and legacy_agent pass on no resources-server cookie, so a retried /run gets a fresh session ID and never meets the fence this close leaves. For a cookie session the base close_resources_session does not compare episode_id, so the "harness-run" fallback is accepted.
  • The PR merges cleanly with current main. ruff check and format are clean.

The two inline comments are not blocking.

assert closes == []


async def test_a_close_that_never_answers_does_not_hide_the_run_error():

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

async def test_a_close_that_never_answers_does_not_hide_the_run_error():
Two paths could get pinned so a later refactor can't silently drop them (non-blocking):

  • Cancellation: delete the close in the cancellation handler (app.py:400) and all 63 tests stay green. Could you add a case that cancels the run during responses() and asserts one /close_session and that CancelledError still propagates, with a close that answers and with one that never answers? I ran exactly that against this head and it passes.
  • This test: it never asserts the close was attempted. With both call sites removed (and the constant kept), it passes. Recording the /close_session call in the post stub and asserting it would make it guard the call too.

A masked sandbox-infra case (one close, mask_sample=True) would round it out.

finally:
_RUN_CONTEXT.reset(token)

async def _close_resources_session(self, body: HarnessAgentRunRequest, cookies: Any, session_id: Any) -> None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

async def _close_resources_session(self, body: HarnessAgentRunRequest, cookies: Any, session_id: Any) -> None:
AGENTS.md asks for a real run on behavior-changing agent work. Could you add to the PR body one forced-failure run of rna_design_claude_code_agent, for example with the policy model unreachable? It would show /close_session answering 200 and rna_design's worker gone right away, not at session_ttl_s. Not blocking: the unit tests here and rna_design's test_a_harness_that_gives_up_releases_the_rollout_by_the_seeded_id cover each side of the call.

@yaoyu-33

Copy link
Copy Markdown
Contributor

/ok to test 0c475a3

@github-actions github-actions Bot removed the sla:review-overdue Review response is over the one-business-day SLA label Nov 17, 2026

This branch had an error being deployed

1 failed deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants