Repository navigation
Conversation
A failed run never reaches /verify, so the resources server kept its per-rollout state until its own TTL. When the seed reply names its session (resources_session_id), harness_agent now posts /close_session for that id with the seed cookies after it closes the box. Session routing refuses a close whose body and cookie name different sessions, so a server that names no session is left alone. Best effort: a failed close is logged and the run's error propagates. Tested with and without a named session. Signed-off-by: Christian Munley <cmunley@nvidia.com>
|
/ok to test 93b5928 |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD — one BLOCKER: best-effort session cleanup can indefinitely delay propagation of a failed run's error.
Reviewed both implementation and tests, including affected sandbox lifecycle, session routing, request contracts, and HTTP retries. The test/import topic was inspected directly after its delegated report was unavailable. No other qualifying findings.
Static review only; tests were not executed. The description's referenced rna_design server is absent from this snapshot, so its worker teardown could not be verified directly.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
The server client retries without a deadline, so a resources server that never answers /close_session would keep a failed run from re-raising its error and hold a semaphore slot. Give the close a 30 s overall deadline; a timeout is logged like any other close failure. Signed-off-by: Christian Munley <cmunley@nvidia.com>
|
/ok to test 318924f |
Resolve harness-agent imports and retain both bounded failed-run session cleanup and MCP forwarding. Signed-off-by: Christian Munley <cmunley@nvidia.com>
|
/ok to test c8638dc |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP — the prior finding is resolved; no qualifying new findings.
The cleanup deadline now bounds both the HTTP request, including internal retries, and status/error-body handling. Timeout reaches the warning handler so the original run error can propagate. Sandbox-first ordering, named-session filtering, and merged cookies are retained. The new regression test checks that a never-answering close yields the original RuntimeError.
This follow-up compared subsequent changes against the previously reviewed head, including merged MCP/model-routing paths and relevant lifecycle tests. No unresolved prior findings or introduced regressions above the historical moderate-impact ceiling were identified.
Static review only; tests and real rollouts were not executed. Two delegated reports were unavailable; their cleanup and lifecycle-test topics were inspected directly.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
Main moved the agent's imports to the layered packages and added an except Exception branch ahead of except BaseException, so the failed-run session close now runs from that branch too (also for masked sandbox failures, which never reach /verify). Signed-off-by: Christian Munley <cmunley@nvidia.com>
|
/ok to test f875a72 |
Resolve conflicts with main. Signed-off-by: Christian Munley <cmunley@nvidia.com>
|
/ok to test 0c475a3 |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP — no unresolved prior findings or qualifying new regressions.
The earlier cleanup-deadline fix remains intact. The 30-second deadline still covers the HTTP request, internal retries, and status/error-body handling. Both failure handlers close the sandbox before attempting resources-session cleanup. The regression tests retain the named/unnamed-session, cookie, and original-error assertions.
I compared subsequent changes against the previously reviewed head across lifecycle handling, layered imports and request contracts, successful model/MCP routing, and regression tests. All three delegated inspections completed. No introduced regression above the historical moderate-impact ceiling was identified.
Static review only; tests and real rollouts weren't executed.
Review generated by codex. This was a findings-only review. Request a full review with /review reset=true.
yaoyu-33
left a comment
There was a problem hiding this comment.
Approving. Thanks for this, @cmunley1: a failed run now gives its rna_design worker back right away instead of after session_ttl_s (4200 s).
What I checked at 0c475a3:
responses_api_agents/harness_agent/tests: 63 passed. Withapp.pyreverted to the merge base, both new tests fail. With only the 30 s deadline removed,test_a_close_that_never_answers_does_not_hide_the_run_errorfails. So they test the fix, not a pass-through.- I added scratch probes (not pushed) and all pass. A masked sandbox-infra failure still returns
mask_sample=True/evaluation_completed=False, and it posts one close. A cancelled run posts one close and still ends inCancelledError, including when the close never answers. - Fences and retries: harness_agent seeds with a cookie seed. Rollout collection and legacy_agent pass on no resources-server cookie, so a retried
/rungets a fresh session ID and never meets the fence this close leaves. For a cookie session the baseclose_resources_sessiondoes not compareepisode_id, so the"harness-run"fallback is accepted. - The PR merges cleanly with current main. ruff check and format are clean.
The two inline comments are not blocking.
| assert closes == [] | ||
|
|
||
|
|
||
| async def test_a_close_that_never_answers_does_not_hide_the_run_error(): |
There was a problem hiding this comment.
- Cancellation: delete the close in the cancellation handler (app.py:400) and all 63 tests stay green. Could you add a case that cancels the run during
responses()and asserts one/close_sessionand thatCancelledErrorstill propagates, with a close that answers and with one that never answers? I ran exactly that against this head and it passes. - This test: it never asserts the close was attempted. With both call sites removed (and the constant kept), it passes. Recording the
/close_sessioncall in thepoststub and asserting it would make it guard the call too.
A masked sandbox-infra case (one close, mask_sample=True) would round it out.
| finally: | ||
| _RUN_CONTEXT.reset(token) | ||
|
|
||
| async def _close_resources_session(self, body: HarnessAgentRunRequest, cookies: Any, session_id: Any) -> None: |
There was a problem hiding this comment.
Gym/responses_api_agents/harness_agent/app.py
Line 405 in 0c475a3
rna_design_claude_code_agent, for example with the policy model unreachable? It would show /close_session answering 200 and rna_design's worker gone right away, not at session_ttl_s. Not blocking: the unit tests here and rna_design's test_a_harness_that_gives_up_releases_the_rollout_by_the_seeded_id cover each side of the call.
|
/ok to test 0c475a3 |
What
A failed harness run never reaches
/verify, so the resources server keeps that rollout's state until its own session TTL. When the seed reply names its session (resources_session_id),harness_agentnow posts/close_sessionfor that id with the seed cookies, after it closes the box.Why
Servers with heavy per-rollout workers (for example
rna_design's tool workers, #4931) otherwise keep a dead rollout's worker alive for the full TTL. Today onlyrna_designnames its session in the seed reply, so this takes effect with #4931 and other servers are unaffected.Tests
responses_api_agents/harness_agent/tests: 40 passed, includingtest_failed_run_closes_the_session_the_seed_nameswith and without a named session andtest_a_close_that_never_answers_does_not_hide_the_run_error. ruff check and format clean.