Repository navigation
feat(nooa): supervise the borrowed sandbox process - #3851
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
77df237 to
e9ba5c2
Compare
e9ba5c2 to
8ca3561
Compare
8ca3561 to
c72a742
Compare
c72a742 to
8581442
Compare
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com> (cherry picked from commit 0b083f2) Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com> NOOA-only rebuild of PR #3850 from a25cb1c. NOOA-only scope: shared infrastructure remains identical to main. Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
init-nikhil
left a comment
There was a problem hiding this comment.
CODEX SELF REVIEW
Reviewed head 76902292d3dfc51fea78dc7bf6e24dfc475e82c1 against actual PR base e10a4ac593dc74b8f26a6707c6333eabd7654228.
Engine: official OpenAI codex-cli 0.154.0, fresh codex exec --sandbox read-only --ephemeral --json review --base process, inheriting the user's existing configuration. Exit status: 0; the checkout remained clean.
No actionable findings were returned by this Codex review. Original final output:
No actionable defects were identified. All 50 sandbox-runner tests passed with in-memory file-transfer substitutes; real-process checkpoint tests could not be run under the read-only restrictions.
These test results used substituted file-transfer helpers and do not establish real-process checkpoint behavior or complete the broader Gymcheck workflow. A no-finding review does not guarantee correctness.
Run evidence: gymcheck-3851-20261020T021500Z/reviewer; preserved invocation metadata, stdout, stderr, and final output. Final-output SHA-256: fed26ff0dafaa1d0c251bd0e6de76862d0af80e1136c9473b0ff1350e616836c.
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
init-nikhil
left a comment
There was a problem hiding this comment.
CODEX SELF REVIEW
Reviewed commit cdb801ac6678b3cab5bf8c0c06ec9ebbf24d2c49 against actual PR base e10a4ac593dc74b8f26a6707c6333eabd7654228. The review ran in a clean local checkout at this exact commit.
Engine: official OpenAI codex-cli 0.154.0, fresh codex exec --sandbox read-only --ephemeral --json review --base process, inheriting the user's existing configuration. Exit status: 0; the checkout remained clean.
No actionable findings were returned by this Codex review. Original final output:
No actionable defects were found. All 70 sandbox-runner tests passed using an in-memory file-transfer shim; the subprocess recovery tests could not be run in the read-only sandbox.
The 70 tests used an in-memory file-transfer shim. Subprocess recovery tests were not executed by this reviewer. A no-finding review does not guarantee correctness; broader workflow validation is recorded separately.
Run evidence: gymcheck-3851-20261020T021500Z/reviewer-repair-2; preserved invocation metadata, stdout, stderr, and final output. Final-output SHA-256: 658270b8c619096c7dd3c7a1b5b960cef9f4d6bfe30a2171ca0e9417430402c8.
|
Verification checks PASS: 13 fresh cases on head
All thirteen cases ran freshly once. Earlier results and incomplete attempts remain preserved as historical evidence. Python 3.13.15, pinned NOOA Scope: local subprocess/file transport. Agent close wiring, environment scoring and Evidence date: 2026-10-20 UTC. These checks ran on the clean local commit before its publication; the published commit and tree are identical. Focused controller tests: 70 passed; |
yaoyu-33
left a comment
There was a problem hiding this comment.
Thanks for the work here. This adds SandboxNOOARunner, which runs the #3850 worker in a borrowed sandbox through the shared SandboxSession. Ordering cleanup before capture, never letting a recovered checkpoint become gradable, and releasing after capture all look right to me. I ran the nooa suite locally (331 passed, 8 skipped: the Linux-only installer tests; the real-subprocess SIGKILL recovery tests ran).
The main item is the inline comment on the result.json download: a transient transport error currently downgrades a completed run to an infrastructure error. There is also a question about how a worker timeout is classified (it ends up as a retryable HTTP 503 in #3849), plus notes on diagnostics, two ignored launch arguments, and three untested paths.
Earlier thread: the 180s provider-default timeout concern is addressed at this head. The runner now passes resolve_agent_timeout(...) (6h ceiling) to SandboxSession.execute, and test_sandbox_runner.py asserts timeout_s == 21660.
PR hygiene: could you link the relevant issue, or say why none is needed (AGENTS.md "Pull Requests")?
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
Signed-off-by: Nikhil Deshmudre <nikhil.deshmudre@gmail.com>
yaoyu-33
left a comment
There was a problem hiding this comment.
Thanks for the follow-up. Most of the earlier comments are fixed: the worker deadline is now a non-transient error, output.log diagnostics are collected on every failure path and kept as a gap, the dead launch arguments are gone, and the missing tests (prepare() failure, worker timeout with a recovered checkpoint, cancel with no evidence) are in. I ran the nooa suite locally on this head (372 passed, 8 skipped: the Linux-only installer tests).
One earlier comment is only partly resolved, and it is the inline comment below: the transient result.json download retry doesn't trigger on the default Docker provider. I have no other blocking comments.
| observations.gaps.append(ObservationGap(code="sandbox_worker_diagnostics", detail=self.diagnostics)) | ||
| self.observations = observations | ||
|
|
||
| async def _read_artifact(self, name: str) -> str: |
There was a problem hiding this comment.
The retry only triggers for transient exception types, but the Docker provider's download_file raises a plain RuntimeError for any non-zero docker cp, so on the default provider a transient copy failure isn't retried and still turns a completed run into a non-retryable capture error. The same RuntimeError is raised when result.json is genuinely absent, so "missing" and "failed" are indistinguishable there, and the tests only pass because the fake raises FileNotFoundError. Could you retry once on any exception other than FileNotFoundError / ValidationError (an absent file just fails twice and is reported with the diagnostics), or probe with [ -f path ] before calling it missing, and add a test where the fake raises a Docker-style RuntimeError?
yaoyu-33
left a comment
There was a problem hiding this comment.
Approving. The one inline comment (retry on Docker's RuntimeError for the result.json download) is worth fixing before merge but doesn't block.
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/39878587375 |
Tests and validation
Head
307a96916f3823cb95fcbbeb25506cf2b0262bab; base3c332f419f0073dd9fc360e0056a47f254cc3d98.Consumes #3850 and supplies #3849. These existing stack PRs track the work; no separate issue is needed.
98f59e549: 81 controller tests passed with NOOA0514723; all-files pre-commit passed.307a9691: 107 controller tests passed; real Docker startup and timeout cases reproduce before the fix and pass afterward, retaining bounded diagnostics and capture errors. An independent transport control preserves the original retryable exception.Assembled-stack tests, Docker smoke and their exact-revision limits are recorded in #4813; they supplement these layer-specific checks.
Provider execution follows the worker budget plus cleanup time, avoiding Docker's three-minute default. Worker budget exhaustion is non-transient and ungradable; genuine transport/provider errors retain transient classification. Diagnostics are limited to 2,000 characters, stripped of control characters, and stored in observation gaps. Checkpoint recovery cannot establish successful completion. Shared-supervisor descendant cleanup remains unchanged.
CI: on the new head (2026-10-21 UTC), DCO passed; OpenCode, Pi and Codex conformance and their aggregate report are failing. Other checks are still running where shown in the Checks tab. GitHub is not all green.