Skip to content

fix(agents): refuse unsupported sandbox platforms at seed - #5949

Open
yaoyu-33 wants to merge 3 commits into
mainfrom
yuya/refuse-unsupported-sandbox-platform
Open

yaoyu-33 wants to merge 3 commits into
mainfrom
yuya/refuse-unsupported-sandbox-platform

Conversation

@yaoyu-33

@yaoyu-33 yaoyu-33 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

What changed and why

SandboxPythonRuntime.install() raised a plain RuntimeError when the task sandbox runs on a platform no runtime is built for: not x86_64/aarch64 Linux, or glibc older than 2.17. Hermes, Stirrup and BrowseComp call install() in their seed hook (_seed_agent_session_state), so the agent route turned that into HTTP 500 (agent_route_failed). The Environment Server treats a 5xx seed as retryable and reruns the whole episode on a new sandbox, but the new sandbox comes from the same image and fails the same way on every attempt.

The agent review checklist (item 2) says a seed the agent cannot serve must raise AgentSeedRejectedError (final 422). This PR does that:

  • nemo_gym/agents/utils/sandbox_python.py: new UnsupportedSandboxPlatformError(RuntimeError), raised by the platform probe (OS or machine), the glibc floor in SandboxPlatform.wheel_platform, and the "no pinned Python build" check. It subclasses RuntimeError, so existing callers that catch RuntimeError behave as before. It is exported from nemo_gym.agents.
  • Hermes, Stirrup and BrowseComp seed hooks: after the failed setup is cleaned up, they turn only that error into AgentSeedRejectedError. BrowseComp already did this for SandboxIsolationError; the other two now use the same pattern. If cleanup fails, the seed keeps today's behavior: the session is retained for close and the original error is raised.
  • Everything else stays retryable: a platform probe that gives no answer (non-zero exit, timeout, short output), a failed host build or download, a failed import check, and timeouts.
  • Harness guide (add-or-update-a-harness.mdx): one sentence on the refusal, next to the existing list of supported sandbox platforms.

No issue: this came up while reviewing a harness PR against the agent review checklist. The fix is in shared code, so it gets its own PR. The DeepAgents sessions PR (#5003) also calls SandboxPythonRuntime.install() in its seed hook. It needs the same mapping once it rebases onto this.

Validation

Current head 5abdfbf22c7b82bd4fd898498377978412db0ad9 merges main 64348970d991bb36ef28ec465ee87cafbe28d760. Both import conflicts were resolved by preserving the new platform error alongside main's platform probe and timeout helpers.

Validated on a Linux workstation with Python 3.13.14:

  • python -m pytest tests/unit_tests/agents/test_sandbox_python.py responses_api_agents/hermes_agent/tests/ responses_api_agents/browsecomp_agent/tests/ tests/unit_tests/test_public_api_facades.py tests/unit_tests/test_public_api.py -q -o addopts= -o log_cli=false: 1,559 passed, with five configuration checks initially failing because reused dependencies exposed old installed Gym metadata (openai==2.44.0) ahead of the merged source's openai==2.45.0. After installing the current metadata and isolating Hermes's dependency paths, python -m pytest responses_api_agents/hermes_agent/tests/test_benchmark_profile.py responses_api_agents/hermes_agent/tests/test_recipe.py --lf -q -o addopts= -o log_cli=false passed all 5 failed checks. No source change was needed.
  • python -m pytest responses_api_agents/stirrup_agent/tests/ -q -o addopts= -o log_cli=false: 186 passed.
  • 1,750 selected tests passed across the initial runs and the corrected-environment retry. Dependencies were reused in task-specific environments; the merged checkout was first in PYTHONPATH and NEMO_GYM_EXTRA_ROOTS, with source imports verified.
  • python -m pre_commit run --all-files: all applicable hooks passed, with no source changes.
  • The workstation's source tree and pushed commit both have Git tree ac12343863e5175baf1ba1b68dc95d407c280183; git diff --check passed and all three PR commits carry DCO sign-offs.

The review found that glibc rejection originally happened after downloading Python. A download failure could therefore hide the permanent incompatibility and still return retryable HTTP 500. The follow-up checks the glibc floor immediately after probing, before any download.

Both strengthened regressions were run against the original installer and failed for glibc 2.12 (with the Darwin control passing), then passed with the fix:

  • Helper: an unavailable Python download must never be called for an unsupported platform.
  • Stirrup: the real installer and local sandbox provider must return HTTP 422 and release session state, even when downloading Python would fail.

The existing tests also cover unsupported OS/architecture, no pinned build, transient probe failures remaining retryable, Hermes cleanup before refusal, and BrowseComp isolation/platform refusal.

Full repository CI is required because this PR changes shared core code. CI authorization has been requested for the current full head SHA; earlier CI results do not validate this merged head.

Rollout evidence

N/A. The change affects only seeds on sandbox images the runtime cannot serve. Such episodes already failed; they now fail once with a 422 instead of being retried. There is no score impact beyond fewer reruns. Supported platforms take the same code path as before, so no model rollout is needed.

Compatibility

UnsupportedSandboxPlatformError is a new public name in nemo_gym.agents. It subclasses RuntimeError, so no caller has to change.

🤖 Generated with Claude Code

SandboxPythonRuntime raised a plain RuntimeError when the task sandbox's
platform has no runtime (not x86_64/aarch64 Linux, or glibc older than
2.17). Hermes, Stirrup and BrowseComp install the runtime in their seed
hook, so the agent route answered 500 and the Environment Server reran
the whole episode on a new sandbox from the same image, which fails the
same way every time.

The helper now raises UnsupportedSandboxPlatformError (a RuntimeError
subclass, exported from nemo_gym.agents) for those cases. Each seed hook
turns only that error into AgentSeedRejectedError (final 422) after its
failed setup is cleaned up. A failed probe, host build, download or
timeout stays retryable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test c8151f1

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

@yaoyu-33
yaoyu-33 marked this pull request as ready for review November 10, 2026 17:44
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test 5abdfbf

@nemo-automation-bot
nemo-automation-bot Bot requested review from a team November 11, 2026 01:46
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Nov 11, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added bug Something isn't working area:agent Agent harnesses and Responses API agent behavior needs-review PR is ready for code review and waiting on a reviewer labels Nov 16, 2026

This branch had an error being deployed

1 failed deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer sla:triage-overdue Review assignment is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants