Skip to content

swe_agents: run agent and grading containers through the shared sandbox layer #5015

Open

Description

@sshiddib

Use cases, pain points, and background
swe_agents (OpenHands) builds and runs its own apptainer exec commands instead of using AsyncSandbox. As a result it gets none of what the shared sandbox layer provides: sandbox lifecycle metrics (#3633), CPU and memory limits through SandboxResources, and exec concurrency. Harnesses on the sandbox layer, such as hermes_sandboxed_agent and opencode_sandboxed_agent, get all three.

From a SWE-smith concurrency study on a Slurm cluster with Apptainer and no Docker:

  • The sandbox panels of a shared Gym dashboard (start-up time, exec time, active sandboxes) stayed empty for swe_agents runs.
  • Container memory is not visible: Gym's process-tree memory showed 332 MiB while the container used 948 MB.
  • The extra time per rollout at high node load was all inside the containers (connect ×2.0–2.4, setup ×4.0–5.5, commands ×2.8–4.2, with model and grading time flat), and none of it showed in Gym's telemetry.

#1685 lists migrating swe_agents onto the provider protocol as tech debt. #2763 leaves the direct-Apptainer SWE path out of scope, and SWE support moved to anyswe_agent (#1738), but swe_agents is still used for OpenHands evals and has a training config.

Description:
Run swe_agents' agent and grading containers through AsyncSandbox with the apptainer (or enroot) provider, keeping its current behaviour and configuration.

Design:

  • responses_api_agents/swe_agents/app.py: replace the direct apptainer exec construction and create_subprocess_shell calls with AsyncSandbox create / exec / stop.
  • Map existing settings (apptainer_memory_limit_mb, command_exec_timeout, swebench_tests_timeout) to SandboxSpec / SandboxResources.
  • Keep the OpenHands runtime setup and the separate grading container as they are.

Out of scope:

  • Changes to the OpenHands agent itself.
  • Replacing swe_agents with anyswe_agent.
  • Cluster-side cgroup delegation.

Acceptance Criteria:

  • swe_agents rollouts emit gym.sandbox.active, startup_duration_ms and exec_duration_ms.
  • SandboxResources CPU and memory are applied where the provider supports them.
  • Runs under Apptainer on Slurm without Docker.
  • Score parity with the current path on a fixed task set, within the confidence interval.

We can provide test runs and a parity check on a Slurm + Apptainer cluster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Agentscommunity-requestIssue reported or requested by someone from the communitywaiting-on-maintainersWaiting on maintainers to respond

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions