Skip to content

feat(sandbox): add apptainer exec.memory_limit_mib per-process memory cap - #4025

Open
linj-glitch wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
linj-glitch:linj/apptainer-exec-memory-limit
Open

linj-glitch wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
linj-glitch:linj/apptainer-exec-memory-limit

Conversation

@linj-glitch

@linj-glitch linj-glitch commented Oct 4, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?

Adds an opt-in per-process memory cap to the Apptainer sandbox provider: exec.memory_limit_mib.

Why. Rootless Apptainer usually cannot enforce cgroup memory limits. The --memory flag that SandboxResources.memory_mib maps to needs cgroups v2 delegation, and the provider skips it when create.extra_start_args contains --fakeroot. With no cap, one command the agent writes can allocate without bound and exhaust host memory, which takes down every sandbox and server on that node.

Example. In SWE-agent RL training on nodes with ~900 GB of RAM, two ad-hoc image-processing scripts that the agent wrote and ran inside Apptainer sandboxes grew to 341 GB and 280 GB RSS. The kernel OOM-killed the training job.

What changes. With exec.memory_limit_mib: N set, every exec'd command runs as

ulimit -d <N*1024> 2>/dev/null; <command>

inside the shell that runs it. For user=<other> the prefix goes into the inner su -s /bin/sh -c command, so the limit also applies after the user switch. The command and its child processes inherit RLIMIT_DATA, so an allocation past the cap fails inside the sandbox (for example, Python raises MemoryError) instead of on the host. If the shell cannot set the limit, the command still runs. The default is None, which leaves commands unchanged.

RLIMIT_DATA caps each process separately, not the sandbox as a whole: a command that forks N workers can still use up to N times the cap. The README says so under Limitations, and also notes that runtimes which reserve large writable address ranges at startup may need a generous value. A sandbox-wide limit would need host-side process accounting, which is out of scope here.

Validation

  • python -m pytest -q tests/unit_tests/test_apptainer_provider.py: 73 passed.
  • New tests: the memory_limit_mib config validation, test_exec_memory_limit_is_off_by_default, test_exec_memory_limit_prefixes_the_command (default user, root, and su user), and test_exec_memory_limit_command_fails_a_runaway_allocation_in_a_real_shell. With this PR's provider.py change reverted, 5 of them fail.
  • The real-shell test runs the exact command string that the provider passes to apptainer exec ... sh -c under the host sh, with a 256 MiB cap. A 32 MiB allocation in a child Python process succeeds, a 1 GiB allocation raises MemoryError, and the command exits non-zero. I also checked by hand with dash that the ulimit -d prefix bounds a child process.
  • ruff check / ruff check --select I / ruff format (v0.9.9, as pinned in .pre-commit-config.yaml) on the touched files: clean.
  • No Apptainer binary was available on the machine where these tests ran, so the limit was not exercised inside a real instance. Because it is set by the shell inside the container, the mechanism is the one the real-shell test exercises.

Rollouts

N/A: this is a provider-level opt-in knob, and it is off by default, so existing rollouts are unchanged. Internally it has been used as a host-protection cap in Apptainer-based SWE-agent RL training. I have no public rollout numbers to share.

Compatibility and benchmark impact

None by default (memory_limit_mib: None). When it is enabled, a command that needs more memory per process than the cap fails inside the sandbox. Size the cap well above what legitimate test suites need.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Documentation added or updated, or N/A with justification. (Apptainer provider README: config example, exec table, Limitations.)
  • Pre-commit checks pass locally (pre-commit run --all-files); see the final-head validation below.
  • New source files include the required Apache-2.0 SPDX header. (N/A: no new files.)
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

🤖 Generated with Claude Code

Readiness validation — October 13, 2026

Reviewed head 0fc7bae0d1a56f7c716bacfd905529319652cdb2 against current main and Apptainer provider history. Main still lacks this optional per-process cap; its default-off behavior and documented per-process limitation are preserved. This PR is the focused proposal for the reproducible rootless-provider memory-exhaustion problem; there is no separate issue.

Fresh validation on this head (Python 3.13.14, CPU, source-bound isolated checkout):

  • python -m pytest -o addopts= -q tests/unit_tests/test_apptainer_provider.py: 73 passed, including the real Linux shell allocation probe.
  • python -m pre_commit run --all-files: passed, without modifying tracked files.
  • The branch merges cleanly with main a032f4b8d in a merge-tree check.

No real Apptainer instance or fresh model rollout was run locally. This optional provider setting is disabled by default; existing author-reported training use remains author-provided. It is a per-process cap, not a sandbox-wide cap. Full mirrored Gym CI and GitHub's required reviews remain merge gates; an Apptainer integration check would strengthen the available evidence.

Current-main integration validation — October 25, 2026

Merged main 7b6326761e9dc5af9a45cd1dbe584f3f15a67ff0 additively at 74394f6f79fafa2a55991031914efb84fe2f45aa. This incorporates #5026's fix for ANSI-colored R2E test summaries, which caused the previous required CI failure. The Apptainer implementation, documentation, and provider tests are unchanged from the previously validated PR head.

Python 3.13.14, isolated source-bound checkout:

  • python -m pytest -o addopts= --import-mode=importlib -q tests/unit_tests/test_apptainer_provider.py responses_api_agents/anyswe_agent/tests/test_r2e_test_setup.py responses_api_agents/anyswe_agent/tests/test_app.py: 118 passed, including 75 provider tests and 43 AnySWE tests. Both plain and colored nested pytest summaries pass.
  • pre-commit run --all-files: passed on the final merge commit; explicit guardword scans of the complete PR diff and this description also passed.
  • Final PR diff and whitespace checked; merge commit has DCO sign-off.

No fresh live Apptainer rollout was run for this baseline integration. Existing author-reported training use and the real-shell allocation probe remain the memory-cap integration evidence. Current-head required CI and approval remain merge gates.

@copy-pr-bot

copy-pr-bot Bot commented Oct 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33 yaoyu-33 added area:sandbox Sandbox providers, isolation, and sandbox runtime contracts complexity:low Localized change in one scope with a small, straightforward review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Oct 5, 2026
@linj-glitch
linj-glitch marked this pull request as draft October 5, 2026 06:11
@linj-glitch
linj-glitch force-pushed the linj/apptainer-exec-memory-limit branch from 88e781e to 044bc37 Compare October 8, 2026 00:15
… cap

Rootless Apptainer usually cannot enforce cgroup memory limits (the
--memory flag needs cgroups v2 delegation and is skipped under
--fakeroot), so one agent-written command that allocates without bound
can exhaust host memory and take down every sandbox and server on the
node. In SWE-agent RL training on nodes with ~900 GB of RAM, two ad-hoc
image-processing scripts written by the agent inside sandboxes reached
341 GB and 280 GB RSS and the kernel OOM-killed the training job.

exec.memory_limit_mib (default None, unchanged behavior) prefixes every
exec'd command with `ulimit -d <KiB> 2>/dev/null;` inside the shell that
runs it (the inner shell under `su`), so the command and its children
inherit RLIMIT_DATA and a runaway allocation fails inside the sandbox
(Python raises MemoryError) instead of on the host. The cap is per
process, not per sandbox; the README says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch
linj-glitch force-pushed the linj/apptainer-exec-memory-limit branch from 044bc37 to 0fc7bae Compare October 8, 2026 00:20
@linj-glitch
linj-glitch marked this pull request as ready for review October 13, 2026 23:08
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 14, 2026
@linj-glitch

Copy link
Copy Markdown
Collaborator Author

/ok to test a012a75

Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch

Copy link
Copy Markdown
Collaborator Author

/ok to test 74394f6

This branch was previously deployed

1 inactive deployment
public — 74394f6f Deployed Oct 25, 2026 by copy-pr-bot[bot] via release / finalize / notify #7194
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:sandbox Sandbox providers, isolation, and sandbox runtime contracts complexity:low Localized change in one scope with a small, straightforward review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:triage-overdue Review assignment is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants