Repository navigation
feat(sandbox): add apptainer exec.memory_limit_mib per-process memory cap - #4025
Open
linj-glitch wants to merge 3 commits into
Open
linj-glitch wants to merge 3 commits into
linj-glitch wants to merge 3 commits into
Conversation
linj-glitch
marked this pull request as draft
October 5, 2026 06:11
linj-glitch
force-pushed
the
linj/apptainer-exec-memory-limit
branch
from
October 8, 2026 00:15
88e781e to
044bc37
Compare
… cap Rootless Apptainer usually cannot enforce cgroup memory limits (the --memory flag needs cgroups v2 delegation and is skipped under --fakeroot), so one agent-written command that allocates without bound can exhaust host memory and take down every sandbox and server on the node. In SWE-agent RL training on nodes with ~900 GB of RAM, two ad-hoc image-processing scripts written by the agent inside sandboxes reached 341 GB and 280 GB RSS and the kernel OOM-killed the training job. exec.memory_limit_mib (default None, unchanged behavior) prefixes every exec'd command with `ulimit -d <KiB> 2>/dev/null;` inside the shell that runs it (the inner shell under `su`), so the command and its children inherit RLIMIT_DATA and a runaway allocation fails inside the sandbox (Python raises MemoryError) instead of on the host. The cap is per process, not per sandbox; the README says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
linj-glitch
force-pushed
the
linj/apptainer-exec-memory-limit
branch
from
October 8, 2026 00:20
044bc37 to
0fc7bae
Compare
linj-glitch
marked this pull request as ready for review
October 13, 2026 23:08
This was referenced Oct 22, 2026
Signed-off-by: Lin Jia <linj@nvidia.com>
Collaborator
Author
|
/ok to test a012a75 |
Signed-off-by: Lin Jia <linj@nvidia.com>
Collaborator
Author
|
/ok to test 74394f6 |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds an opt-in per-process memory cap to the Apptainer sandbox provider:
exec.memory_limit_mib.Why. Rootless Apptainer usually cannot enforce cgroup memory limits. The
--memoryflag thatSandboxResources.memory_mibmaps to needs cgroups v2 delegation, and the provider skips it whencreate.extra_start_argscontains--fakeroot. With no cap, one command the agent writes can allocate without bound and exhaust host memory, which takes down every sandbox and server on that node.Example. In SWE-agent RL training on nodes with ~900 GB of RAM, two ad-hoc image-processing scripts that the agent wrote and ran inside Apptainer sandboxes grew to 341 GB and 280 GB RSS. The kernel OOM-killed the training job.
What changes. With
exec.memory_limit_mib: Nset, every exec'd command runs asinside the shell that runs it. For
user=<other>the prefix goes into the innersu -s /bin/sh -ccommand, so the limit also applies after the user switch. The command and its child processes inheritRLIMIT_DATA, so an allocation past the cap fails inside the sandbox (for example, Python raisesMemoryError) instead of on the host. If the shell cannot set the limit, the command still runs. The default isNone, which leaves commands unchanged.RLIMIT_DATAcaps each process separately, not the sandbox as a whole: a command that forks N workers can still use up to N times the cap. The README says so under Limitations, and also notes that runtimes which reserve large writable address ranges at startup may need a generous value. A sandbox-wide limit would need host-side process accounting, which is out of scope here.Validation
python -m pytest -q tests/unit_tests/test_apptainer_provider.py: 73 passed.memory_limit_mibconfig validation,test_exec_memory_limit_is_off_by_default,test_exec_memory_limit_prefixes_the_command(default user,root, andsuuser), andtest_exec_memory_limit_command_fails_a_runaway_allocation_in_a_real_shell. With this PR'sprovider.pychange reverted, 5 of them fail.apptainer exec ... sh -cunder the hostsh, with a 256 MiB cap. A 32 MiB allocation in a child Python process succeeds, a 1 GiB allocation raisesMemoryError, and the command exits non-zero. I also checked by hand withdashthat theulimit -dprefix bounds a child process.ruff check/ruff check --select I/ruff format(v0.9.9, as pinned in.pre-commit-config.yaml) on the touched files: clean.Rollouts
N/A: this is a provider-level opt-in knob, and it is off by default, so existing rollouts are unchanged. Internally it has been used as a host-protection cap in Apptainer-based SWE-agent RL training. I have no public rollout numbers to share.
Compatibility and benchmark impact
None by default (
memory_limit_mib: None). When it is enabled, a command that needs more memory per process than the cap fails inside the sandbox. Size the cap well above what legitimate test suites need.Checklist
exectable, Limitations.)pre-commit run --all-files); see the final-head validation below.git commit -s) (so the DCO check passes).🤖 Generated with Claude Code
Readiness validation — October 13, 2026
Reviewed head
0fc7bae0d1a56f7c716bacfd905529319652cdb2against current main and Apptainer provider history. Main still lacks this optional per-process cap; its default-off behavior and documented per-process limitation are preserved. This PR is the focused proposal for the reproducible rootless-provider memory-exhaustion problem; there is no separate issue.Fresh validation on this head (Python 3.13.14, CPU, source-bound isolated checkout):
python -m pytest -o addopts= -q tests/unit_tests/test_apptainer_provider.py: 73 passed, including the real Linux shell allocation probe.python -m pre_commit run --all-files: passed, without modifying tracked files.a032f4b8din a merge-tree check.No real Apptainer instance or fresh model rollout was run locally. This optional provider setting is disabled by default; existing author-reported training use remains author-provided. It is a per-process cap, not a sandbox-wide cap. Full mirrored Gym CI and GitHub's required reviews remain merge gates; an Apptainer integration check would strengthen the available evidence.
Current-main integration validation — October 25, 2026
Merged main
7b6326761e9dc5af9a45cd1dbe584f3f15a67ff0additively at74394f6f79fafa2a55991031914efb84fe2f45aa. This incorporates #5026's fix for ANSI-colored R2E test summaries, which caused the previous required CI failure. The Apptainer implementation, documentation, and provider tests are unchanged from the previously validated PR head.Python 3.13.14, isolated source-bound checkout:
python -m pytest -o addopts= --import-mode=importlib -q tests/unit_tests/test_apptainer_provider.py responses_api_agents/anyswe_agent/tests/test_r2e_test_setup.py responses_api_agents/anyswe_agent/tests/test_app.py: 118 passed, including 75 provider tests and 43 AnySWE tests. Both plain and colored nested pytest summaries pass.pre-commit run --all-files: passed on the final merge commit; explicit guardword scans of the complete PR diff and this description also passed.No fresh live Apptainer rollout was run for this baseline integration. Existing author-reported training use and the real-shell allocation probe remain the memory-cap integration evidence. Current-head required CI and approval remain merge gates.