Repository navigation
feat(sandbox): add apptainer create.private_tmp option - #4005
linj-glitch wants to merge 5 commits into
Conversation
adc9b0d to
5c2bba2
Compare
Apptainer binds the host /tmp into every instance by default, so all sandboxes on a node share one /tmp with each other and with host-side processes (e.g. Ray's session directory under /tmp/ray). Agents commonly create and delete scratch dirs under /tmp, and graders write fixed-name files there, so concurrent sandboxes can clobber each other's files. Add an opt-in `create.private_tmp` flag. When set, `instance start` gets `--no-mount tmp` plus a bind of a per-instance `tmp` directory (mode 1777, inside the instance's staging dir) over /tmp. The directory is removed together with the staging dir. Default off; no behavior change for existing configs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
5c2bba2 to
b6956ed
Compare
|
/ok to test b6956ed |
alexandery-nvidia
left a comment
There was a problem hiding this comment.
Approving from the consumer side; this is one of the four Lin flagged for Ananth's focus.
The per-instance /tmp with the 1777 staging directory and the default-off knob look right, and the two tests pin both settings. Ran tests/unit_tests/test_apptainer_provider.py at the PR head locally: 70 passed. One gap worth closing or documenting, inline.
…eMo#4031) ## What does this PR do? By default `apptainer instance start` bind-mounts the invoking user's home into the instance. When the Gym server runs as root, as it usually does inside a training container, that home is `/root`, so the server's `/root` replaces the task image's `/root` in every `anyswe_agent` sandbox on the node. Task images keep toolchains under `/root`. Example: in the R2E-Gym image `namanjain12/pillow_final:e40033dfc46f3bb3908079357a5eefc7c9db1725`, ``` /testbed/.venv/bin/python -> /root/.local/share/uv/python/cpython-3.9.21-linux-x86_64-gnu/bin/python3.9 ``` With the server's home mounted over `/root` that link dangles. The agent cannot run the repository's Python or tests, and the grader's `run_tests.sh` (`.venv/bin/python -m pytest -rA r2e_tests`) fails with "No such file or directory", so the rollout is graded unresolved. Anything an agent or grader writes under `/root` also lands in the server's own home and is visible to every other sandbox on the node. `anyswe_agent` never needs the host home: the agent already runs with `HOME=/sandbox/home`. This PR resolves the sandbox provider through a new `_resolve_sandbox_provider()`, which for the Apptainer provider appends `--no-home` to `create.extra_start_args`. It adds the flag once, does not mutate the agent config, accepts `create` as a mapping or an `ApptainerCreateConfig`, and returns other providers unchanged. The README documents it. Relation to other open PRs: NVIDIA-NeMo#4024 moves the grader's own files (eval script, patch) to the per-instance `/sandbox` directory, so concurrent graders stop overwriting each other. That PR deliberately does not use `--no-home`. This PR addresses a different symptom: the task image's own `/root` content being hidden. NVIDIA-NeMo#4005 adds an opt-in Apptainer `private_tmp` for `/tmp`; this PR does not depend on it. ## Validation - `python -m pytest -q responses_api_agents/anyswe_agent/tests tests/unit_tests/test_apptainer_provider.py`: 95 passed (run with the worktree's modules pre-imported; `nemo_gym` reorders `sys.path` on import, which can otherwise pick up another editable checkout). - New tests in `responses_api_agents/anyswe_agent/tests/test_apptainer_home.py`: - inline config: `--no-home` is appended after the existing args, other `create`/`exec` keys are kept, and the input config is not mutated; - named config (`sandbox_provider: sandbox` with `default_metadata`), empty `create`, and a dataclass `create`; - the flag is never added twice; - Docker and OpenSandbox configs come back unchanged; - `test_apptainer_instance_start_carries_no_home`: builds a real `ApptainerProvider` from the resolved config (binary and `_run` faked) and checks that the `apptainer instance start` argv contains `--no-home` before the image. - The new tests fail on `main` (`_resolve_sandbox_provider` does not exist there). - I confirmed the symlink above by listing the image's squashfs. I did not run Apptainer itself for this PR: the login host I prepared it on has no Apptainer. - `ruff check`, `ruff check --select I` and `ruff format` (v0.9.9) on the touched files: clean. ## Rollouts Found in internal Claude Code evaluation and training with the Apptainer provider and a Gym server running as root in its container. Our runs have used `--no-home` for all anyswe sandboxes since; before it, R2E-Gym repository interpreters under `/root` were unusable inside the sandboxes. I have no public rollout numbers to share. ## Compatibility and benchmark impact Only the Apptainer provider is affected. Sandboxes no longer see the server user's home. A setup that relied on files from the server's home inside the task sandbox would need to bind them explicitly (`exec.default_binds` or `provider_options.binds`). anyswe already requires a writable container root (it creates `/trajectories_mount`), so writes under `/root` keep working. No config change. ## Checklist - [x] I have read the [contributing guidelines](https://github.com/NVIDIA-NeMo/Gym/blob/main/CONTRIBUTING.md). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Documentation added or updated, or N/A with justification. - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format checks pass). (ruff on the touched files.) - [x] New source files include the required Apache-2.0 SPDX header. - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Lin Jia <linj@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
|
/ok to test 939b7a2 |
Signed-off-by: Lin Jia <linj@nvidia.com>
|
/ok to test ad79e70 |
Apptainer's tmp mount group covers both /tmp and /var/tmp, so --no-mount tmp dropped the host /var/tmp bind as well and only /tmp was re-bound; with a plain SIF image anything writing /var/tmp hit the read-only image. The instance now gets a private 1777 directory at each path (tmp and var_tmp under its staging dir), both removed with the instance. Signed-off-by: Lin Jia <linj@nvidia.com>
|
Closed the |
Signed-off-by: Lin Jia <linj@nvidia.com>
|
/ok to test c39a94d |
Adds opt-in
create.private_tmpso each Apptainer instance gets separate staging-backed/tmpand/var/tmpdirectories with mode1777, both removed during instance cleanup; the default remainsfalse.Apptainer's
--no-mount tmpdisables the host mounts for both paths. The October 25 follow-up (b02091b25d244172f05c0aeea7311dbfe51d884d) now rebinds both paths, so workloads writing/var/tmpno longer fall through to a read-only plain SIF image. The README and regression test cover this behavior, addressing the review feedback and matching Apptainer's mount implementation.Current-head validation — October 25, 2026
Merged main
7b6326761e9dc5af9a45cd1dbe584f3f15a67ff0additively, preserving the author's follow-up commit and including the shared colored R2E summary/CI setup fix (#5026). No conflict resolution was needed; the PR diff remains limited to the Apptainer provider, its README, and provider tests.Validated
c39a94d202b20dd5675daa82cce5159436e17e8don Python 3.13.14 with provider and AnySWE imports confirmed to resolve to this checkout:pre-commit run --all-files: passed on the final head.git diff --check: passed.Test selection:
Earlier validation and rollout scope
The October 24 refresh merged main
f36e126547866601bee9c0671c9b59484ef83723, including Terminus shell recovery (#4966) and MiniSWE test isolation (#4735). Atad79e700fbd36545e8e3373e003af83aabd7e58c, 72 provider tests, full pre-commit, and actual PR diff/description checks passed.The original feature was exercised in internal training runs. The October 24 refresh had no live Apptainer binary available; this October 25 refresh adds local regression validation, with no fresh live Apptainer integration run or model rollout for this feature. Current-head CI and repository-required approval remain merge gates.