Skip to content

feat(sandbox): add apptainer create.private_tmp option - #4005

Open
linj-glitch wants to merge 5 commits into
NVIDIA-NeMo:mainfrom
linj-glitch:linj/apptainer-private-tmp
Open

linj-glitch wants to merge 5 commits into
NVIDIA-NeMo:mainfrom
linj-glitch:linj/apptainer-private-tmp

Conversation

@linj-glitch

@linj-glitch linj-glitch commented Oct 4, 2026 •

Copy link
Copy Markdown
Collaborator

Adds opt-in create.private_tmp so each Apptainer instance gets separate staging-backed /tmp and /var/tmp directories with mode 1777, both removed during instance cleanup; the default remains false.

Apptainer's --no-mount tmp disables the host mounts for both paths. The October 25 follow-up (b02091b25d244172f05c0aeea7311dbfe51d884d) now rebinds both paths, so workloads writing /var/tmp no longer fall through to a read-only plain SIF image. The README and regression test cover this behavior, addressing the review feedback and matching Apptainer's mount implementation.

Current-head validation — October 25, 2026

Merged main 7b6326761e9dc5af9a45cd1dbe584f3f15a67ff0 additively, preserving the author's follow-up commit and including the shared colored R2E summary/CI setup fix (#5026). No conflict resolution was needed; the PR diff remains limited to the Apptainer provider, its README, and provider tests.

Validated c39a94d202b20dd5675daa82cce5159436e17e8d on Python 3.13.14 with provider and AnySWE imports confirmed to resolve to this checkout:

  • Apptainer provider, AnySWE app/R2E setup, and CI environment tests: 196 passed, including 72 provider tests and colored/uncolored R2E cases.
  • pre-commit run --all-files: passed on the final head.
  • Actual PR diff and description guardword checks, plus git diff --check: passed.

Test selection:

python -m pytest -o addopts= --import-mode=importlib -q \
  tests/unit_tests/test_apptainer_provider.py \
  responses_api_agents/anyswe_agent/tests/test_app.py \
  responses_api_agents/anyswe_agent/tests/test_r2e_test_setup.py \
  tests/unit_tests/test_ci_environment.py

Earlier validation and rollout scope

The October 24 refresh merged main f36e126547866601bee9c0671c9b59484ef83723, including Terminus shell recovery (#4966) and MiniSWE test isolation (#4735). At ad79e700fbd36545e8e3373e003af83aabd7e58c, 72 provider tests, full pre-commit, and actual PR diff/description checks passed.

The original feature was exercised in internal training runs. The October 24 refresh had no live Apptainer binary available; this October 25 refresh adds local regression validation, with no fresh live Apptainer integration run or model rollout for this feature. Current-head CI and repository-required approval remain merge gates.

@copy-pr-bot

copy-pr-bot Bot commented Oct 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33 yaoyu-33 added area:sandbox Sandbox providers, isolation, and sandbox runtime contracts complexity:low Localized change in one scope with a small, straightforward review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Oct 5, 2026
@linj-glitch
linj-glitch marked this pull request as draft October 5, 2026 06:12
@linj-glitch
linj-glitch force-pushed the linj/apptainer-private-tmp branch from adc9b0d to 5c2bba2 Compare October 8, 2026 00:15
Apptainer binds the host /tmp into every instance by default, so all
sandboxes on a node share one /tmp with each other and with host-side
processes (e.g. Ray's session directory under /tmp/ray). Agents commonly
create and delete scratch dirs under /tmp, and graders write fixed-name
files there, so concurrent sandboxes can clobber each other's files.

Add an opt-in `create.private_tmp` flag. When set, `instance start` gets
`--no-mount tmp` plus a bind of a per-instance `tmp` directory (mode
1777, inside the instance's staging dir) over /tmp. The directory is
removed together with the staging dir. Default off; no behavior change
for existing configs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch
linj-glitch force-pushed the linj/apptainer-private-tmp branch from 5c2bba2 to b6956ed Compare October 8, 2026 00:20
@linj-glitch
linj-glitch marked this pull request as ready for review October 8, 2026 00:30
@linj-glitch

Copy link
Copy Markdown
Collaborator Author

/ok to test b6956ed

@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 8, 2026 00:38
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Oct 9, 2026

@alexandery-nvidia alexandery-nvidia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving from the consumer side; this is one of the four Lin flagged for Ananth's focus.

The per-instance /tmp with the 1777 staging directory and the default-off knob look right, and the two tests pin both settings. Ran tests/unit_tests/test_apptainer_provider.py at the PR head locally: 70 passed. One gap worth closing or documenting, inline.

khazic pushed a commit to khazic/Gym-lao that referenced this pull request Oct 13, 2026
…eMo#4031)

## What does this PR do?

By default `apptainer instance start` bind-mounts the invoking user's
home into the instance. When the Gym server runs as root, as it usually
does inside a training container, that home is `/root`, so the server's
`/root` replaces the task image's `/root` in every `anyswe_agent`
sandbox on the node.

Task images keep toolchains under `/root`. Example: in the R2E-Gym image
`namanjain12/pillow_final:e40033dfc46f3bb3908079357a5eefc7c9db1725`,

```
/testbed/.venv/bin/python -> /root/.local/share/uv/python/cpython-3.9.21-linux-x86_64-gnu/bin/python3.9
```

With the server's home mounted over `/root` that link dangles. The agent
cannot run the repository's Python or tests, and the grader's
`run_tests.sh` (`.venv/bin/python -m pytest -rA r2e_tests`) fails with
"No such file or directory", so the rollout is graded unresolved.
Anything an agent or grader writes under `/root` also lands in the
server's own home and is visible to every other sandbox on the node.

`anyswe_agent` never needs the host home: the agent already runs with
`HOME=/sandbox/home`. This PR resolves the sandbox provider through a
new `_resolve_sandbox_provider()`, which for the Apptainer provider
appends `--no-home` to `create.extra_start_args`. It adds the flag once,
does not mutate the agent config, accepts `create` as a mapping or an
`ApptainerCreateConfig`, and returns other providers unchanged. The
README documents it.

Relation to other open PRs: NVIDIA-NeMo#4024 moves the grader's own files (eval
script, patch) to the per-instance `/sandbox` directory, so concurrent
graders stop overwriting each other. That PR deliberately does not use
`--no-home`. This PR addresses a different symptom: the task image's own
`/root` content being hidden. NVIDIA-NeMo#4005 adds an opt-in Apptainer
`private_tmp` for `/tmp`; this PR does not depend on it.

## Validation

- `python -m pytest -q responses_api_agents/anyswe_agent/tests
tests/unit_tests/test_apptainer_provider.py`: 95 passed (run with the
worktree's modules pre-imported; `nemo_gym` reorders `sys.path` on
import, which can otherwise pick up another editable checkout).
- New tests in
`responses_api_agents/anyswe_agent/tests/test_apptainer_home.py`:
- inline config: `--no-home` is appended after the existing args, other
`create`/`exec` keys are kept, and the input config is not mutated;
- named config (`sandbox_provider: sandbox` with `default_metadata`),
empty `create`, and a dataclass `create`;
  - the flag is never added twice;
  - Docker and OpenSandbox configs come back unchanged;
- `test_apptainer_instance_start_carries_no_home`: builds a real
`ApptainerProvider` from the resolved config (binary and `_run` faked)
and checks that the `apptainer instance start` argv contains `--no-home`
before the image.
- The new tests fail on `main` (`_resolve_sandbox_provider` does not
exist there).
- I confirmed the symlink above by listing the image's squashfs. I did
not run Apptainer itself for this PR: the login host I prepared it on
has no Apptainer.
- `ruff check`, `ruff check --select I` and `ruff format` (v0.9.9) on
the touched files: clean.

## Rollouts

Found in internal Claude Code evaluation and training with the Apptainer
provider and a Gym server running as root in its container. Our runs
have used `--no-home` for all anyswe sandboxes since; before it, R2E-Gym
repository interpreters under `/root` were unusable inside the
sandboxes. I have no public rollout numbers to share.

## Compatibility and benchmark impact

Only the Apptainer provider is affected. Sandboxes no longer see the
server user's home. A setup that relied on files from the server's home
inside the task sandbox would need to bind them explicitly
(`exec.default_binds` or `provider_options.binds`). anyswe already
requires a writable container root (it creates `/trajectories_mount`),
so writes under `/root` keep working. No config change.

## Checklist

- [x] I have read the [contributing
guidelines](https://github.com/NVIDIA-NeMo/Gym/blob/main/CONTRIBUTING.md).
- [x] The change is focused; unrelated "drive-by" edits are tracked as
separate issues/PRs.
- [x] Tests added or updated and pass locally, or N/A for docs-only /
non-code changes (so CI unit/server checks pass when applicable).
- [x] Documentation added or updated, or N/A with justification.
- [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so
CI lint/format checks pass). (ruff on the touched files.)
- [x] New source files include the required Apache-2.0 SPDX header.
- [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check
passes).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Lin Jia <linj@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch

Copy link
Copy Markdown
Collaborator Author

/ok to test 939b7a2

Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch

Copy link
Copy Markdown
Collaborator Author

/ok to test ad79e70

Apptainer's tmp mount group covers both /tmp and /var/tmp, so --no-mount tmp
dropped the host /var/tmp bind as well and only /tmp was re-bound; with a
plain SIF image anything writing /var/tmp hit the read-only image. The
instance now gets a private 1777 directory at each path (tmp and var_tmp
under its staging dir), both removed with the instance.

Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch

Copy link
Copy Markdown
Collaborator Author

Closed the /var/tmp gap in the code rather than the README: private_tmp: true now binds a second per-instance 1777 directory (<staging>/var_tmp) at /var/tmp, using the same staging-dir pattern as /tmp, so nothing in the sandbox falls through to the read-only image after --no-mount tmp. The README row and the mount-group paragraph say so; test_create_private_tmp_binds_per_instance_tmp asserts both binds and modes, and the default-path test asserts neither directory is created. tests/unit_tests/test_apptainer_provider.py: 72 passed; pre-commit clean.

Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch

Copy link
Copy Markdown
Collaborator Author

/ok to test c39a94d

This branch was previously deployed

1 inactive deployment
public — c39a94d2 Deployed Oct 25, 2026 by copy-pr-bot[bot] via release / finalize / notify #7198
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:sandbox Sandbox providers, isolation, and sandbox runtime contracts complexity:low Localized change in one scope with a small, straightforward review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants