Skip to content

feat(sandbox): verify code_gen submissions in sandboxes - #4607

Merged
ananthsub merged 6 commits into
mainfrom
yifuw/sandbox-verify-code-gen
Oct 21, 2026
Merged

ananthsub merged 6 commits into
mainfrom
yifuw/sandbox-verify-code-gen

Conversation

@yfw

@yfw yfw commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Part of #3216 (migrate in-scope resources servers to the Sandbox API). Running model-written code in remote sandboxes keeps it off the training cluster.

This adds an opt-in sandbox path to code_gen. With sandbox_provider set, each verification runs the unchanged LCB harness in a fresh provider sandbox instead of a Ray worker, with the same per-test time and memory limits. Without it, behavior is unchanged.

  • Shared lifecycle: nemo_gym/verifier_utils/sandbox_verification.py provides run_verification_sandbox, SandboxVerificationConfig and SandboxVerificationError. It starts a sandbox, uploads the harness and request, runs it and parses the result, and always stops the sandbox. The CCC sandbox path (stacked PR) reuses it.
    • Each call works in its own subdirectory, so providers that share /tmp between sandboxes (local, and Apptainer by default) can't mix up concurrent verifications.
    • A missing provider SDK (a venv without nemo-gym[sandbox]) raises instead of masking every sample.
  • Masking: a sandbox or runner failure returns mask_sample with failure_kind provider_unavailable or verifier_error, not a zero reward. Wrong answers, timeouts and crashes are still normal results.
  • No answers in reach: the default network policy denies all egress on OpenSandbox, so submissions can't fetch answers. The runner deletes request.json, which holds the expected outputs, before running the candidate, so the sandbox exposes no more than a Ray worker does.
  • Core dumps: the runner sets a one-byte RLIMIT_CORE before running candidates. The AWS cells pipe core dumps to systemd-coredump with an unlimited core limit, so every crash took 4–7 s to write a dump. Near a time limit, that turned crashes into timeouts. A zero limit doesn't help because the kernel ignores it for piped dumps; at exactly 1 byte it skips the dump.
  • What runs where: everything that runs only in the sandbox is in resources_servers/code_gen/sandbox/: sandbox_runner.py, uploaded with every verification, and the image's Dockerfile. sandbox_execution.py runs in the server and names the upload set: the runner plus three lcb_integration grading files (__init__.py, correctness.py, testing_util.py), which the Ray path also runs. Nothing else is uploaded. The README has a table of what runs where.
  • Image: sandbox/Dockerfile, with the server venv's CPython build and numpy, scipy and pandas versions (scipy and pandas are pinned in requirements.txt too), so candidates that import them grade the same as on Ray. The config reads the image from CODE_GEN_SANDBOX_IMAGE.
  • Config example: the README shows how to enable the sandbox with a config file as well as with command-line overrides, and which settings only some providers support.

Validation

  • Unit tests (resources_servers/code_gen venv) with CODE_GEN_SANDBOX_IMAGE set to an image built from sandbox/: 73 passed. Without the image, the Docker test is skipped. tests/unit_tests/test_sandbox_verification.py: 11 passed. Regression tests that fail without their fixes: test_candidates_cannot_dump_core (RLIMIT_CORE), test_candidate_cannot_read_expected_outputs_from_the_request, test_each_call_gets_its_own_work_dir and test_missing_provider_sdk_fails_loudly.
  • pre-commit run on the changed files: clean.
  • Layout check on Cell-5 after moving the sandbox files to sandbox/: sandbox_execution.py uploaded the runner from there and graded a correct answer, a correct answer that imports scipy, and a wrong answer as expected.
  • Docker provider with the image built from this branch: the opt-in memory-limit test passes, and correct solutions that import scipy or pandas pass.
  • 80 concurrent verifications on the local provider, half right and half wrong: 0 wrong rewards.
  • Both README launch configurations resolve with gym env resolve.
  • LCB replay on Cell-5 (shipped config, ECR image): 1362 stored rollouts regraded with 64 concurrent sandboxes.
    • 0 masked.
    • Mean reward 0.8722 vs 0.8715 stored.
    • 3 per-rollout differences, all tests right at their time limit; two flip in opposite directions between runs.
    • This replay and the scale test below used the earlier numpy-only image.
  • RL scale test (Nemotron 3.5 Super, 64 nodes, Cell-5): a code-only RLVR recipe (its code_gen and CCC rows), sandbox vs Ray workers, otherwise identical.
    • 0 masked samples, 0 sandbox failures, no rate limiting.
    • code_gen reward 0.541 vs 0.565 over steps 1–9; per-step differences change sign (noise).
    • Verification takes 13–15 s per rollout in a sandbox (create, upload, run) vs 0.1–12 s on Ray. Mean step time is 346 vs 317 s, dominated by long generations.
  • Egress smoke on Cell-5: network access is blocked. A correct answer, a wrong answer and a submission that tries to fetch the answer are all graded correctly.

Rollout evidence

The RL scale test above. Earlier, a 5-step GRPO smoke run on HSG, host mode vs sandbox mode, otherwise identical:

  • code_gen reward 0.095 vs 0.096.
  • Generation time +41% and step time +26% in sandbox mode.

Compatibility

Opt-in only. Existing configs, and runs without sandbox_provider, are unchanged.

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Oct 14, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yfw yfw added area:sandbox Sandbox providers, isolation, and sandbox runtime contracts feature New capabilities, enhancements, or enablement work labels Oct 14, 2026
@yfw
yfw force-pushed the yifuw/sandbox-verify-code-gen branch from 4bc13be to dd37714 Compare October 14, 2026 19:24
@yfw
yfw marked this pull request as ready for review October 16, 2026 10:14
@yfw

yfw commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 6999591

@yfw yfw added the needs-review PR is ready for code review and waiting on a reviewer label Oct 16, 2026
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 16, 2026 10:20
@yaoyu-33 yaoyu-33 added the complexity:high Cross-component or shared-contract change with a large review surface label Oct 16, 2026
@yfw

yfw commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test fc91e33

yfw and others added 4 commits October 21, 2026 14:51
Add an opt-in sandbox path to code_gen: with sandbox_provider set, each
verification runs the unchanged LCB harness in a fresh provider sandbox
instead of a Ray worker, with the same per-test limits. Sandbox or runner
failures return mask_sample with failure_kind provider_unavailable or
verifier_error instead of a zero reward. The default network policy denies
all egress. The runner sets a one-byte RLIMIT_CORE so hosts that pipe core
dumps to systemd-coredump (as the AWS cells do) do not stall a crashing
candidate for seconds and turn it into a timeout.

The sandbox lifecycle lives in nemo_gym.sandbox.verification
(run_verification_sandbox, SandboxVerificationConfig,
SandboxVerificationError) so other verifiers can reuse it; the
competitive_coding_challenges sandbox path will build on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
Shows how to enable the sandbox path with a config file (provider config via config_paths,
sandbox_provider, image), alongside the existing command-line overrides, and points the
server config's comment at it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
- Delete request.json before grading: it holds every expected output, and
  candidates run in its directory with a real open().
- Give each verification its own work subdirectory, so providers that share
  /tmp between sandboxes (local, Apptainer) cannot mix up concurrent results.
- Let ImportError from a missing provider SDK propagate instead of masking
  every sample as provider_unavailable.
- Add scipy and pandas to the verifier image at the versions the code_gen venv
  resolves, and pin them in requirements.txt, so candidates that import them
  grade the same as on Ray.
- README: add --model-type to the launch examples, note the stack hard limit,
  and clarify num_processes, CPU limits, network_policy support and masking.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
The runner cannot raise a zero hard limit to one byte, so the test failed
instead of skipping on hosts that set it that way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
yfw and others added 2 commits October 21, 2026 14:51
…tils

nemo_gym.sandbox holds only the sandbox lifecycle APIs and providers. The
single-use verification helper is a verifier utility built on them, so it
moves to nemo_gym/verifier_utils/sandbox_verification.py, next to
agent_utils for CLI harnesses. The code is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
The runner and the image's Dockerfile sat at the server root next to the
server modules, so it was hard to tell what runs in the sandbox. Both now
live in sandbox/: sandbox_runner.py runs only in the sandbox, and the
Dockerfile builds its image. sandbox_execution.py names the upload set
explicitly: the runner plus the three lcb_integration grading files the
Ray path also runs. Inside the sandbox the files keep their paths, and no
grading code changes. The README gains a table of what runs where.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
@ananthsub
ananthsub force-pushed the yifuw/sandbox-verify-code-gen branch from fc91e33 to 7b52b4b Compare October 21, 2026 21:51
@ananthsub

Copy link
Copy Markdown
Contributor

/ok to test 7b52b4b

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/39690556063

Merged via the queue into main with commit e6b2da9 Oct 21, 2026
51 of 57 checks passed
@ananthsub
ananthsub deleted the yifuw/sandbox-verify-code-gen branch October 21, 2026 23:23

This branch was previously deployed

1 inactive deployment
public — 7b52b4b5 Deployed Oct 21, 2026 by copy-pr-bot[bot] via release / finalize / notify #6492
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:sandbox Sandbox providers, isolation, and sandbox runtime contracts complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants