Repository navigation
feat(RL): OpenEnv terminal-bench rollouts without Docker - #461
Open
ishandhanani wants to merge 5 commits into
Open
ishandhanani wants to merge 5 commits into
ishandhanani wants to merge 5 commits into
Conversation
…smoke recipe tbench2_env (huggingface/OpenEnv) in TB2_MODE=local runs task commands in its own container's shell, so an OpenEnv environment can serve rollouts on a pool node with no Docker and no nested container. examples/miles/openenv-tbench2-smoke.yaml runs the server as a one-node service (readiness on GET /health) and a benchmark step that exercises the client contract: reset(task_id), one exec, evaluate. benchmarks/rl/openenv/tbench2_smoke.py is that client; its README explains where OpenEnv servers sit in a recipe.
benchmarks/rl/miles/recipes/openenv_tbench2_qwen3.py keeps Miles's Terminal-Bench-2 adapter and shared launch helpers and swaps the GLM-4.7-Flash profile for a dense Qwen3-4B one (TP=2, EP=1, qwen parsers, TITO qwen3), with --num-rollout overridable and OPENENV_SITE appended to the ray job's PYTHONPATH so rollout workers import the env client without shadowing the image's packages. examples/miles/qwen3-4b-openenv-tbench2.yaml: the ray pool owns the node, the tbench2_env server rides on it in TB2_MODE=local, and the launcher points the adapter at localhost:8003. Local mode shares a task's source directory across concurrent episodes; the recipe says so.
openenv_launch_common.cleanup() does `pgrep -f sglang | xargs kill`. Every process in the Miles image runs under /opt/sglang/bin/python3, so the pattern matches the `ray start --block` head, its dashboard and agents in the sibling service container (enroot shares the PID namespace), and `ray job submit` then finds port 8265 refused. Job 15583 on sa-b200: head received SIGTERM at 08:04:47, cleanup started at the same second. Under srt-slurm the allocation is fresh and Ray belongs to the service, so the recipe skips the cleanup.
… the openenv README docs/miles.md gains an "OpenEnv without Docker" subsection: what TB2_MODE=local is, the two recipes, the two Python installs, why the upstream cleanup must not run, and what local mode does not isolate. benchmarks/rl/openenv/README.md records the client-only site directory procedure: resolve against the image's interpreter, install only what is missing with --no-deps, pin mcp 1.x.
Job 15591 ran the loop end to end (32 sessions, a GRPO step) but 31 evaluates failed on files another episode had removed: tbench2_env's local mode stages the verifier at /tests and /logs/verifier per container and runs agents in the shared task directory, so overlapping episodes corrupt each other. The example now sets MAX_CONCURRENT_ENVS=1 (the Miles adapter queues episodes on the capacity signal), sizes the rollout to 2 prompts x 4 samples, and keeps TB2_OUTPUT_DIR under /logs so per-episode terminal logs are job artifacts. docs/miles.md records the finding and that TB2_WITHHOLD_TESTS deletes tests/ and solution/ from the checkout on disk.
ishandhanani
marked this pull request as ready for review
September 16, 2026 08:49
ishandhanani
requested review from
alec-flowers,
csahithi,
hjjq,
kedarpotdar-nv,
kyleliang-nv,
nlevin-ui and
qiching
as code owners
September 16, 2026 08:49
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OpenEnv environments as pool services, no Docker and no hosted sandbox, driven by Miles on
main(#452 + #455).What
examples/miles/openenv-tbench2-smoke.yaml+benchmarks/rl/openenv/tbench2_smoke.py: the Terminal-Bench-2 env server (huggingface/OpenEnvtbench2_env,TB2_MODE=local) as a one-node generic service with aGET /healthreadiness probe; the benchmark step doesreset(task_id), oneexec,evaluateagainst it. No GPUs touched.benchmarks/rl/miles/recipes/openenv_tbench2_qwen3.py+examples/miles/qwen3-4b-openenv-tbench2.yaml: Miles's Terminal-Bench-2 adapter with a dense Qwen3-4B profile (TP=2, EP=1, qwen parsers, TITOqwen3). Therayservice owns the pool, the env server rides on it (placement.pool: train), and the launcher points the adapter atlocalhost:8003.OPENENV_SITEappends a client-only site dir to the ray job'sPYTHONPATH.benchmarks/rl/openenv/README.md: the two Python installs (a venv for the server; a client-only site dir for the rollout workers, resolved against the image's interpreter,--no-deps,mcppinned to 1.x) and why a venv with--system-site-packagesis not enough when the image'spython3is itself a venv.docs/miles.md: "OpenEnv without Docker" subsection.Validated on sa-b200
/health200,reset(headless-terminal)in 14 s, exec ran in the task shell,evaluatereturned the canonical harness verdictopenenv_launch_common.cleanup()doespgrep -f sglang | xargs kill; everything in the image runs under/opt/sglang/bin/python3and enroot shares the PID namespace, so it killed therayservice's head. The recipe no longer calls itray jobsucceeded, tachometer parquet writtenWhat 15591 also showed
62 of 64 episodes were dropped with no canonical verdict and rewards were 0. Local mode stages the verifier at
/testsand/logs/verifierper container and runs agents in the shared task directory, so 32 concurrent episodes corrupt each other. The example now runsMAX_CONCURRENT_ENVS=1(the Miles adapter queues episodes on the capacity signal), 2 prompts x 4 samples per step, and keepsTB2_OUTPUT_DIRunder/logs. That configuration has not been run yet.TB2_WITHHOLD_TESTS=1deletestests/andsolution/from the checkout on disk;git checkout -- .between runs.Real parallelism needs one filesystem per episode: per-episode enroot containers nested in the trainer's container (works on sa-b200 with the host's enroot tooling at its own paths) or a hosted sandbox provider. Neither is in this PR.
No
src/srtctlchanges.