Repository navigation
feat(deepcad): run seven harnesses through shared agent sessions - #6254
gerryche123 wants to merge 21 commits into
Conversation
Signed-off-by: Rohit Jena <rohit.rango@gmail.com>
Signed-off-by: Rohit Jena <rohit.rango@gmail.com>
Co-authored-by: Yi-Fu Wu <yifu.wu@gmail.com> Signed-off-by: Rohit Jena <rohitrango@users.noreply.github.com>
Signed-off-by: Rohit Jena <rohit.rango@gmail.com>
Signed-off-by: Rohit Jena <rohit.rango@gmail.com>
Co-authored-by: Yi-Fu Wu <yifu.wu@gmail.com> Signed-off-by: Rohit Jena <rohitrango@users.noreply.github.com>
Co-authored-by: Yi-Fu Wu <yifu.wu@gmail.com> Signed-off-by: Rohit Jena <rohitrango@users.noreply.github.com>
Signed-off-by: Rohit Jena <rohit.rango@gmail.com>
Signed-off-by: Rohit Jena <rohit.rango@gmail.com>
…ion budgets Signed-off-by: Tong Che <tongc@nvidia.com>
d8bd8f5 to
6540d21
Compare
Signed-off-by: Tong Che <tongc@nvidia.com>
yfw
left a comment
There was a problem hiding this comment.
HOLD. Two blockers: the generation budget never stops a harness (nemo_gym/models/rollout_budget.py:83-94), and test_harness_configs.py:80 turns the CI Server suite red.
Running one DeepCAD environment through seven stock agents with _copy environment servers is the shape Gym wants for multi-harness work. Routing, configuration keys, /ng-sampling/ and rollout correlation are correct for all seven harnesses. The merge with main kept the prepare_row and lazy-row behavior from #5950. With the budget off, nothing changes for other environments. After the fixes in the comments below, this recipe can be the reference example for one environment run by many harnesses.
Process:
- No merge conflict with main now, so no rebase is needed.
- DCO passes.
- CI ran only DCO and the lock-freshness job. A maintainer needs to comment
/ok to testwith the full 40-character head SHA to start the other jobs.
Local runs (CPU only):
- pre-commit passes with no rewrites.
- deepcad tests in deepcad's own venv: 327 passed, 2 failed (see
test_harness_configs.py:80), 4 skipped. - The PR's core budget tests: 11 of 11 pass. All model tests: 2308 passed.
Notes:
- The comments on
rollout_budget.py:83-94andbase_responses_api_model.py:286are about the budget, so their design advice applies to its core PR. - The test comments on
test_harness_configs.py:41-42andtest_opencode_manifest.py:186-192are small and do not block.
Generated by Claude Code
| for agent in self.config.agent_servers: | ||
| try: | ||
| response = await server_client.post( | ||
| server_name=agent, | ||
| url_path=f"/rollouts/{rollout_id}/stop", | ||
| json={"reason": BUDGET_STOP_REASON}, | ||
| ) | ||
| await raise_for_status(response) | ||
| except ClientResponseError as error: | ||
| if error.status == 404: | ||
| continue | ||
| raise |
There was a problem hiding this comment.
nemo_gym/models/rollout_budget.py:83-94
On call 11 this loop stops no harness: the first agent in agent_servers answers HTTP 501, and the loop returns that 501 to the harness.
OPENCODE.md says what the budget is for:
This enables grading the artifact when the shared ten-call budget stops the agent
This is what happens on call 11 instead:
- These lines post
/rollouts/{id}/stopto each name inagent_servers, in order. The loop goes on only after a 404 and raises every other error. SimpleResponsesAPIAgent.stop_rolloutanswers501 "This agent cannot stop a rollout"for every agent that does not override_rollout_sandbox_session, before it looks up the rollout (base_responses_api_agent.py L1389-L1390). Of the seven agents, onlycodex_agentoverrides it (codex_agent/app.py L794).deepcad_opencode_agentis first in both lists (opencode.yaml L105, multiharness.yaml L121-L128). So every rollout that makes an 11th call gets HTTP 501 at the harness, Codex rollouts included.- No agent is stopped, so no response carries the
ng_rollout_stopmarker, and the budget-stop grading branch never runs (opencode_app.py L201-L215). - When a harness gives up on that 501, every adapter except Claude Code reports a failed model call. The default outcome policy records that as a retryable failure with no reward (agent_outcome.py L95, L113-L120). A retried attempt gets a new
-a<N>rollout ID and a new counter, so the cap is ten calls for each attempt. Claude Code reports no outcome, so its failed run goes to/verifyand scores an unmasked 0 (deepcad:agent_incomplete) (agent_outcome.py L136-L138).
Even Codex cannot stop on the recipe's Docker workspace. After launch, SandboxSession.expire raises ExpiryUnsupportedError unless the provider uploads files independently of running commands, as Local and OpenSandbox with background_exec do (sandbox_session.py L339-L343). stop_rollout turns that error into a 501 too (base_responses_api_agent.py L1405-L1406).
The PR introduced this. The budget tests use the _stoppable_agent test fixture (test_agent_sessions.py L481-L485), never a real harness. The PR's own test_stop_checks_actual_http_status_before_identity[501] expects this raise (test_rollout_budget_integration.py L93-L97).
Suggested fix: please move the generation budget out of this PR into its own core PR. It is a multi-file change, so it cannot be a suggestion. The lines to remove are in the fold below. Without those lines, this PR can merge before the core PR. The core PR can then land with these three parts:
- Stop support for every agent that it lists. feat(environment): stop the agent once a live world ends the episode #5838 (open) adds it for OpenCode only.
- A test that builds each listed agent and fails unless its stop answers 404, not 501, for an unknown rollout.
- The provider requirement above, in its docs.
Without the budget, the recipe has no hard ten-call bound. OpenCode keeps its advisory build.steps (opencode.yaml L67-L68). Hermes and Claude Code keep max_turns (hermes_agent.yaml L11, claude_code_agent.yaml L11). Pi, Codex and OpenClaw set no call limit in their configurations, and NOOA leaves max_policy_calls unset (nooa_agent/config.py L62-L64). The 1800 s agent timeout is the hard bound (opencode.yaml L51).
Two smaller fixes look tempting and do not work. Treating 501 like 404 in this loop only turns call 11 into the 503 "no agent owns this rollout" (L106) for the six other agents. Putting codex_agent first fixes only Codex, and Codex still fails on Docker.
Lines to remove from this PR
nemo_gym/models/rollout_budget.py, the whole file.- In
base_responses_api_model.py: the provenance header (L15-L16), the import (L87), therollout_generation_budgetfield (L286), the_generation_budgetattribute (L355), its setup (L398-L399),_admit_generation(L874-L876) and its four calls (L902, L910, L969, L977), and the refusal ofnother than 1 (L881-L882). tests/unit_tests/models/test_rollout_budget.pyandtests/unit_tests/models/test_rollout_budget_integration.py.- Both
rollout_generation_budgetblocks: opencode.yaml L101-L105 and multiharness.yaml L120-L128. If you do not add theinline_tool_output_imagesline from the comment on multiharness.yaml L119, remove L117-L119 too. An emptyvllm_model:there replaces the wholevllm_modelblock from opencode.yaml. rollout_correlation: true(opencode.yaml L4), which none of the seven agents needs: they add the rollout prefix with or without it.- The budget-stop branch in
opencode_app.py(L201-L208 andor budget_stoppedat L213), and the import that it leaves unused (L18). - In
test_harness_configs.py: L34 and L72-L75. - In
test_opencode_manifest.py: theCAD_BUDGET_DIRlines L129, L246 and L294. - In
test_opencode_sessions.py: the testtest_only_incomplete_native_budget_stop_can_collect_file(L210-L232), thebudget_stoppedcase of L256-L275, and the import that they leave unused (L13). - In
OPENCODE.md: the budget-stop exception, from "An incomplete response" to "remain ineligible." (L23-L28), in L100-L116 the budget-directory paragraph (L100-L102) and theCAD_BUDGET_DIRlines (L105 and L113), the budget limit (L141-L144), and the budget mentions at L157 and L172. - The
CAD_BUDGET_DIRtext in the PR body.
|
|
||
|
|
||
| class BaseResponsesAPIModelConfig(BaseRunServerInstanceConfig): | ||
| rollout_generation_budget: RolloutGenerationBudgetConfig | None = None |
There was a problem hiding this comment.
nemo_gym/models/base_responses_api_model.py:286
rollout_generation_budget is a setting of the whole model server, so every environment on the shared policy_model also gets the ten-call cap.
The budget counts calls by rollout ID only (rollout_budget.py L71-L77), and rollout_correlation: true (opencode.yaml L4) gives every agent's calls a rollout ID. The budget has no environment filter. NeMo-RL recipes run many environments on one policy_model, for example RL prefetch_super_all_envs.yaml L66-L81. A NeMo-RL run that loads the CAD opencode.yaml next to example_multi_step.yaml caps the example_multi_step rollouts at ten calls too. Their call 11 fails, with a 501 today and with a 503 once the stop works. The PR introduced this.
Suggested fix, for the core PR that the comment on rollout_budget.py L83-L94 asks for: let the environment server carry the cap with each episode. It already carries sampling values that way, in the /ng-sampling/ segment (base_environment_server.py L359-L368). The model server then counts and stops only the rollouts that carry a cap. This also removes the agent_servers list and its ordering problem. It is a design change in several files, so it cannot be a suggestion.
A startup guard that refuses other agents on this model server is the worse fix. It blocks every run that puts CAD next to other environments on one policy_model, and NeMo-RL trains that way.
| @@ -0,0 +1,128 @@ | |||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
| # SPDX-License-Identifier: Apache-2.0 | |||
| # Seven concurrent native pairings; prepare each row with --file-input --harness NAME. | |||
There was a problem hiding this comment.
resources_servers/deepcad/configs/multiharness.yaml:3
None of the seven pairings in this recipe has a real-model rollout yet, and Gym asks for real smoke rollouts for this kind of change.
Gym's AGENTS.md says:
For behavior-changing environment or agent work: run representative real smoke rollouts with a model and inspect agent and verifier behavior.
The PR body says that the rollouts are pending and that "This remains a draft", but the PR is out of draft.
Suggested fix:
- Please run the five example tasks (
data/example.jsonl) through each pairing that you keep. For each harness, please report the number of episodes, the mean reward, and whether a target PNG and a preview PNG reached the policy model. Please keep a harness in the recipe only after its smoke run passes. The Harness-Ready Environments runbook covers running one environment through several harnesses. - Please list these changes under "Compatibility and benchmark impact" in the PR body:
- The
INSTRUCTIONStext changed for every OpenCode row, standalone and lazy rows included (prepare_opencode.py L21-L23). - For prepared rows (not lazy rows), sampling values move to the
_ng_model_samplingmarker (prepare_opencode.py L117-L119). A row's output cap then applies as the smaller of the row's cap and the harness's own cap (OPENCODE.md L145-L147). Before, OpenCode refused a row cap below its own (opencode_agent/app.py L1557-L1567). Lazy rows keep the values in the request as well, so OpenCode still refuses a smaller cap there. --file-inputOpenCode rows have no system message. Sosystem_prompt_mode: replaceno longer replaces OpenCode's build prompt, and the CAD text becomes user text (opencode_agent/app.py L1899-L1904).
- The
Signed-off-by: Tong Che <tongc@nvidia.com>
Signed-off-by: Tong Che <tongc@nvidia.com>
Signed-off-by: Tong Che <tongc@nvidia.com>
Signed-off-by: Tong Che <tongc@nvidia.com>
Signed-off-by: Tong Che <tongc@nvidia.com>
Signed-off-by: Tong Che <tongc@nvidia.com>
Signed-off-by: Tong Che <tongc@nvidia.com>
Signed-off-by: Tong Che <tongc@nvidia.com>
Signed-off-by: Tong Che <tongc@nvidia.com>
What does this PR do?
Adds a CAD recipe composing seven agent configurations with the shared CAD resources server and final-file verifier from merged #5950. #5720 is the Agentic-MME composition reference. No separate issue is needed for this incremental onboarding work.
Work in progress following review: seven configured routes are not seven qualified real-model pairings. Real-model pairing and training qualification below remain open.
Commit
e0e16688a925524f18cebfe1bd946ff7aa10d80eaddresses configuration, image settings and test coverage:deepcad_<harness>_agentname inherited from the stock configuration. Environment routing and preparedagent_refagree.deepcad_policy_modelcopies the deployment'spolicy_model. CAD budget, image and token settings apply to the copy; the original model configuration is unchanged. Both use the same backend._ng_model_sampling; RL can enforce its own sampling policy.Configured harnesses: OpenCode, Pi, Hermes, Codex, Claude Code, OpenClaw and NOOA. All share the existing resources server and geometry scorer.
Remaining blockers
c09f03bab0fd97327b2313e4026bb069ea6a9305; Codex already used that path. The Docker upload-capacity fix is included in6e87e4beb6963cf2bb75d3f78fc51e055849d2a3. Actual end-to-end stop and budget-stop grading across all seven native clients remain unqualified; adapter unit tests do not establish a uniform ten-generation experiment.Validation
OpenSandbox runtime reuse on exact public
f98268eedalso passed: native installation verified the three pinned NOOA packages and preserved CadQuery 2.8, OpenCode, its plugin archive and the renderer; a fresh sandbox with denied network egress then reused the same source-bound runtime. Both owned sandboxes confirmed teardown. This prepares runtime reuse for real-model evaluation; it is not policy quality or training-token acceptance.Installed public NOOA control on exact
f98268eed: the existing installer provisioned all three public19caab169b018476ac433d040f6ae3f06aeff101packages inside the non-root linux/amd64 CAD workspace, verified the source fingerprint96f48a7bacf877535747a9fd87499658bf9943d9647fff1f8d1066f50f0f8fdb, and reused the same runtime generation. The installed native entrypoint made two actual HTTP calls to a scripted backend; the second contained all three original CAD target PNGs plus the retained renderer preview byte-for-byte and in order, with sampling 0.6/0.95/4096 and matching served references. UID 10001 asset reads, child termination, container teardown and backend-port cleanup passed. The first fixture attempt remains failed because root-owned mode-0600 test assets were unreadable; the corrected attempt used identical bytes with mode 0644 and an explicit target-user read check. Installer scratch-removal warnings remain recorded; the owned container was subsequently removed. This validates installed runtime and image transport on Docker, not OpenSandbox snapshot reuse, a full AgentHTTP/ModelServer run, real-policy quality, or canonical training tensors.Hermes file-image runtime at
f98268eed: the CAD recipe enables image input and the existing vision tool. The existingSandboxPythonRuntimeinstaller provisions isolated Hermes0ee05d72f2b30385d4861fd8e30511c6d51100e4with pinned dependencies; the default host runtime is unchanged. The adapter binds the native runtime context to the same primary model endpoint, so file images do not require an auxiliary model. A real linux/amd64 Docker cold-install/reuse control and native-client scripted HTTP control passed: target and terminal-generated preview pixels preserved in order, four served model calls, fifth-call 409 budget denial, matching close references, stopped session and complete sandbox/process cleanup. No GPU or real-policy inference was used. Primary-checkout image, sandbox and CAD configuration suites passed 236 tests, 3 skipped; scoped hooks passed, with the private guardword catalog unavailable. Numeric temperature mapping was source-audited but not exercised by this image fixture. Real-model CAD quality and canonical training tensors remain unqualified.Upstream merge at
32f7d2700: the only conflict was adjacent additions in the converter test file; both image-preservation and upstream thinking-block tests were retained. On the exact merged tree, converter/NOOA suites passed 324 tests, and CAD configuration/manifest, model-alias capture and rollout-budget suites passed 58 tests with 1 optional Hermes-dependency skip. Scoped hooks passed; the private guardword catalog was unavailable. These tests do not qualify post-merge native client rollouts.CAD workspace permissions at
84041f93a: the image now creates/opt/nemo-gym-nooaowned by the existing non-rootcaduser, as required by NOOA's native installer. An actual linux/amd64 Docker run with networking disabled, all capabilities dropped and no-new-privileges verified UID 10001, runtime-directory writes, a non-writable/optparent, required installer utilities and CadQuery 2.8 solid generation. This is a prerequisite check, not an installed NOOA runtime or real-model result. Scoped hooks completed; the private guardword catalog was unavailable.Public NOOA at
785845f45also passed a separate nativesandbox_entrypoint/Runner CPU control with two actual HTTP requests to a scripted backend. The second request preserved three CAD target PNGs and the renderer-produced preview byte-for-byte and in order, sampling 0.6/0.95/4096, served response IDs and native child/server cleanup. Native tools and prefill remained enabled. This validates local native HTTP image transport, not an installed OpenSandbox runtime or model quality.NOOA image conversion at
785845f45: 320 tests passed across the shared converter and GymLLM suites using the actual pinned NOOA package. The adapter reuses the shared converter, preserving ordered image parts, existing Responses-format tool content, call IDs and caller inputs. These are conversion/replay tests, not native target/preview delivery or model quality acceptance. Scoped hooks and package layering passed; the private guardword catalog was unavailable.Shared stop integration at
c09f03bab0fd97327b2313e4026bb069ea6a9305: 76 focused tests passed, 4 skipped. Tests cover adapter hooks, shared seed/stop lifecycle, idempotent stopping and preserved response metadata; optional Hermes/NOOA runtime imports caused the four skips. They use mocked sandbox execution and do not qualify seven native CLI runs. Scoped hooks passed; the private guardword catalog was unavailable.Docker provider fix at
6e87e4beb6963cf2bb75d3f78fc51e055849d2a3: 91 provider tests passed. A real Docker smoke with one exec slot occupied confirmed that a separate bounded upload could deliver the expiry marker, the native supervisor stopped the process, the exec client returned, and the owned container was removed. The corrected smoke passed with no cleanup errors; the first attempt's fixture incorrectly treated its started sandbox as borrowed and remains recorded as failed. This validates the Docker provider path, not seven-harness stopping, CAD scoring, or model quality. Scoped hooks passed; the private guardword catalog was unavailable.Earlier configuration-only validation at
e0e16688a, against an isolated checkout ofbbaf99426839daf3f1a539e3bdc74d9e54b57fffplus its seven changed files:46 passed, 2 skipped, Python 3.13.14, existing Gym development venv. The two skips are optional Hermes/NOOA runtime boundary checks; all seven routing/configuration cases executed. No model or native client was run.
The model-alias regression sends requests through two real model ASGI applications and retrieves both manifests through the original
policy_modelcontrol endpoint. It checks model identity and call/response IDs. Worker acknowledgements are synthetic; this is not real tensor or optimizer validation.Scoped pre-commit, Ruff, formatting and whitespace checks passed. The private guardword catalog was unavailable, so that content scan was skipped. The commit has DCO sign-off. These checks do not establish full CI or rollout readiness.
Compatibility and benchmark impact
policy_modelbefore the CAD recipe, which createsdeepcad_policy_model._ng_model_sampling. Their output limit is the smaller of the row and native request caps. Lazy rows retain their request fields too, and OpenCode can still reject a row cap below its own.target_imagesfield. Lazy file-input conversion requires materialization first.CAD_BUDGET_DIR. The current stop limitation above means the intended hard ten-generation stopping/grading protocol must not be treated as validated.Code provenance
The admission counter, model dispatch hooks and budget tests are adapted from NVIDIA NeMo Gym, Apache-2.0, at d8c548915431e59169890d34cd8cea56ef2a905e. Original and affected paths:
nemo_gym/models/rollout_budget.pynemo_gym/models/base_responses_api_model.pytests/unit_tests/models/test_rollout_budget.pytests/unit_tests/models/test_rollout_budget_integration.pyEach affected file retains the source revision/license. The port excludes final-call prompt/tool-choice steering and unrelated harness/capture/scorer changes.
Hermes image runtime dependencies are resolved from the MIT-licensed upstream pyproject.toml at the exact pinned revision into
responses_api_agents/hermes_agent/image-runtime-constraints.txt; that file records the revision, license and generation command. The adapter calls upstream APIs; it does not vendor the image implementation.