Skip to content

feat(deepcad): run seven harnesses through shared agent sessions - #6254

Open
gerryche123 wants to merge 21 commits into
NVIDIA-NeMo:mainfrom
gerryche123:tongc/multi-harnesses-support-for-rl-tasks
Open

gerryche123 wants to merge 21 commits into
NVIDIA-NeMo:mainfrom
gerryche123:tongc/multi-harnesses-support-for-rl-tasks

Conversation

@gerryche123

@gerryche123 gerryche123 commented Nov 16, 2026 •

Copy link
Copy Markdown

What does this PR do?

Adds a CAD recipe composing seven agent configurations with the shared CAD resources server and final-file verifier from merged #5950. #5720 is the Agentic-MME composition reference. No separate issue is needed for this incremental onboarding work.

Work in progress following review: seven configured routes are not seven qualified real-model pairings. Real-model pairing and training qualification below remain open.

Commit e0e16688a925524f18cebfe1bd946ff7aa10d80e addresses configuration, image settings and test coverage:

  • Each CAD agent has a deepcad_<harness>_agent name inherited from the stock configuration. Environment routing and prepared agent_ref agree.
  • deepcad_policy_model copies the deployment's policy_model. CAD budget, image and token settings apply to the copy; the original model configuration is unchanged. Both use the same backend.
  • Tool-output image conversion is enabled on the CAD model; OpenClaw declares image input.
  • The recipe no longer requires a sampling marker on every request. Prepared rows still preserve source sampling in _ng_model_sampling; RL can enforce its own sampling policy.
  • Configuration tests load the deployment model first, instantiate its actual typed configuration, check complete file-input activations, and compare blended benchmark configurations.

Configured harnesses: OpenCode, Pi, Hermes, Codex, Claude Code, OpenClaw and NOOA. All share the existing resources server and geometry scorer.

Remaining blockers

  • The six remaining native adapters now expose their existing supervisor to the shared rollout-stop lifecycle in c09f03bab0fd97327b2313e4026bb069ea6a9305; Codex already used that path. The Docker upload-capacity fix is included in 6e87e4beb6963cf2bb75d3f78fc51e055849d2a3. Actual end-to-end stop and budget-stop grading across all seven native clients remain unqualified; adapter unit tests do not establish a uniform ten-generation experiment.
  • Public Hermes target-file and generated-preview transport passed the local native scripted control described below. Public NOOA passed installed-runtime HTTP image transport and fresh OpenSandbox snapshot reuse, as described below. Neither control establishes real-model quality or training-token qualification.
  • Fresh real-model rollouts of the five bundled CAD examples through each retained pairing remain pending. Required evidence includes episode count, mean reward, target and preview images reaching the policy model, captured calls and cleanup.
  • Full CI and the isolated DeepCAD environment test command remain pending. Local tests using an existing Gym development environment do not replace them.

Validation

OpenSandbox runtime reuse on exact public f98268eed also passed: native installation verified the three pinned NOOA packages and preserved CadQuery 2.8, OpenCode, its plugin archive and the renderer; a fresh sandbox with denied network egress then reused the same source-bound runtime. Both owned sandboxes confirmed teardown. This prepares runtime reuse for real-model evaluation; it is not policy quality or training-token acceptance.

Installed public NOOA control on exact f98268eed: the existing installer provisioned all three public 19caab169b018476ac433d040f6ae3f06aeff101 packages inside the non-root linux/amd64 CAD workspace, verified the source fingerprint 96f48a7bacf877535747a9fd87499658bf9943d9647fff1f8d1066f50f0f8fdb, and reused the same runtime generation. The installed native entrypoint made two actual HTTP calls to a scripted backend; the second contained all three original CAD target PNGs plus the retained renderer preview byte-for-byte and in order, with sampling 0.6/0.95/4096 and matching served references. UID 10001 asset reads, child termination, container teardown and backend-port cleanup passed. The first fixture attempt remains failed because root-owned mode-0600 test assets were unreadable; the corrected attempt used identical bytes with mode 0644 and an explicit target-user read check. Installer scratch-removal warnings remain recorded; the owned container was subsequently removed. This validates installed runtime and image transport on Docker, not OpenSandbox snapshot reuse, a full AgentHTTP/ModelServer run, real-policy quality, or canonical training tensors.

Hermes file-image runtime at f98268eed: the CAD recipe enables image input and the existing vision tool. The existing SandboxPythonRuntime installer provisions isolated Hermes 0ee05d72f2b30385d4861fd8e30511c6d51100e4 with pinned dependencies; the default host runtime is unchanged. The adapter binds the native runtime context to the same primary model endpoint, so file images do not require an auxiliary model. A real linux/amd64 Docker cold-install/reuse control and native-client scripted HTTP control passed: target and terminal-generated preview pixels preserved in order, four served model calls, fifth-call 409 budget denial, matching close references, stopped session and complete sandbox/process cleanup. No GPU or real-policy inference was used. Primary-checkout image, sandbox and CAD configuration suites passed 236 tests, 3 skipped; scoped hooks passed, with the private guardword catalog unavailable. Numeric temperature mapping was source-audited but not exercised by this image fixture. Real-model CAD quality and canonical training tensors remain unqualified.

Upstream merge at 32f7d2700: the only conflict was adjacent additions in the converter test file; both image-preservation and upstream thinking-block tests were retained. On the exact merged tree, converter/NOOA suites passed 324 tests, and CAD configuration/manifest, model-alias capture and rollout-budget suites passed 58 tests with 1 optional Hermes-dependency skip. Scoped hooks passed; the private guardword catalog was unavailable. These tests do not qualify post-merge native client rollouts.

CAD workspace permissions at 84041f93a: the image now creates /opt/nemo-gym-nooa owned by the existing non-root cad user, as required by NOOA's native installer. An actual linux/amd64 Docker run with networking disabled, all capabilities dropped and no-new-privileges verified UID 10001, runtime-directory writes, a non-writable /opt parent, required installer utilities and CadQuery 2.8 solid generation. This is a prerequisite check, not an installed NOOA runtime or real-model result. Scoped hooks completed; the private guardword catalog was unavailable.

Public NOOA at 785845f45 also passed a separate native sandbox_entrypoint/Runner CPU control with two actual HTTP requests to a scripted backend. The second request preserved three CAD target PNGs and the renderer-produced preview byte-for-byte and in order, sampling 0.6/0.95/4096, served response IDs and native child/server cleanup. Native tools and prefill remained enabled. This validates local native HTTP image transport, not an installed OpenSandbox runtime or model quality.

NOOA image conversion at 785845f45: 320 tests passed across the shared converter and GymLLM suites using the actual pinned NOOA package. The adapter reuses the shared converter, preserving ordered image parts, existing Responses-format tool content, call IDs and caller inputs. These are conversion/replay tests, not native target/preview delivery or model quality acceptance. Scoped hooks and package layering passed; the private guardword catalog was unavailable.

Shared stop integration at c09f03bab0fd97327b2313e4026bb069ea6a9305: 76 focused tests passed, 4 skipped. Tests cover adapter hooks, shared seed/stop lifecycle, idempotent stopping and preserved response metadata; optional Hermes/NOOA runtime imports caused the four skips. They use mocked sandbox execution and do not qualify seven native CLI runs. Scoped hooks passed; the private guardword catalog was unavailable.

Docker provider fix at 6e87e4beb6963cf2bb75d3f78fc51e055849d2a3: 91 provider tests passed. A real Docker smoke with one exec slot occupied confirmed that a separate bounded upload could deliver the expiry marker, the native supervisor stopped the process, the exec client returned, and the owned container was removed. The corrected smoke passed with no cleanup errors; the first attempt's fixture incorrectly treated its started sandbox as borrowed and remains recorded as failed. This validates the Docker provider path, not seven-harness stopping, CAD scoring, or model quality. Scoped hooks passed; the private guardword catalog was unavailable.

Earlier configuration-only validation at e0e16688a, against an isolated checkout of bbaf99426839daf3f1a539e3bdc74d9e54b57fff plus its seven changed files:

python -m pytest -q \
  resources_servers/deepcad/tests/test_harness_configs.py \
  resources_servers/deepcad/tests/test_opencode_manifest.py \
  tests/unit_tests/models/test_model_alias_capture.py

46 passed, 2 skipped, Python 3.13.14, existing Gym development venv. The two skips are optional Hermes/NOOA runtime boundary checks; all seven routing/configuration cases executed. No model or native client was run.

The model-alias regression sends requests through two real model ASGI applications and retrieves both manifests through the original policy_model control endpoint. It checks model identity and call/response IDs. Worker acknowledgements are synthetic; this is not real tensor or optimizer validation.

Scoped pre-commit, Ruff, formatting and whitespace checks passed. The private guardword catalog was unavailable, so that content scan was skipped. The commit has DCO sign-off. These checks do not establish full CI or rollout readiness.

Compatibility and benchmark impact

  • Regenerate manifests to use CAD-specific agent names; configure the deployment policy_model before the CAD recipe, which creates deepcad_policy_model.
  • Configuration load order still matters: load stock/model settings, then CAD, then profiles such as HLE Pi. The reverse HLE-first/CAD-last order reimports stock Pi defaults and is not supported. SWE-Hermes blends were checked in both orders; no arbitrary-order guarantee is claimed.
  • The CAD instructions change standalone and lazy OpenCode rows as well as file-input rows.
  • Materialized rows move sampling fields into _ng_model_sampling. Their output limit is the smaller of the row and native request caps. Lazy rows retain their request fields too, and OpenCode can still reject a row cap below its own.
  • File-input rows contain one user message with workspace paths and no inline images. OpenCode therefore retains its build system prompt instead of replacing it with CAD system text.
  • Inline targets remain in the explicit resource-seed target_images field. Lazy file-input conversion requires materialization first.
  • Each independent budget experiment requires a fresh shared CAD_BUDGET_DIR. The current stop limitation above means the intended hard ten-generation stopping/grading protocol must not be treated as validated.
  • Geometry scoring and reference isolation remain unchanged. The explicit budget-stop grading branch changes eligibility relative to completed-only runs; ordinary failures and malformed stop markers retain their previous behavior. No quality or convergence parity is claimed.
  • The earlier internal CAD resumed optimizer update is separate from this public recipe and does not qualify it.

Code provenance

The admission counter, model dispatch hooks and budget tests are adapted from NVIDIA NeMo Gym, Apache-2.0, at d8c548915431e59169890d34cd8cea56ef2a905e. Original and affected paths:

  • nemo_gym/models/rollout_budget.py
  • nemo_gym/models/base_responses_api_model.py
  • tests/unit_tests/models/test_rollout_budget.py
  • tests/unit_tests/models/test_rollout_budget_integration.py

Each affected file retains the source revision/license. The port excludes final-call prompt/tool-choice steering and unrelated harness/capture/scorer changes.

Hermes image runtime dependencies are resolved from the MIT-licensed upstream pyproject.toml at the exact pinned revision into responses_api_agents/hermes_agent/image-runtime-constraints.txt; that file records the revision, license and generation command. The adapter calls upstream APIs; it does not vendor the image implementation.

rohitrango and others added 11 commits November 10, 2026 09:42
Signed-off-by: Rohit Jena <rohit.rango@gmail.com>
Signed-off-by: Rohit Jena <rohit.rango@gmail.com>
Co-authored-by: Yi-Fu Wu <yifu.wu@gmail.com>
Signed-off-by: Rohit Jena <rohitrango@users.noreply.github.com>
Signed-off-by: Rohit Jena <rohit.rango@gmail.com>
Signed-off-by: Rohit Jena <rohit.rango@gmail.com>
Co-authored-by: Yi-Fu Wu <yifu.wu@gmail.com>
Signed-off-by: Rohit Jena <rohitrango@users.noreply.github.com>
Co-authored-by: Yi-Fu Wu <yifu.wu@gmail.com>
Signed-off-by: Rohit Jena <rohitrango@users.noreply.github.com>
Signed-off-by: Rohit Jena <rohit.rango@gmail.com>
Signed-off-by: Rohit Jena <rohit.rango@gmail.com>
…ion budgets

Signed-off-by: Tong Che <tongc@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Nov 16, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added feature New capabilities, enhancements, or enablement work area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers labels Nov 16, 2026
@yfw
yfw deleted the branch NVIDIA-NeMo:rohit/cad-rl November 17, 2026 17:13
@yfw yfw closed this Nov 17, 2026
@yfw yfw reopened this Nov 17, 2026
@yfw
yfw changed the base branch from rohit/cad-rl to main November 17, 2026 17:36
@yfw
yfw self-requested a review November 17, 2026 17:37
Signed-off-by: Tong Che <tongc@nvidia.com>
@gerryche123
gerryche123 marked this pull request as ready for review November 18, 2026 18:16
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Nov 18, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Nov 18, 2026

@yfw yfw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD. Two blockers: the generation budget never stops a harness (nemo_gym/models/rollout_budget.py:83-94), and test_harness_configs.py:80 turns the CI Server suite red.

Running one DeepCAD environment through seven stock agents with _copy environment servers is the shape Gym wants for multi-harness work. Routing, configuration keys, /ng-sampling/ and rollout correlation are correct for all seven harnesses. The merge with main kept the prepare_row and lazy-row behavior from #5950. With the budget off, nothing changes for other environments. After the fixes in the comments below, this recipe can be the reference example for one environment run by many harnesses.

Process:

  • No merge conflict with main now, so no rebase is needed.
  • DCO passes.
  • CI ran only DCO and the lock-freshness job. A maintainer needs to comment /ok to test with the full 40-character head SHA to start the other jobs.

Local runs (CPU only):

  • pre-commit passes with no rewrites.
  • deepcad tests in deepcad's own venv: 327 passed, 2 failed (see test_harness_configs.py:80), 4 skipped.
  • The PR's core budget tests: 11 of 11 pass. All model tests: 2308 passed.

Notes:

Generated by Claude Code

Comment on lines +83 to +94
for agent in self.config.agent_servers:
try:
response = await server_client.post(
server_name=agent,
url_path=f"/rollouts/{rollout_id}/stop",
json={"reason": BUDGET_STOP_REASON},
)
await raise_for_status(response)
except ClientResponseError as error:
if error.status == 404:
continue
raise

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nemo_gym/models/rollout_budget.py:83-94

On call 11 this loop stops no harness: the first agent in agent_servers answers HTTP 501, and the loop returns that 501 to the harness.

OPENCODE.md says what the budget is for:

This enables grading the artifact when the shared ten-call budget stops the agent

(OPENCODE.md L26-L27)

This is what happens on call 11 instead:

  1. These lines post /rollouts/{id}/stop to each name in agent_servers, in order. The loop goes on only after a 404 and raises every other error.
  2. SimpleResponsesAPIAgent.stop_rollout answers 501 "This agent cannot stop a rollout" for every agent that does not override _rollout_sandbox_session, before it looks up the rollout (base_responses_api_agent.py L1389-L1390). Of the seven agents, only codex_agent overrides it (codex_agent/app.py L794).
  3. deepcad_opencode_agent is first in both lists (opencode.yaml L105, multiharness.yaml L121-L128). So every rollout that makes an 11th call gets HTTP 501 at the harness, Codex rollouts included.
  4. No agent is stopped, so no response carries the ng_rollout_stop marker, and the budget-stop grading branch never runs (opencode_app.py L201-L215).
  5. When a harness gives up on that 501, every adapter except Claude Code reports a failed model call. The default outcome policy records that as a retryable failure with no reward (agent_outcome.py L95, L113-L120). A retried attempt gets a new -a<N> rollout ID and a new counter, so the cap is ten calls for each attempt. Claude Code reports no outcome, so its failed run goes to /verify and scores an unmasked 0 (deepcad:agent_incomplete) (agent_outcome.py L136-L138).

Even Codex cannot stop on the recipe's Docker workspace. After launch, SandboxSession.expire raises ExpiryUnsupportedError unless the provider uploads files independently of running commands, as Local and OpenSandbox with background_exec do (sandbox_session.py L339-L343). stop_rollout turns that error into a 501 too (base_responses_api_agent.py L1405-L1406).

The PR introduced this. The budget tests use the _stoppable_agent test fixture (test_agent_sessions.py L481-L485), never a real harness. The PR's own test_stop_checks_actual_http_status_before_identity[501] expects this raise (test_rollout_budget_integration.py L93-L97).

Suggested fix: please move the generation budget out of this PR into its own core PR. It is a multi-file change, so it cannot be a suggestion. The lines to remove are in the fold below. Without those lines, this PR can merge before the core PR. The core PR can then land with these three parts:

  1. Stop support for every agent that it lists. feat(environment): stop the agent once a live world ends the episode #5838 (open) adds it for OpenCode only.
  2. A test that builds each listed agent and fails unless its stop answers 404, not 501, for an unknown rollout.
  3. The provider requirement above, in its docs.

Without the budget, the recipe has no hard ten-call bound. OpenCode keeps its advisory build.steps (opencode.yaml L67-L68). Hermes and Claude Code keep max_turns (hermes_agent.yaml L11, claude_code_agent.yaml L11). Pi, Codex and OpenClaw set no call limit in their configurations, and NOOA leaves max_policy_calls unset (nooa_agent/config.py L62-L64). The 1800 s agent timeout is the hard bound (opencode.yaml L51).

Two smaller fixes look tempting and do not work. Treating 501 like 404 in this loop only turns call 11 into the 503 "no agent owns this rollout" (L106) for the six other agents. Putting codex_agent first fixes only Codex, and Codex still fails on Docker.

Lines to remove from this PR



class BaseResponsesAPIModelConfig(BaseRunServerInstanceConfig):
rollout_generation_budget: RolloutGenerationBudgetConfig | None = None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nemo_gym/models/base_responses_api_model.py:286

rollout_generation_budget is a setting of the whole model server, so every environment on the shared policy_model also gets the ten-call cap.

The budget counts calls by rollout ID only (rollout_budget.py L71-L77), and rollout_correlation: true (opencode.yaml L4) gives every agent's calls a rollout ID. The budget has no environment filter. NeMo-RL recipes run many environments on one policy_model, for example RL prefetch_super_all_envs.yaml L66-L81. A NeMo-RL run that loads the CAD opencode.yaml next to example_multi_step.yaml caps the example_multi_step rollouts at ten calls too. Their call 11 fails, with a 501 today and with a 503 once the stop works. The PR introduced this.

Suggested fix, for the core PR that the comment on rollout_budget.py L83-L94 asks for: let the environment server carry the cap with each episode. It already carries sampling values that way, in the /ng-sampling/ segment (base_environment_server.py L359-L368). The model server then counts and stops only the rollouts that carry a cap. This also removes the agent_servers list and its ordering problem. It is a design change in several files, so it cannot be a suggestion.

A startup guard that refuses other agents on this model server is the worse fix. It blocks every run that puts CAD next to other environments on one policy_model, and NeMo-RL trains that way.

@@ -0,0 +1,128 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
# Seven concurrent native pairings; prepare each row with --file-input --harness NAME.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resources_servers/deepcad/configs/multiharness.yaml:3

None of the seven pairings in this recipe has a real-model rollout yet, and Gym asks for real smoke rollouts for this kind of change.

Gym's AGENTS.md says:

For behavior-changing environment or agent work: run representative real smoke rollouts with a model and inspect agent and verifier behavior.

(AGENTS.md L14-L15)

The PR body says that the rollouts are pending and that "This remains a draft", but the PR is out of draft.

Suggested fix:

  1. Please run the five example tasks (data/example.jsonl) through each pairing that you keep. For each harness, please report the number of episodes, the mean reward, and whether a target PNG and a preview PNG reached the policy model. Please keep a harness in the recipe only after its smoke run passes. The Harness-Ready Environments runbook covers running one environment through several harnesses.
  2. Please list these changes under "Compatibility and benchmark impact" in the PR body:
    • The INSTRUCTIONS text changed for every OpenCode row, standalone and lazy rows included (prepare_opencode.py L21-L23).
    • For prepared rows (not lazy rows), sampling values move to the _ng_model_sampling marker (prepare_opencode.py L117-L119). A row's output cap then applies as the smaller of the row's cap and the harness's own cap (OPENCODE.md L145-L147). Before, OpenCode refused a row cap below its own (opencode_agent/app.py L1557-L1567). Lazy rows keep the values in the request as well, so OpenCode still refuses a smaller cap there.
    • --file-input OpenCode rows have no system message. So system_prompt_mode: replace no longer replaces OpenCode's build prompt, and the CAD text becomes user text (opencode_agent/app.py L1899-L1904).

@github-actions github-actions Bot removed the sla:review-overdue Review response is over the one-business-day SLA label Nov 18, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the risk:high Autobot PR risk: high label Nov 18, 2026
Signed-off-by: Tong Che <tongc@nvidia.com>
@gerryche123
gerryche123 requested a review from a team as a code owner November 19, 2026 00:53
Tong Che added 7 commits November 18, 2026 17:16
Signed-off-by: Tong Che <tongc@nvidia.com>
Signed-off-by: Tong Che <tongc@nvidia.com>
Signed-off-by: Tong Che <tongc@nvidia.com>
Signed-off-by: Tong Che <tongc@nvidia.com>
Signed-off-by: Tong Che <tongc@nvidia.com>
Signed-off-by: Tong Che <tongc@nvidia.com>
Signed-off-by: Tong Che <tongc@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers community-request Issue reported or requested by someone from the community feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer risk:high Autobot PR risk: high

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants