Repository navigation
feat(deepagents): run agent sessions in process or inside a task sandbox - #5003
Conversation
|
/ok to test 674c828 |
|
CI note: the non-blocking |
|
/ok to test a1e2d88 |
yaoyu-33
left a comment
There was a problem hiding this comment.
SHIP WITH CARE. Thanks for closing these session gaps. The new seed, close and restart-only checkpoint paths check out against SimpleAgent and single_agent_turn:
/verifykeeps the Resources seed cookies.- Restart-only tracking covers the overridden
responses(). - The seed's 422 is final.
a1e2d88 fixes the test_global_config.py assertion that failed Unit tests / Test at 674c828.
The inline comments hold 1 RISK and 2 NOTEs:
- RISK: the documented
--agent-typepath breaks once #4953 lands. - NOTE:
skip_verificationno longer applies on the new relay. - NOTE: a test gap for optional grants.
PR hygiene
- Could the Validation section list the core
pytest tests/unit_tests/run at a1e2d88? That suite is where the stalelegacy_agentassertion showed up. - The KiloCode conformance failure matches scheduled main run 39984595370, so I agree it needs no change here.
Follow-up outside this PR
- In the Tavily pairing, the agent's own
TavilySearchtool (reasoning_search_agent.py:50) never applies the resources server'sexclude_domains_file_pathpolicy. - Yet
tavily_search/README.mdsays that policy applies to every Tavily configuration. - This dates from #2929 and is worth its own issue.
Not covered by this review: a live Tavily rollout and GPU tests. Reviewed by a nine-reviewer team.
|
/ok to test e73a26b |
- Set the TB 2.1 full evaluation to 8 repeats per task, as the benchmark's OpenCode, Terminus 2, and Inkling configs declare; the README's run config declares none. - Note that exporting NEMO_GYM_ALLOW_UNSUPPORTED_PAIRING=1 also works with gym eval run --no-serve. - Link the deepagents session caveat to #5003, which fixes it. - Say in Step 4 that the probes are optional at the Evaluation level. - Replace "very few unhealthy rollouts" with a stated cause for each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…t_turn names none A legacy_agent relay names no resources server: the agent's /run uses its own. A single_agent_turn_legacy server that replaces the relay had to pin one, and a pinned name breaks a benchmark that reaches the server through an _inherit_from rename of the agent. The rename also renames the resources server, and Gym follows renames only for agent references, so the pinned name dangles (for example indic/math_500 through math_with_judge's server). resources_server is now optional. Unset, the server binds the resources server the bound agent names, as the relay did, so it follows the agent rename. A server that sets none in front of an agent that names none fails at startup instead of on the first episode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
About 307 configs reach simple_agent through a legacy_agent relay, so a crash repeats the whole episode. Under single_agent_turn_legacy, it repeats only the current step, and simple_agent resumes inside its turn (GYMINT-549). add_legacy_agent_environment_servers.py gains --switch-simple-agent. It replaces each relay in front of a simple_agent with a single_agent_turn_legacy server of the same name, so rows keep their routing. The server names no resources server, so it binds the agent's, as the relay did, and follows an agent rename. It grants direct HTTP access to the resources server's tools, the routes the agent's /run called. It resolves each config with its own config_paths, as Gym does, since instance names are unique only within a run. A relay stays, listed with its reason, when its agent binds no resources server, declares a grant simple_agent sessions reject (a required MCP access or a second direct HTTP access), binds a resources server whose seed_session takes only the legacy body (19 relays today, read from the server's source so a server passes once it migrates), or its run cannot load an include. A relay that inherits another keeps inheriting it, and the script refuses to switch only one of the two. The default mode is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
The pairing guard only compared `_inherit_from` children that carry their own `legacy_agent` key, so a run that merged a switched relay with a same-name `legacy_agent` override (the hle and apex_shortlist opencode/pi recipes) or a bare `_inherit_from` child exited 0 with two environment server types. - Resolve every run after the planned switches. A relay that a run merges with a legacy_agent server stays on the relay; when the relay it inherits switches, the script refuses (exit 2) and names the config. Top-level nulls now remove instances, as in Gym. - Keep relays whose resources server reads the row's rollout coordinates (genrm_compare), whose implementation is not under resources_servers/, or that set keys the switched stanza would drop. - Build every rewrite before writing any file; a stanza that cannot be found exits 2 instead of raising StopIteration. - Dedupe scanned paths after resolving them. - Give the hle and apex_shortlist opencode/pi recipes a legacy_agent server of their own, so their benchmarks can switch. - Fix the docstring and docs examples, document the new stay reasons, note how to inherit the single_agent_turn templates, and test the legacy adapter with resources_server unset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test 7579a80 |
1 similar comment
|
/ok to test 7579a80 |
main split tests/unit_tests/tooling/test_add_legacy_agent_environment_servers.py into per-topic files (#5946). Take main's version of that file and move this branch's test_deepagents_sandbox_template_composes_with_one_environment into test_agent_environment_default_configs.py, beside the OpenCode and OpenClaw session-template tests it sat between. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test eb161da |
Resolve fern/versions/latest/pages/agent-server/integrate-existing-agents.mdx: main and this branch each added a row to the sandboxed-harness table. Keep main's Copilot row and this branch's DeepAgents row, each as written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test cc4625d |
Main added mini-SWE beside DeepAgents in AGENT_SESSION_AGENT_TYPES and in the two harness tables; keep both rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yuya@nvidia.com>
|
/ok to test 2a4a00f |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test f38e4f5 |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test 201a545 |
Take main's release.yaml: #6066 starts the changelog at the previous release tag for main and pull-request pushes, which replaces this branch's PR-only workaround. Keep Droid beside DeepAgents in AGENT_SESSION_AGENT_TYPES and the harness tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yuya@nvidia.com>
|
/ok to test e637ab1 |
Main added OpenHands beside DeepAgents in the harness tools table; keep both rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yuya@nvidia.com>
|
/ok to test a7b6eb4 |
#5775 left the benchmarks/healthbench_professional/prepare.py entry unterminated, so detect-secrets cannot read the baseline and the secrets-detector check fails on every PR. Restore the missing is_secret field and closing brackets. Signed-off-by: Chris Wing <cwing@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b2a2791) Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test e180806 |
Main added Cline beside DeepAgents in the harness tools table; keep both rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yuya@nvidia.com>
|
/ok to test 7e9a588 |
Main added kimi_code_agent beside langchain_deepagents_agent in AGENT_SESSION_AGENT_TYPES; keep both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yuya@nvidia.com>
|
/ok to test ba9db04 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43337740900 |
DeepAgents now honors Gym's agent-session lifecycle through one
app.pyentrypoint.tool_profile: builtinruns its file, shell and subagent tools inside a task sandbox;tavily_searchretains the in-process search graph. No separate issue is needed for this focused session integration and harness-layout correction.Compatibility and limits
Existing config filenames, agent names and launch commands remain valid.
reasoning_search_agent.pyremains a compatibility subclass; shipped configs useapp.py. Search retains legacy/runand custombuild_agent(model)overrides; builtin execution requires an Environment Server.Both profiles and legacy search
/runnow reject non-null requestmax_output_tokenswith HTTP 422 before graph work or resource seeding. Remove the field and configure the Model Server's per-call cap. Other supported sampling controls remain available, and corrected requests can reuse a seeded session.This remains Evaluation-level support: invocation/tool/turn ownership gaps are explicit, and checkpoints restart unfinished rollouts. In-process search does not enforce the task timeout or support rollout stop. Terminal-Bench 2.1 remains an unsupported pairing. Historical full-evaluation results in the existing results comment are not scores for this implementation.
Validation
python -m pytest responses_api_agents/langchain_deepagents_agent/tests -q -o addopts=— 103 passed.pre-commit run --files <changed paths>— passed on the Linux workstation; no full-repository pre-commit claim.nvidia/nemotron-3.5-super-vl-120b-a12b, agent-owned Dockerdebian:bookworm-slim, temperature 0, top_p 1, Model Server max_tokens 4096, thinking disabled, 180-second activation budget. Responsecompleted, reward 0.0, unmasked, no failure outcome. Verifier output was inspected; all task process groups and containers were removed. This one-row smoke checks runtime and verification behavior, not benchmark qualification. Existing representative evidence covers search and builtin profiles; no live Tavily API qualification is claimed.requestis skipped. Full behavioral CI is not established by that status. Checks.Targeted regression controls: 3 tests fail against the pre-fix production sources and pass in the full fixed-source suite above.
Review follow-up (2026-11-09)
Fixed retirement cleanup and retry, partial-start ownership, pre-launch rollout stop, correctable tool admission, graph-agent composition, builtin model networking defaults, provider metadata and unknown usage after an interrupted successful response body. No separate issue is needed for these review corrections.
Validated on the Linux workstation at
b0ae73be2: component tests plustests/unit_tests/config/test_global_config.py— 350 passed;pre-commit run --all-filespassed. One unchanged Reasoning Gym example with NIM Nemotron 3.5 Super, agent-owned Docker Debian bookworm-slim, temperature 0/top_p 1/per-call cap 4096: completed, reward 0.0, unmasked, no failure kind. Inspected agent and verifier output; task container removed. This is runtime evidence, not benchmark qualification.Review follow-up (2026-11-10)
Merged main (
6c21277b4; one docs-table conflict inintegrate-existing-agents.mdx, where the DeepAgents row now says "Not tested" under Harbor task folders) and fixed the findings of a review against the agent review checklist. No separate issue is needed for these review corrections.app.pydeclarescalls_model_from_sandbox: its default builtin profile calls the Model Server from the sandbox, and Harbor reads that literal from source.context_length_exceededendsincompleteand is verified; any other refusal endsfailedwithmodel_request_errorand its status. Retryable statuses, transport errors and legacy/runstill raise.instructions,temperatureandtop_pnow reach every model call; before, they were silently dropped. Score impact: search-profile runs whose rows or overrides set these fields can score differently. The bundled example rows set none of them.wall_timestop counts only when the supervisor's receipt shows the deadline. A SIGTERM from inside the sandbox, such as a task'spkill python3, ends the runfailedwithagent_run_error.failedwithmodel_transport_error, which is retried, instead of grading an empty answer. A rollout stop stays a graded stop.cancelledonly for a run that never launched. Runner error text is capped.Validated at
b537de56a:python -m pytest responses_api_agents/langchain_deepagents_agent/tests -q -o addopts=on the Linux workstation: 133 passed.tests/unit_tests/config/test_global_config.py,tests/unit_tests/tooling/test_add_legacy_agent_environment_servers.pyandtests/unit_tests/test_session_contract_repo.pypass on the Linux workstation.tests/unit_tests/test_public_api_facades.pypasses on a git checkout (863 passed).pre-commit run --files <every changed path>passed.nvidia/nemotron-3.5-super-vl-120b-a12b, thinking off, Model Server per-call cap 4096:single_agent_turn_legacy, with requestinstructions,temperature: 0.6andtop_p: 0.95added. Both completed with reward 1.0, and all 4 captured upstream calls carried those sampling values and the instructions.debian:bookworm-slim, temperature 0, top_p 1. It completed with reward 0.0, unmasked: the answer is correct but has no<answer>tags.agent_timeout_sec: 8. The deadline hit before the first model reply, and the run endedfailedwithmodel_transport_errorfrom the runner's report, so it is retryable and not scored.CI repair (2026-11-11)
Final head
201a5454809803ed6561ddeb60b9ad1e537d08d6scopes release-rehearsal changelogs on PR mirror pushes to changes against the default branch. The previous release validation failed before Bash could start because repository-wide notes exceeded the environment-variable size limit. Release branches and explicit manual changelog overrides retain their existing behavior. The shared template's general large-note fix is tracked in NVIDIA-NeMo/FW-CI-templates#619.The release repair passed release validation at f38e4f56e, including both previously failing jobs. Then merge main
122724b0b965b8029b37db2d53779e2a19f6e82dto clear the lock-freshness advisory and bring the shared OpenAI pin and component lock to 2.45.0. This merge is clean and preserves main's compatibility-entrypoint initialization. No separate issue is needed for these focused CI and dependency-integration repairs.Validation on the Linux workstation against the final tree:
python -m pytest responses_api_agents/langchain_deepagents_agent/tests tests/unit_tests/config/test_global_config.py tests/unit_tests/tooling/test_agent_environment_default_configs.py tests/unit_tests/test_session_contract_repo.py --import-mode=importlib -q -o addopts= -o log_cli=false— 395 passed.python scripts/component_locks.py --jobs 4, using uv 0.13.4 — 283 projects, zero failures.pre-commit run --all-filespassed using Python 3.13 hook environments, with no file changes.cd fern && npm run checkpassed.@actions/expressionsevaluator (PR mirror, main, release branch, deployment branch, and manual dispatch/override cases).Both the source and PR mirror point to the final SHA. CI restarted for it: release validation and behavioral CI are pending.
CI repair — November 12, 2026
Source
e180806f1f103ef53b535a1a1a039951d36bccc0cherry-picks the three-line baseline syntax repair from #6083. The inherited HealthBench entry was missing its closing object and array, so the secrets detector stopped before scanning. No detector settings, baseline entries, or adapter runtime code were removed or changed.Validated on Linux workstation
wsagainst the exact committed tree:detect-secrets-hook --baseline .github/workflows/config/.secrets.baseline <all PR-changed files>passed with detect-secrets 1.5.0;pre-commit run --all-filespassed using Python 3.13 hook environments; hooks left tracked files unchanged. Model rollout: N/A for this baseline-only repair. No separate issue is needed; #6083 is the upstream fix. Fresh CI requested for this exact source.