Repository navigation
feat(crewai): add the CrewAI harness adapter with agent sessions - #5540
Merged
Merged
Conversation
Run one CrewAI crew turn per agent session in the Agent Server (GYMINT-563). Seed borrows the Resources Server's task sandbox and lists the episode's MCP grants; the activation runs the crew behind the session cookie on the rollout-prefixed /v1/responses route; close cancels the crew, kills commands still running in the sandbox and returns an AgentObservationBundle. - CrewAI 1.15.34, pinned. Its model is a CrewAI BaseLLM whose every call is a Chat Completions request through the Gym Model Server's rollout-prefixed route, over Gym's aiohttp client, with x-session-id set to the invocation. - Tools: a shell that runs each command in its own process group in the borrowed sandbox, and the tools of MCP grants, reached with the MCP SDK over an httpx transport backed by Gym's aiohttp client. - CrewAI telemetry, trace upload and version checks are off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…ools Add the adapter README (version, capability level, session lifecycle, tools, request fields, evidence, pairings and limits) and a row for CrewAI in the Tools table of the add-or-update-a-harness page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…l timeouts
Found in the first EOS smoke: OpenSandbox returns a command's stderr within
its output, so bash's job-control notice for the shell runner's killed
watchdog ("Killed (sleep ...)") reached the model after every command. The
runner now sends its own stderr to /dev/null once the command and its
watchdog have started.
A model request that times out raises TimeoutError through CrewAI; it was
taken for the crew's own deadline and graded as incomplete. Only the
deadline's expiry now ends a run as incomplete.
The process-group kill tests run on Linux only: macOS's bash 3.2 sometimes
leaves a subshell of a killed group running, which Linux does not.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…wai-sessions-0db917
…d register conformance probes - When the Environment Server lends no sandbox and sandbox_provider is set, the adapter starts its own sandbox with start_with_setup (provider default metadata, sandbox_config.metadata, then session labels; ttl_s defaults to timeout plus ten minutes) and stops it at close. Close also records the agent's SandboxObservation, from lifecycle_observation, for lent and own sandboxes. - Terminal-Bench 2.1 lists crewai_agent in allowed_agents, and its README harness table has a CrewAI row. - The harness conformance probes register crewai: they give the agent its own local sandbox, which the adapter now supports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test 0c3c7ae |
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-5540-c8716869b276.docs.buildwithfern.com/nemo/gym |
… outcome Close no longer returns a trajectory of its own. Rollout collection then builds the turns from the captured calls and joins every call that carried the session header to the invocation, including failed attempts that have no response ID. The agent's sandbox record now carries how the crew's run ended instead of unknown. The harness conformance probes failed both checks. The shell tests get a 10-second default deadline, and only the deadline tests shorten it, so a loaded machine does not time out a plain command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…w meet Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test 63b6727 |
main (#4837) moved every server to a committed pyproject.toml and uv.lock and installs harnesses with `uv sync --locked`. The CrewAI adapter follows: pyproject.toml pins crewai==1.15.34 and mcp<2 (the SDK's 1.x client API, as Hermes pins it) with the shared [tool.uv] settings, uv.lock is generated with uv 0.13.4, and requirements.txt is gone. The lock resolves the versions the adapter was tested with (crewai 1.15.34, mcp 1.30.0, httpx 0.28.1). Conflict in scripts/harness_conformance/ci.py: keep main's locked sync and add crewai to the harnesses that need no further install. The conformance runner README now describes the locked install and the current tuples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test 7657c9f |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test e985036 |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test 7b1f06a |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test 9bd3410 |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test 2111706 |
weijiac0619
previously approved these changes
Nov 12, 2026
Contributor
|
checked @AdamRajfer 's two blockers and the bot’s risks against the current head. LGTM |
github-merge-queue
Bot
removed this pull request from the merge queue due to a conflict with the base branch
Nov 12, 2026
Preserve CrewAI together with the DeepAgents, Cline, and Kimi Code documentation and retain configuration coverage for both CrewAI and Cline. Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test c871686 |
weijiac0619
approved these changes
Nov 13, 2026
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43346914024 |
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43347400937 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a sandboxed CrewAI 1.15.34 adapter with Gym agent sessions. The Agent Server prepares the runtime; one crew turn runs inside the task sandbox under Gym's supervisor, using the Gym Model Server, a task-local shell and granted MCP tools. No separate issue is needed for this focused adapter integration.
harness/agent_run_errorevent, the shell/MCP operation and bounded cause facts through HTTP 503, activation retry and close. Observable HTTP status or transport causes are preserved without serializing URLs, headers or bodies. Actual sandbox-provider failures keep their shared-layer classifications. Generic crew exceptions remain failed/unknown; ordinary tool errors reach the model as tool results.Retry-Afterseconds/dates are honored, with each wait capped at 60 seconds. The five default waits total at most 31 seconds withoutRetry-After. Deadlines and stop interrupt backoff; deterministic 4xx responses are not retried and exhaustion retains the final status. CrewAI may additionally retry rate-limited calls itself.Compatibility and limits
New session-only adapter; no host
/run. Non-null requestmax_output_tokensis refused with HTTP 422 before binding activation: CrewAI cannot enforce an aggregate response budget. Remove the request field and configure a per-call Model Server cap; temperature, top_p and length-stop handling remain supported, and a corrected request can reuse its session.The adapter is registered in harness conformance, with existing eight-scenario P0 evidence supporting its Observability claim. Training is unqualified: task commands share the runner's user and can modify its report. Terminal-Bench 2.1 remains an unsupported development pairing requiring
--allow-unsupported-pairing; premature supported-agent declarations are removed. No full benchmark evaluation is claimed.Validation
python -m pytest responses_api_agents/crewai_agent/tests -q -o addopts=— 90 passed. Regressions include actual shell/MCP faults through HTTP response and close, corrected budget requests, 429 recovery/exhaustion, and cancellation during backoff.pre-commit run --files <changed paths>— passed on the Linux workstation; no full-repository pre-commit claim.nvidia/nemotron-3.5-super-vl-120b-a12b, agent-owned Dockerdebian:bookworm-slim, temperature 0, top_p 1, Model Server max_tokens 4096, thinking disabled, 180-second activation budget. Responsecompleted, reward 1.0, unmasked, no failure outcome. Verifier output was inspected; all task process groups and containers were removed. This one-row smoke checks runtime and verification behavior, not benchmark qualification. Previous borrowed-sandbox and MCP-grant smokes exercise both tool paths; they do not establish a full benchmark result.requestis skipped. Full behavioral CI is not established by that status. Checks.Targeted regression controls: 5 tests fail against the pre-fix production sources and pass in the full fixed-source suite above.
Follow-up merge at
3ac7e34e0a4ba03c1b500e72262396371d3dbb90preserves the harness migration coverage in main’s newly split test files. All existing test bodies and parameters were compared before/after the move. Workstation validation of the exact merged tree: 180 migration and scheduling tests passed; fullpre-commit run --all-filespassed using Python 3.13 hook environments. Model rollout: N/A for this test-layout conflict resolution.Shared CI test repairs — November 10
At
83b7f4e37551532c0829a3e512ebfb4462c1917b, wait for the detached Pi test child to publish its own PID after leaving the harness process group. This prevents natural shutdown from killing a child that has not detached yet, while retaining the process-survival assertion for natural exit, timeout, and cancellation. Also include the shared synthetic resume-test isolation used by Copilot, Droid, and OpenHands, with explicit runtime-drift rejection coverage.Validation on the Linux workstation: a controlled startup delay reproduced the missing-child failure before the Pi fix; all 3 delayed shutdown modes passed afterward; all 5 Pi runner tests and 13 HTTP resume/recovery and runtime-fingerprint tests passed; full
pre-commit run --all-filespassed on the final changes using Python 3.13 hook environments. Model rollout: N/A, test-only changes. No separate issue is needed for these CI repairs.Conflict and CI repair (2026-11-11)
Final head
9bd3410f737454b84ce05f2fc7d7170c22086c08incorporates main122724b0b965b8029b37db2d53779e2a19f6e82d. Resolve all six conflicts while preserving CrewAI and the newly added agent registrations/documentation. Keep the Pi test's detached-child readiness handshake together with main's explicit deadline trigger and teardown. CrewAI's adapter runtime files are unchanged by this merge.Regenerate the CrewAI component lock to match main's OpenAI 2.45.0 pin. All 284 managed projects pass
python scripts/component_locks.py --jobs 4with uv 0.13.4, and full pre-commit passes again on the final lock update. The selected runtime tests already used OpenAI 2.45.0.Also scope PR release-rehearsal changelogs to changes against the default branch. This avoids the shared release template's environment-variable size failure while preserving normal release branches and manual overrides (NVIDIA-NeMo/FW-CI-templates#619). No separate issue is needed for these focused integration and CI repairs.
Validation on the Linux workstation:
responses_api_agents/crewai_agent/tests,responses_api_agents/pi_agent/tests/test_sandbox_runner.py,tests/unit_tests/config/test_global_config.py,tests/unit_tests/tooling/test_agent_environment_default_configs.py,tests/unit_tests/tooling/test_agent_environment_session_templates.py,tests/unit_tests/test_session_contract_repo.py,scripts/harness_conformance/tests/test_ci_install.py, andtests/unit_tests/test_fern_docs_links.py. Command:python -m pytest <listed paths> --import-mode=importlib -q -o addopts= -o log_cli=false. The initial run found stale installed Gym metadata requiring OpenAI 2.44.0; after installing this checkout with its pinned 2.45.0 requirement, the two affected configuration suites passed all 259 tests. All 56 initial failures were that metadata mismatch; the other 173 selected tests had passed.pre-commit run --all-filespassed with Python 3.13 hook environments; hooks left the source unchanged.cd fern && npm run checkpassed with zero errors and one warning.@actions/expressionsevaluator.GitHub reports the branch mergeable. CI restarted for this exact source: behavioral CI and release validation are pending.
Conflict repair — November 12, 2026
Source
2111706dc05c0dc590847bc0e33112ca0173b616merges mainb2a27913e248fe4c7e9a975e7df6b2fbe18aa3fb. Preserve both CrewAI and Droid in the session-agent registry, retain CrewAI alongside the merged OpenHands and Droid documentation, and adopt main's release workflow and secrets-baseline fix from #6083. The CrewAI adapter runtime and component tests are unchanged from the preceding PR head. No separate issue is needed for this focused conflict resolution.Exact-tree validation on Linux workstation
ws:python -m pytest responses_api_agents/crewai_agent/tests -q -o addopts= -o log_cli=false: 94 passed.python -m pytest tests/unit_tests/config/test_global_config.py tests/unit_tests/tooling/test_agent_environment_default_configs.py tests/unit_tests/tooling/test_agent_environment_session_templates.py tests/unit_tests/test_fern_docs_links.py -q -o addopts= -o log_cli=false: 323 passed, 24 subtests passed.pre-commit run --all-files: passed with Python 3.13 hook environments; no tracked changes.detect-secrets-hook --baseline .github/workflows/config/.secrets.baseline <all PR-changed files>: passed with detect-secrets 1.5.0.uv lock --check --project responses_api_agents/crewai_agent: passed with uv 0.13.4.cd fern && npm run check: zero errors and one warning. The matching Fern configuration on the Cline tree reports that the missing-redirect check is skipped without a Fern credential. No redirects changed.Model rollout: N/A for this registry/documentation/release-workflow conflict resolution and inherited baseline fix; adapter runtime behavior is unchanged. Fresh CI requested for the exact new source.