Skip to content

feat(crewai): add the CrewAI harness adapter with agent sessions - #5540

Merged
yaoyu-33 merged 29 commits into
mainfrom
claude/gymint-563-crewai-sessions-0db917
Nov 13, 2026
Merged

yaoyu-33 merged 29 commits into
mainfrom
claude/gymint-563-crewai-sessions-0db917

Conversation

@yaoyu-33

@yaoyu-33 yaoyu-33 commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Adds a sandboxed CrewAI 1.15.34 adapter with Gym agent sessions. The Agent Server prepares the runtime; one crew turn runs inside the task sandbox under Gym's supervisor, using the Gym Model Server, a task-local shell and granted MCP tools. No separate issue is needed for this focused adapter integration.

  • Seed installs a host-built, version-matched private Python runtime in a borrowed or agent-owned sandbox. Invalid configuration is a final refusal; operational failures retain their original causes through cleanup. Owned sandboxes receive TTL/metadata and optional network policy.
  • Activation preserves rollout routing, cookies and model-attempt ownership. CrewAI's synchronous worker calls return to the runner's event loop so deadlines, close and rollout stop can cancel requests, tools and retry waits. Close returns invocation/model references, conversation, timed tools, sandbox outcome, honest usage gaps and cleanup confirmation.
  • Runner infrastructure failures retain a stable harness/agent_run_error event, the shell/MCP operation and bounded cause facts through HTTP 503, activation retry and close. Observable HTTP status or transport causes are preserved without serializing URLs, headers or bodies. Actual sandbox-provider failures keep their shared-layer classifications. Generic crew exceptions remain failed/unknown; ordinary tool errors reach the model as tool results.
  • Transient model errors receive five adapter retries by default. Full-jitter exponential backoff uses a one-second base; positive Retry-After seconds/dates are honored, with each wait capped at 60 seconds. The five default waits total at most 31 seconds without Retry-After. Deadlines and stop interrupt backoff; deterministic 4xx responses are not retried and exhaustion retains the final status. CrewAI may additionally retry rate-limited calls itself.

Compatibility and limits

New session-only adapter; no host /run. Non-null request max_output_tokens is refused with HTTP 422 before binding activation: CrewAI cannot enforce an aggregate response budget. Remove the request field and configure a per-call Model Server cap; temperature, top_p and length-stop handling remain supported, and a corrected request can reuse its session.

The adapter is registered in harness conformance, with existing eight-scenario P0 evidence supporting its Observability claim. Training is unqualified: task commands share the runner's user and can modify its report. Terminal-Bench 2.1 remains an unsupported development pairing requiring --allow-unsupported-pairing; premature supported-agent declarations are removed. No full benchmark evaluation is claimed.

Validation

  • Final source: 4b1cc4aa8; all 8 changed-file hashes match the workstation sources, and the commit has DCO sign-off.
  • Linux component suite: python -m pytest responses_api_agents/crewai_agent/tests -q -o addopts= — 90 passed. Regressions include actual shell/MCP faults through HTTP response and close, corrected budget requests, 429 recovery/exhaustion, and cancellation during backoff.
  • pre-commit run --files <changed paths> — passed on the Linux workstation; no full-repository pre-commit claim.
  • Real-model rollout: one unchanged Reasoning Gym example, NIM nvidia/nemotron-3.5-super-vl-120b-a12b, agent-owned Docker debian:bookworm-slim, temperature 0, top_p 1, Model Server max_tokens 4096, thinking disabled, 180-second activation budget. Response completed, reward 1.0, unmasked, no failure outcome. Verifier output was inspected; all task process groups and containers were removed. This one-row smoke checks runtime and verification behavior, not benchmark qualification. Previous borrowed-sandbox and MCP-grant smokes exercise both tool paths; they do not establish a full benchmark result.
  • Latest-head GitHub checks: six reported checks passed (DCO, lint, both lock checks, copyright, secrets); request is skipped. Full behavioral CI is not established by that status. Checks.

Targeted regression controls: 5 tests fail against the pre-fix production sources and pass in the full fixed-source suite above.

Follow-up merge at 3ac7e34e0a4ba03c1b500e72262396371d3dbb90 preserves the harness migration coverage in main’s newly split test files. All existing test bodies and parameters were compared before/after the move. Workstation validation of the exact merged tree: 180 migration and scheduling tests passed; full pre-commit run --all-files passed using Python 3.13 hook environments. Model rollout: N/A for this test-layout conflict resolution.

Shared CI test repairs — November 10

At 83b7f4e37551532c0829a3e512ebfb4462c1917b, wait for the detached Pi test child to publish its own PID after leaving the harness process group. This prevents natural shutdown from killing a child that has not detached yet, while retaining the process-survival assertion for natural exit, timeout, and cancellation. Also include the shared synthetic resume-test isolation used by Copilot, Droid, and OpenHands, with explicit runtime-drift rejection coverage.

Validation on the Linux workstation: a controlled startup delay reproduced the missing-child failure before the Pi fix; all 3 delayed shutdown modes passed afterward; all 5 Pi runner tests and 13 HTTP resume/recovery and runtime-fingerprint tests passed; full pre-commit run --all-files passed on the final changes using Python 3.13 hook environments. Model rollout: N/A, test-only changes. No separate issue is needed for these CI repairs.

Conflict and CI repair (2026-11-11)

Final head 9bd3410f737454b84ce05f2fc7d7170c22086c08 incorporates main 122724b0b965b8029b37db2d53779e2a19f6e82d. Resolve all six conflicts while preserving CrewAI and the newly added agent registrations/documentation. Keep the Pi test's detached-child readiness handshake together with main's explicit deadline trigger and teardown. CrewAI's adapter runtime files are unchanged by this merge.

Regenerate the CrewAI component lock to match main's OpenAI 2.45.0 pin. All 284 managed projects pass python scripts/component_locks.py --jobs 4 with uv 0.13.4, and full pre-commit passes again on the final lock update. The selected runtime tests already used OpenAI 2.45.0.

Also scope PR release-rehearsal changelogs to changes against the default branch. This avoids the shared release template's environment-variable size failure while preserving normal release branches and manual overrides (NVIDIA-NeMo/FW-CI-templates#619). No separate issue is needed for these focused integration and CI repairs.

Validation on the Linux workstation:

  • 432 distinct tests and 24 subtests passed across the selected suites: responses_api_agents/crewai_agent/tests, responses_api_agents/pi_agent/tests/test_sandbox_runner.py, tests/unit_tests/config/test_global_config.py, tests/unit_tests/tooling/test_agent_environment_default_configs.py, tests/unit_tests/tooling/test_agent_environment_session_templates.py, tests/unit_tests/test_session_contract_repo.py, scripts/harness_conformance/tests/test_ci_install.py, and tests/unit_tests/test_fern_docs_links.py. Command: python -m pytest <listed paths> --import-mode=importlib -q -o addopts= -o log_cli=false. The initial run found stale installed Gym metadata requiring OpenAI 2.44.0; after installing this checkout with its pinned 2.45.0 requirement, the two affected configuration suites passed all 259 tests. All 56 initial failures were that metadata mismatch; the other 173 selected tests had passed.
  • Full pre-commit run --all-files passed with Python 3.13 hook environments; hooks left the source unchanged.
  • cd fern && npm run check passed with zero errors and one warning.
  • The workflow's actual changelog-selection expression passed seven routing/override cases using GitHub's @actions/expressions evaluator.
  • Model rollout: N/A for this conflict/test/CI repair; the CrewAI adapter runtime is unchanged. Earlier real-model evidence remains pinned to its recorded source.

GitHub reports the branch mergeable. CI restarted for this exact source: behavioral CI and release validation are pending.

Conflict repair — November 12, 2026

Source 2111706dc05c0dc590847bc0e33112ca0173b616 merges main b2a27913e248fe4c7e9a975e7df6b2fbe18aa3fb. Preserve both CrewAI and Droid in the session-agent registry, retain CrewAI alongside the merged OpenHands and Droid documentation, and adopt main's release workflow and secrets-baseline fix from #6083. The CrewAI adapter runtime and component tests are unchanged from the preceding PR head. No separate issue is needed for this focused conflict resolution.

Exact-tree validation on Linux workstation ws:

  • python -m pytest responses_api_agents/crewai_agent/tests -q -o addopts= -o log_cli=false: 94 passed.
  • python -m pytest tests/unit_tests/config/test_global_config.py tests/unit_tests/tooling/test_agent_environment_default_configs.py tests/unit_tests/tooling/test_agent_environment_session_templates.py tests/unit_tests/test_fern_docs_links.py -q -o addopts= -o log_cli=false: 323 passed, 24 subtests passed.
  • pre-commit run --all-files: passed with Python 3.13 hook environments; no tracked changes.
  • detect-secrets-hook --baseline .github/workflows/config/.secrets.baseline <all PR-changed files>: passed with detect-secrets 1.5.0.
  • uv lock --check --project responses_api_agents/crewai_agent: passed with uv 0.13.4.
  • cd fern && npm run check: zero errors and one warning. The matching Fern configuration on the Cline tree reports that the missing-redirect check is skipped without a Fern credential. No redirects changed.

Model rollout: N/A for this registry/documentation/release-workflow conflict resolution and inherited baseline fix; adapter runtime behavior is unchanged. Fresh CI requested for the exact new source.

yaoyu-33 and others added 5 commits November 3, 2026 10:56
Run one CrewAI crew turn per agent session in the Agent Server (GYMINT-563).
Seed borrows the Resources Server's task sandbox and lists the episode's MCP
grants; the activation runs the crew behind the session cookie on the
rollout-prefixed /v1/responses route; close cancels the crew, kills commands
still running in the sandbox and returns an AgentObservationBundle.

- CrewAI 1.15.34, pinned. Its model is a CrewAI BaseLLM whose every call is
  a Chat Completions request through the Gym Model Server's rollout-prefixed
  route, over Gym's aiohttp client, with x-session-id set to the invocation.
- Tools: a shell that runs each command in its own process group in the
  borrowed sandbox, and the tools of MCP grants, reached with the MCP SDK
  over an httpx transport backed by Gym's aiohttp client.
- CrewAI telemetry, trace upload and version checks are off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…ools

Add the adapter README (version, capability level, session lifecycle,
tools, request fields, evidence, pairings and limits) and a row for CrewAI
in the Tools table of the add-or-update-a-harness page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…l timeouts

Found in the first EOS smoke: OpenSandbox returns a command's stderr within
its output, so bash's job-control notice for the shell runner's killed
watchdog ("Killed (sleep ...)") reached the model after every command. The
runner now sends its own stderr to /dev/null once the command and its
watchdog have started.

A model request that times out raises TimeoutError through CrewAI; it was
taken for the crew's own deadline and graded as incomplete. Only the
deadline's expiry now ends a run as incomplete.

The process-group kill tests run on Linux only: macOS's bash 3.2 sometimes
leaves a subshell of a killed group running, which Linux does not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…d register conformance probes

- When the Environment Server lends no sandbox and sandbox_provider is set,
  the adapter starts its own sandbox with start_with_setup (provider default
  metadata, sandbox_config.metadata, then session labels; ttl_s defaults to
  timeout plus ten minutes) and stops it at close. Close also records the
  agent's SandboxObservation, from lifecycle_observation, for lent and own
  sandboxes.
- Terminal-Bench 2.1 lists crewai_agent in allowed_agents, and its README
  harness table has a CrewAI row.
- The harness conformance probes register crewai: they give the agent its
  own local sandbox, which the adapter now supports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33 yaoyu-33 added area:agent Agent harnesses and Responses API agent behavior feature New capabilities, enhancements, or enablement work labels Nov 3, 2026
@yaoyu-33

yaoyu-33 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 0c3c7ae

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

yaoyu-33 and others added 2 commits November 4, 2026 09:43
… outcome

Close no longer returns a trajectory of its own. Rollout collection then
builds the turns from the captured calls and joins every call that carried
the session header to the invocation, including failed attempts that have
no response ID. The agent's sandbox record now carries how the crew's run
ended instead of unknown. The harness conformance probes failed both checks.

The shell tests get a 10-second default deadline, and only the deadline
tests shorten it, so a loaded machine does not time out a plain command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…w meet

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 63b6727

main (#4837) moved every server to a committed pyproject.toml and uv.lock
and installs harnesses with `uv sync --locked`. The CrewAI adapter follows:
pyproject.toml pins crewai==1.15.34 and mcp<2 (the SDK's 1.x client API,
as Hermes pins it) with the shared [tool.uv] settings, uv.lock is
generated with uv 0.13.4, and requirements.txt is gone. The lock resolves
the versions the adapter was tested with (crewai 1.15.34, mcp 1.30.0,
httpx 0.28.1).

Conflict in scripts/harness_conformance/ci.py: keep main's locked sync and
add crewai to the harnesses that need no further install. The conformance
runner README now describes the locked install and the current tuples.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 7657c9f

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e985036

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test 7b1f06a

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test 9bd3410

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test 2111706

weijiac0619
weijiac0619 previously approved these changes Nov 12, 2026
@weijiac0619

Copy link
Copy Markdown
Contributor

checked @AdamRajfer 's two blockers and the bot’s risks against the current head. LGTM

@yaoyu-33
yaoyu-33 added this pull request to the merge queue Nov 12, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 12, 2026
Preserve CrewAI together with the DeepAgents, Cline, and Kimi Code documentation and retain configuration coverage for both CrewAI and Cline.

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test c871686

@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team November 13, 2026 01:26
@yaoyu-33
yaoyu-33 added this pull request to the merge queue Nov 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43346914024

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/43347400937

Merged via the queue into main with commit 1f88c43 Nov 13, 2026
76 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants