Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
1d342a8
feat(crewai): add the CrewAI harness adapter with agent sessions
yaoyu-33 Nov 3, 2026
0bd10e3
docs(crewai): document the CrewAI adapter and list it among harness t…
yaoyu-33 Nov 3, 2026
8c51006
fix(crewai): keep runner notices out of tool output and classify mode…
yaoyu-33 Nov 3, 2026
0cc7587
Merge remote-tracking branch 'origin/main' into claude/gymint-563-cre…
yaoyu-33 Nov 3, 2026
0c3c7ae
feat(crewai): start an agent-owned sandbox, declare TB2.1 support, an…
yaoyu-33 Nov 3, 2026
e169278
fix(crewai): let collection join retried calls and report the sandbox…
yaoyu-33 Nov 4, 2026
63b6727
docs(crewai): state the Observability level the conformance probes no…
yaoyu-33 Nov 4, 2026
7657c9f
Merge origin/main and lock the CrewAI adapter environment
yaoyu-33 Nov 4, 2026
e985036
Merge origin/main (#5589 moves the unit tests)
yaoyu-33 Nov 4, 2026
bc6812b
fix(crewai): keep deadlines, retries and tool failures gradable
yaoyu-33 Nov 5, 2026
613224a
Merge remote-tracking branch 'origin/main' into review-fix/5540
yaoyu-33 Nov 5, 2026
97cb8ff
Merge origin/main
yaoyu-33 Nov 5, 2026
ffbda95
refactor(crewai): run the crew inside the task sandbox
yaoyu-33 Nov 5, 2026
7d3a670
ci(conformance): build CrewAI's sandbox runtime before its probes
yaoyu-33 Nov 5, 2026
5aeb419
Merge the CrewAI review fixes (bc6812b54) into the sandbox shape
yaoyu-33 Nov 5, 2026
37cde14
docs(crewai): document network_access
yaoyu-33 Nov 5, 2026
311bbc9
Merge origin/main into claude/gymint-563-crewai-sessions-0db917
yaoyu-33 Nov 7, 2026
919fc76
fix(crewai): preserve session outcomes and qualification limits
yaoyu-33 Nov 7, 2026
738a751
test(crewai): cover unknown failure reporting
yaoyu-33 Nov 7, 2026
4b1cc4a
fix(crewai): preserve infrastructure causes and bound model retries
yaoyu-33 Nov 7, 2026
d6f2c52
fix(crewai_agent): keep CrewAI from retrying model calls
yaoyu-33 Nov 9, 2026
6dc8087
chore(crewai): merge main and resolve harness table
yaoyu-33 Nov 10, 2026
3ac7e34
chore(ci): merge main and preserve CrewAI migration coverage
yaoyu-33 Nov 10, 2026
d25a0fb
test(agentic-if): isolate resume checks from concurrent source changes
yaoyu-33 Nov 10, 2026
83b7f4e
test(pi): wait for detached child readiness before shutdown
yaoyu-33 Nov 10, 2026
7b1f06a
fix(crewai): resolve main conflicts and release validation
yaoyu-33 Nov 11, 2026
9bd3410
fix(crewai): align the component lock with the shared SDK pin
yaoyu-33 Nov 11, 2026
2111706
chore(crewai): merge main and preserve harness registrations
yaoyu-33 Nov 12, 2026
c871686
chore(crewai): resolve harness integration conflicts with main
yaoyu-33 Nov 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions benchmarks/terminal_bench_2_1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

Terminal-Bench 2.1 evaluates agents on tasks in isolated containers. The benchmark
prepares task data and owns sandbox setup, verification, and cleanup. Pi, Codex,
OpenCode, OpenClaw, and Copilot can run their model/tool loops in the task sandbox through
Gym's Environment Server.
OpenCode, OpenClaw, Copilot, and CrewAI can run their model/tool loops in the task sandbox
through Gym's Environment Server.

- Integration profile: `custom-gym-verifier`
- Scorer: `terminal_bench_2_1`
Expand Down Expand Up @@ -83,10 +83,12 @@ the harness for this taskset.
| [OpenCode](../../responses_api_agents/opencode_agent/README.md) | `responses_api_agents/opencode_agent/configs/opencode_agent.yaml` | `opencode_agent` | `context_window: 32768`, `max_output_tokens: 8192`, `timeout: 600` |
| [OpenClaw](../../responses_api_agents/openclaw_agent/README.md) | `responses_api_agents/openclaw_agent/configs/openclaw_agent.yaml` | `openclaw_agent` | `context_window: 32768`, `model_timeout_seconds: 600`, `timeout: 600` |
| [Copilot](../../responses_api_agents/copilot_agent/README.md) | `responses_api_agents/copilot_agent/configs/copilot_agent.yaml` | `copilot_agent` | `context_window: 32768`, `max_output_tokens: 8192`, `timeout: 600` |
| [CrewAI](../../responses_api_agents/crewai_agent/README.md) (development pairing) | `responses_api_agents/crewai_agent/configs/crewai_agent.yaml` | `crewai_agent` | `context_window_size: 131072`, `shell_timeout_seconds: 300`, `timeout: 900` |

Pi, Codex, OpenCode, and OpenClaw are allowed by this benchmark. Copilot remains experimental pending
full qualification; use `--allow-unsupported-pairing` to try its smoke settings. These are single-task
smoke settings; adjust context windows and budgets for your model and evaluation.
Pi, Codex, OpenCode, and OpenClaw are allowed by this benchmark. Copilot and CrewAI remain
unsupported development pairings pending full benchmark qualification; add
`--allow-unsupported-pairing` to both `gym env start` and `gym eval run` when selecting either.
These are single-task smoke settings; adjust context windows and budgets for your model and evaluation.
See each harness's README for runtime prerequisites and model requirements.

### Model and sandbox provider
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,7 @@ The current adapters support the following tool sources:
| Codex | Codex's shell and file tools. Agent sessions require `sandbox_mode: danger-full-access`, because the task sandbox supplies isolation. | `mcp` only. Each granted MCP server becomes a Codex `mcp_servers` entry with the same name. Required `direct_http` grants are rejected. | Responses (streaming) |
| OpenCode | OpenCode's built-in tools. Session `opencode_config` permits only `permission`, `tools`, `attachment`, `compaction.auto` and `agent.build.steps`. | `mcp` only. Each granted MCP server becomes an OpenCode `remote` MCP server with the same name; a required one that does not connect fails the activation. Required `direct_http` grants are rejected. | Chat Completions |
| OpenClaw | `read`, `write`, `edit`, `exec`, and `process`. | `mcp` only. Each granted MCP server becomes an OpenClaw `mcp.servers` entry with the same name, probed before the run; a required one that does not connect fails the activation. Required `direct_http` grants are rejected. | Chat Completions |
| CrewAI | `shell`, which runs each command in the task's working directory. The crew runs in the task sandbox: the borrowed one, or an agent-owned one when none is lent. | `mcp` only. Each tool of a granted MCP server becomes a CrewAI tool under its own name; a required server that does not list its tools fails the activation. Required `direct_http` grants are rejected. | Chat Completions |
| DeepAgents (`app.py`, `tool_profile: builtin`) | deepagents' own tools: `write_todos`, `ls`, `read_file`, `write_file`, `edit_file`, `glob`, `grep`, `execute`, and `task` subagents. `shell_timeout_seconds` sets the default limit of each `execute` call. | None in agent sessions. Required grants are rejected; keep `resources_tool_transports: []`. | Responses |
| Cline | Cline's built-in tools (shell commands, file reads and edits); `command_permissions` restricts its shell commands. | None in agent sessions. Required grants are rejected and optional ones skipped; keep `resources_tool_transports: []`. | Chat Completions |
| OpenHands | OpenHands' terminal (subprocess backend), file editor, task tracker, `finish` and `think`, with its default condenser. | None. Required grants are rejected; keep `resources_tool_transports: []`. | Chat Completions |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ These adapters run a harness inside a sandbox. The `single_agent_turn` Environme
| mini-SWE (`mini_swe_agent`, `mini_swe_agent_sandboxed.yaml`) | Responses | MCP only, through a task-local CLI that its bash tool calls | Not wired up; evaluation only | Not declared | Not declared | Only when the task runs as root: a task user is refused |
| Droid (`droid_agent`) | Chat Completions (streaming) | None | Not wired up; evaluation only | Not declared | Not declared | Not validated |
| Goose (`goose_agent`) | Chat Completions | None | Not validated; evaluation only | Not declared | Not declared | Not validated |
| CrewAI (`crewai_agent`) | Chat Completions | MCP only | Token capture untested; training not qualified | Not declared | Not declared (development only) | Not validated |
| Kimi Code (`kimi_code_agent`) | Chat Completions | None | Not validated; evaluation only | Not declared | Not declared | Not validated |

*Declared* means the benchmark's Resources server lists the agent in `allowed_agents`. Gym rejects any other pairing unless you pass `--allow-unsupported-pairing`. Declared support is not a validation: it says nothing about accuracy or training readiness. Check that a pairing works with a real model and the benchmark's verifier before you rely on it.
Expand Down
1 change: 1 addition & 0 deletions nemo_gym/config/global_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,7 @@ def _installed_version(distribution: str) -> Optional[str]:
"cline_agent",
"codex_agent",
"copilot_agent",
"crewai_agent",
"droid_agent",
"goose_agent",
"harbor_oracle_agent",
Expand Down
Loading
Loading