Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -291,7 +291,7 @@ The Dataset column links to publicly available datasets (e.g., on HuggingFace).
| Openenv | games | Maze navigation environment via OpenEnv. Agent navigates an 8x8 grid to find the exit. | - | - | - | - | <a href='resources_servers/openenv/configs/openenv_maze.yaml'>openenv_maze.yaml</a> | - |
| Over Refusal Detection | | - | - | ✓ | - | TBD | <a href='resources_servers/over_refusal_detection/configs/over_refusal_detection.yaml'>over_refusal_detection.yaml</a> | - |
| Physics Judge | math | Physics QA verified by NeMo Skills' physics judge LLM with math-verify symbolic fallback | Score open-ended physics benchmarks (e.g. PHYSICS) where the judge emits [Correct] / [Incorrect] verdicts | - | - | - | <a href='resources_servers/physics_judge/configs/physics_judge.yaml'>physics_judge.yaml</a> | - |
| Pinchbench | | - | - | - | - | - | <a href='responses_api_agents/pinchbench/configs/pinchbench.yaml'>pinchbench.yaml</a> | - |
| Pinchbench | agent | PinchBench benchmark integration | Evaluate a model as the brain of an OpenClaw agent on real-world tasks. | - | - | - | <a href='responses_api_agents/pinchbench/configs/pinchbench.yaml'>pinchbench.yaml</a> | - |
| Polymath | math | PolyMath multilingual math benchmark with weighted (difficulty) and per-language metrics | Improve multilingual math reasoning across 18 languages and 4 difficulty tiers | - | - | - | <a href='resources_servers/polymath/configs/polymath.yaml'>polymath.yaml</a> | - |
| Proof Genselect | math | Pairwise proof selection with binary correctness reward | - | - | - | - | <a href='resources_servers/proof_genselect/configs/proof_genselect.yaml'>proof_genselect.yaml</a> | - |
| Proof Judge | math | Theorem proving with verifier + meta-verifier judge (combined env) | - | - | - | - | <a href='resources_servers/proof_judge/configs/proof_judge.yaml'>proof_judge.yaml</a> | - |
Expand Down
64 changes: 64 additions & 0 deletions benchmarks/pinchbench/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# PinchBench

All 147 tasks of [PinchBench](https://github.com/pinchbench/skill) — calendar, email triage,
CSV/log analysis, coding, research, writing — measuring how well a model performs as the brain
of an [OpenClaw](https://github.com/openclaw/openclaw) agent.

This benchmark config is a thin narrowing of the `pinchbench` agent server, which owns the whole
integration (one sandbox per task, in-sandbox OpenClaw gateway, PinchBench's own grading harness).
Read [`responses_api_agents/pinchbench/README.md`](../../responses_api_agents/pinchbench/README.md)
first — architecture, the per-task image build, config knobs, and the parity validation against
vanilla standalone PinchBench all live there.

## Verification

Grading is PinchBench's own, run inside the sandbox: deterministic checks (`automated`), an LLM
judge (`llm_judge`), or a weighted mix of both (`hybrid`). Reward is **continuous** in `[0,1]`,
not pass/fail — threshold it if you need a binary signal. Each rollout also returns
`grading_type`, `grading_breakdown`, `grading_notes` and `raw_rollout`.

`num_repeats` is 3: temperature 1.0 plus live web search make a single pass noisy, and 3 is what
the agent-side parity validation used. Trust aggregates, not individual runs.

## Data preparation

```
gym eval prepare --benchmark pinchbench
```

The PinchBench skill is not vendored. `prepare.py` clones it at `v2.0.0` — the same ref
`responses_api_agents/pinchbench/Dockerfile.benchmark` bakes into the per-task image — and writes
`data/pinchbench_benchmark.jsonl`, one row per manifest task:

```json
{"responses_create_params": {"input": [{"role": "user", "content": "<the task's ## Prompt section>"}]},
"verifier_metadata": {"task_id": "task_sanity"}}
```

`task_id` is the authoritative selector: at run time `run_task.sh` passes it to
`benchmark.py --suite`, which loads the full task (prompt + assets + grading) from the skill inside
the sandbox. `input` carries the prompt for readability only. **Subset by dropping rows** — no
config change needed.

Set `PINCHBENCH_SKILL_DIR` to an existing `v2.0.0` checkout to skip the clone. Preparation fails
loudly if the manifest is not 147 tasks or if the regenerated rows do not byte-match the committed
`responses_api_agents/pinchbench/data/example.jsonl`, so upstream drift can't slip through.

## Quickstart

Build the per-task image first (`setup_scripts/build_image.sh --apptainer`), then:

```
gym eval run --benchmark pinchbench --model-type vllm_model \
+sandbox_image=<pinchbench.sif | docker://pinchbench-openclaw:latest> \
+model_base_url=<endpoint/v1> +model_api_key=<key> +model_name=<model> \
+judge_model=<judge> +judge_base_url=<endpoint/v1> +judge_api_key=<key> \
+brave_api_key=<key>
```

Those eight keys are declared in `config.yaml` with `null` defaults so `gym list` and
`gym eval prepare` resolve without them; each also reads a `PINCHBENCH_*` env var. A `null` still
reaching the agent fails fast — the fields are required in `PinchBenchAgentConfig`.

For slow or highly concurrent endpoints, raise `openclaw_provider_timeout_seconds` **and**
`task_timeout_s` together; see the agent README's "OpenClaw LLM idle timeout" section.
14 changes: 14 additions & 0 deletions benchmarks/pinchbench/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
45 changes: 45 additions & 0 deletions benchmarks/pinchbench/config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# PinchBench — 147 real-world OpenClaw agent tasks. Run:
# gym eval prepare --benchmark pinchbench
# gym eval run --benchmark pinchbench --model-type vllm_model \
# +sandbox_image=<pinchbench.sif | docker://pinchbench-openclaw:latest> \
# +model_base_url=<endpoint/v1> +model_api_key=<key> +model_name=<model> \
# +judge_model=<judge> +judge_base_url=<endpoint/v1> +judge_api_key=<key> \
# +brave_api_key=<key>

# Chain to existing agent config
config_paths:
- responses_api_agents/pinchbench/configs/pinchbench.yaml

# Run-time knobs the agent config interpolates. Declared here so `gym list` and
# `gym eval prepare` resolve without them; a null reaching the agent fails fast in
# PinchBenchAgentConfig (the fields are required `str`). Supply real values on the
# command line (`+model_base_url=...`, which merges last) or via these env vars.
sandbox_image: ${oc.env:PINCHBENCH_SANDBOX_IMAGE,null}
model_base_url: ${oc.env:PINCHBENCH_MODEL_BASE_URL,null}
model_api_key: ${oc.env:PINCHBENCH_MODEL_API_KEY,null}
model_name: ${oc.env:PINCHBENCH_MODEL_NAME,null}
judge_model: ${oc.env:PINCHBENCH_JUDGE_MODEL,null}
judge_base_url: ${oc.env:PINCHBENCH_JUDGE_BASE_URL,null}
judge_api_key: ${oc.env:PINCHBENCH_JUDGE_API_KEY,null}
brave_api_key: ${oc.env:PINCHBENCH_BRAVE_API_KEY,null}
tavily_api_key: ${oc.env:PINCHBENCH_TAVILY_API_KEY,null}

# `_inherit_from` keeps this benchmark isolated from the generic agent instance
# above, so overriding the dataset list here does not mutate `pinchbench_agent`.
pinchbench_benchmark_agent:
_inherit_from: pinchbench_agent
responses_api_agents:
pinchbench:
datasets:
- name: pinchbench
type: benchmark
jsonl_fpath: benchmarks/pinchbench/data/pinchbench_benchmark.jsonl
prompt_config: null
prepare_script: benchmarks/pinchbench/prepare.py
# Reward is continuous and temperature 1.0 + live web search make a single
# pass noisy; the parity validation in the agent README used n=3.
num_repeats: 3
license: MIT
Loading
Loading