Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 47 additions & 1 deletion benchmarks/gdpval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,51 @@ Downloads `openai/gdpval` from HuggingFace and writes
gym eval prepare --benchmark gdpval
```

## Prerequisites for every run

The Stirrup agent needs Ray, and two settings have no working default. Set up all
three before any `gym eval run` below. A judge-only run (`JUDGE_ONLY=true`, see
[Fresh vs. cached deliverables](#fresh-vs-cached-deliverables)) executes no task
and needs Ray and the deliverables directory, not the sandbox.

Run every command in this README from the repository root: `gym eval prepare`
looks for `benchmarks/gdpval/prepare.py` relative to the current directory, and
the exports below use `$PWD`.

**Ray.** Ray is an optional NeMo Gym extra. Without it where `gym` runs,
`gym eval run` stops with ``These servers use Ray, but Ray isn't installed where
`gym` runs: gdpval_stirrup_agent``. Add `--extra ray` to your `uv sync`
(`nemo-gym[ray]` with pip):

```bash
uv sync --extra dev --extra ray
```

**Apptainer sandbox.** GDPVal tasks run their code only inside the Apptainer
image built from
[`containers/gdpval.def`](../../responses_api_agents/stirrup_agent/containers/gdpval.def),
and the host that runs the agent server needs the `apptainer` binary. The agent
rejects its local backend for GDPVal. Without the image, every task fails with
`RuntimeError: GDPval requires the Apptainer sandbox ...` and the run stops with
`None of the N dispatched rollouts produced a result`, writing no aggregate
metrics. Build the image from its own directory, verify it as described in
[Apptainer Sandboxing](../../responses_api_agents/stirrup_agent/README.md#apptainer-sandboxing),
and point the benchmark at it with an absolute path:

```bash
(cd responses_api_agents/stirrup_agent/containers && apptainer build gdpval.sif gdpval.def)
export GDPVAL_CONTAINER_PATH=$PWD/responses_api_agents/stirrup_agent/containers/gdpval.sif
```

**Absolute deliverables directory.** The agent saves each task's deliverables
under `persist_deliverables_dir` and requires an absolute path. The config
default, `output/gdpval/deliverables`, is relative, so the agent server exits at
startup with `persist_deliverables_dir must be an absolute path` unless you set:

```bash
export PERSIST_DELIVERABLES_DIR=$PWD/output/gdpval/deliverables
```

## Run rubric mode (default)

Each deliverable is scored 0–1 against the task rubric.
Expand Down Expand Up @@ -670,7 +715,8 @@ judge proxies start idle; drop them with `~<name>` if you want a clean run.

After `gym eval run` returns, the resources server's
`/aggregate_metrics` endpoint emits headline scores in
`results/<output>_metrics.json`:
`<output stem>_aggregate_metrics.json` next to the rollouts file (for example
`results/gdpval_rubric_aggregate_metrics.json`):

- Rubric mode: `mean/reward` (pass@1 equivalent), computed over rows with a
usable judge response. Rows flagged `invalid_judge_response` are left out and
Expand Down
6 changes: 5 additions & 1 deletion benchmarks/gdpval/config.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
# GDPVal benchmark — Stirrup agent + GDPVal resources server.
#
# Run:
# Run from the repository root. Needs the `ray` extra and both exports; see
# "Prerequisites for every run" in README.md:
# uv sync --extra dev --extra ray
# export GDPVAL_CONTAINER_PATH=/abs/path/to/gdpval.sif # Apptainer image built from containers/gdpval.def
# export PERSIST_DELIVERABLES_DIR=/abs/path/to/deliverables
# gym eval prepare --benchmark gdpval
# gym eval run \
# --model-type vllm_model --benchmark gdpval \
Expand Down
32 changes: 22 additions & 10 deletions responses_api_agents/stirrup_agent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,10 +66,11 @@ For each GDPVal task, the agent:

## Prerequisites

1. **Install NeMo Gym** (see the [top-level README](../../README.md)):
1. **Install NeMo Gym** with the `ray` extra; the agent runs on Ray (see the
[top-level README](../../README.md)):
```bash
uv venv --python 3.13.14 && source .venv/bin/activate
uv sync --extra dev
uv sync --extra dev --extra ray
```
2. **Install document-generation dependencies** (needed for the GDPVal deliverable formats —
`.docx`, `.xlsx`, `.pptx`, `.pdf`):
Expand All @@ -80,8 +81,9 @@ For each GDPVal task, the agent:
# System-level (Ubuntu/Debian):
sudo apt install libreoffice libpango1.0-dev libcairo2-dev libgdk-pixbuf2.0-dev
```
3. **(Optional) Install Apptainer** if you want sandboxed code execution
(see [Apptainer Sandboxing](#apptainer-sandboxing)).
3. **Install Apptainer.** Required for `task: gdpval`: the agent rejects its local
backend for GDPVal and runs every task in the Apptainer image (see
[Apptainer Sandboxing](#apptainer-sandboxing)). Other tasks use the local backend.

## Quick Start

Expand All @@ -94,7 +96,10 @@ agent with the GDPVal resources server and supports
# 1. Prepare the GDPVal benchmark JSONL.
gym eval prepare --benchmark gdpval

# 2. Collect rollouts end-to-end (servers spin up automatically).
# 2. Collect rollouts end-to-end (servers spin up automatically). Both paths
# must be absolute; build gdpval.sif first (see Apptainer Sandboxing).
GDPVAL_CONTAINER_PATH=/abs/path/to/gdpval.sif \
PERSIST_DELIVERABLES_DIR=/abs/path/to/deliverables \
JUDGE_API_KEY=... HF_TOKEN=... \
gym eval run \
--model-type openai_model \
Expand All @@ -109,7 +114,7 @@ gym eval run \
Each output line contains `responses_create_params`, the full `response`, a
`reward` in `[0, 1]`, and a `judge_response` with per-criterion breakdown.
Aggregate metrics (`mean/reward` for rubric mode, ELO for comparison mode)
land in `results/gdpval_rubric_metrics.json`.
land in `results/gdpval_rubric_aggregate_metrics.json`.

## Configuration

Expand All @@ -125,7 +130,7 @@ The agent reads its Hydra config at `configs/stirrup_agent.yaml`. Notable keys:
| `user_prompt_template` | `???` | Path to the user prompt Jinja2 template. |
| `resources_server` | required | Reference to the GDPVal resources server (which scores the deliverable via `/verify`). |
| `gdpval_container_path` | `null` | Path to an Apptainer `.sif` (see below). |
| `persist_deliverables_dir` | `null` | If set, each task's artifacts land in `<dir>/task_<task_id>/`. The resources server reads this dir to score the deliverable. |
| `persist_deliverables_dir` | `null` | If set, each task's artifacts land in `<dir>/task_<task_id>/`. The resources server reads this dir to score the deliverable. Must be an absolute path. |
| `execute_only` | `false` | If true, run tasks and cache deliverables but **skip judging** — no `reward` / `judge_response` (see [Task-Only Execution Mode](#task-only-execution-mode)). Mutually exclusive with `judge_only`. |
| `judge_only` | `false` | If true, skip task execution and score the deliverables already cached under `persist_deliverables_dir` (see [Judge-Only Mode](#judge-only-mode)). Mutually exclusive with `execute_only`. |
| `rerun_incomplete` | `false` | If true, skip the rollout for tasks that already **finished** (a finish marker is cached) and only re-run the ones that didn't (see [Task Re-run Mode](#task-re-run-mode)). Composes with `execute_only` and `judge_only`. |
Expand Down Expand Up @@ -390,9 +395,13 @@ gym eval run \

### Apptainer Sandboxing

Some GDPVal tasks ask the model to install packages or run untrusted code. By default the
agent uses a local sandbox; setting `gdpval_container_path` to an Apptainer `.sif` routes
all `code_exec` calls through a persistent container.
GDPVal requires this sandbox. Its tasks ask the model to install packages, run
untrusted code, and use a heavy toolchain (TeX Live, the data/ML/document stack)
that the evaluation container does not carry. Setting `gdpval_container_path` to
an Apptainer `.sif` routes all `code_exec` calls through a persistent container.
With it unset, every GDPVal task fails with
`RuntimeError: GDPval requires the Apptainer sandbox ...`; the agent never falls
back to the local backend for GDPVal.

Build the supplied container definition. Build **from the `containers/`
directory**: the definition stages the vendored GDPval-AA v2 manifests via
Expand Down Expand Up @@ -433,6 +442,9 @@ stirrup_agent:
gdpval_container_path: /abs/path/to/gdpval.sif
```

The [GDPVal benchmark](../../benchmarks/gdpval/README.md) reads the path from the
`GDPVAL_CONTAINER_PATH` environment variable instead.

### Pairwise ELO Judging

Pairwise comparison vs. a reference model is built into the GDPVal resources
Expand Down
Loading