Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ harbor run -p harbor_cookbook/recipes/<name> -a claude-code -m anthropic/claude-
|:--|:--|
| [simple&#8209;task](harbor_cookbook/recipes/simple-task/) | Minimal single-container task. |
| [multi&#8209;container](harbor_cookbook/recipes/multi-container/) | Docker Compose task where the agent interacts with a locally hosted REST API. |
| [postgres&#8209;snapshot&#8209;regrade](harbor_cookbook/recipes/postgres-snapshot-regrade/) | Capture PostgreSQL state, grade it in a separate verifier, and regrade the saved snapshot. |
| [mcp&#8209;tools](harbor_cookbook/recipes/mcp-tools/) | Giving the agent custom tools via a locally hosted FastMCP server. |
| [multi&#8209;reward](harbor_cookbook/recipes/multi-reward/) | Multiple independent verifiers each producing their own score. |
| [simulated&#8209;user](harbor_cookbook/recipes/simulated-user/) | Agent discovers requirements by talking to a simulated user. |
Expand Down
108 changes: 108 additions & 0 deletions harbor_cookbook/recipes/postgres-snapshot-regrade/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# PostgreSQL snapshot and regrade

This recipe demonstrates grading a database task from a recorded artifact.
The agent repairs order totals in a PostgreSQL sidecar. A collect hook captures
the resulting rows as JSON, and a separate verifier grades that snapshot.
The same saved trial can then be regraded after the original containers have
been deleted, without running the agent or database again.

The repair is deliberately small: three incorrect totals, one empty order,
and one already correct order. The example teaches artifact collection and
regrading, rather than measuring difficult SQL problem solving.

## How it works

1. Docker Compose starts `main` with `psql` and a `database` sidecar with
PostgreSQL 17.11. Connection settings are provided through `PG*`
environment variables. The database has no published host port.
2. The agent can read the three tables and update only `orders.total_cents`.
The database's `pg_hba.conf` permits remote connections only for this
restricted role. The initialization password is a demo value; it cannot
be used to connect remotely as the database superuser.
3. After the agent finishes, Harbor stops `main`. The sidecar
`[[verifier.collect]]` hook then runs `capture-state` as `postgres` using a
local Unix socket. One SQL statement captures all grading inputs from a
consistent database snapshot, with deterministic row ordering. The script
renames the output into place only after the query succeeds.
4. The artifact declaration names both its absolute path and service:

```toml
artifacts = [{ source = "/snapshot/state.json", service = "database" }]
```

Harbor records the file and its artifact manifest in the trial directory.
It restores the file at `/snapshot/state.json` in the separate verifier
built from `tests/Dockerfile`.
5. The verifier checks the saved rows, including the zero-total case and
preservation of other data. It has no PostgreSQL client or server and
makes no connection to the original database. Test dependencies are
installed by `tests/test.sh`, so the verifier needs internet access for
installation.
6. `harbor trial regrade` creates a new trial from the recorded artifacts and
runs the verifier again. It leaves the source trial unchanged and does
not rerun the collect hook against a live database.

Keeping the artifact as JSON also lets the verifier inspect data without
executing a SQL dump. For a larger task, include every piece of state needed
by its verifier in the capture query; unrecorded database state cannot be
recovered later by regrading.

## Run without model credentials

Requirements: Docker with Compose and [uv](https://docs.astral.sh/uv/).
Run these commands from the repository root. They use released Harbor 0.24.0
and Python 3.13.17.

```bash
recipe=harbor_cookbook/recipes/postgres-snapshot-regrade
runs=$(mktemp -d)

# The reference solution repairs all totals: expected reward 1.
uvx --python 3.13.17 --from 'harbor==0.24.0' harbor run \
-p "$recipe" -a oracle -e docker --jobs-dir "$runs" --job-name oracle

# An agent that does nothing leaves three incorrect totals: expected reward 0.
uvx --python 3.13.17 --from 'harbor==0.24.0' harbor run \
-p "$recipe" -a nop -e docker --jobs-dir "$runs" --job-name nop
```

Both jobs use Harbor's default environment cleanup. A reward of zero from
Nop is expected; an exception or missing reward is an infrastructure failure.
The verifier writes test details to `verifier/ctrf.json` and its score to
`verifier/reward.txt` inside each trial directory.

## Regrade the saved trial

Select the single Oracle trial, rather than the parent job directory:

```bash
source_trial=$(python3 - "$runs/oracle" <<'PY'
from pathlib import Path
import sys

trials = list(Path(sys.argv[1]).glob("*/result.json"))
assert len(trials) == 1, f"Expected one trial, found {len(trials)}"
print(trials[0].parent)
PY
)

# Inspect the manifest to see which service supplied the recorded artifact.
cat "$source_trial/artifacts/manifest.json"

uvx --python 3.13.17 --from 'harbor==0.24.0' harbor trial regrade \
"$source_trial" -p "$recipe" -e docker \
--trials-dir "$runs/regraded" --trial-name oracle-regraded
```

The regraded trial should also receive reward **1**, even though the original
database no longer exists. Repeating the selection with `$runs/nop` and using
a different `--trial-name` should preserve reward **0**.

Keep the complete source trial directory, including its manifest and artifact
files. A missing required snapshot is a regrade error. The manifest maps saved
files to their service and original path; the verifier reads
`/snapshot/state.json`, not a guessed path beneath the host trial directory.

To try a revised verifier, edit only `tests/` and regrade into a new output
directory. Compare both trials' CTRF reports to see how the grading changed,
while keeping the original agent output and snapshot intact.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
FROM ubuntu:24.04@sha256:68ee22004a9109b44973313b02ff7b1de250f1de0f5ecb628d73f579aaff33ac

RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates postgresql-client \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /app
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# PostgreSQL 17.11; use the official image for database initialization.
FROM postgres:17-bookworm@sha256:858bb5e98eed02eb8cafbcdc007f9a65d1b613972028944a8ec2133ccaf80d27

COPY init.sql /docker-entrypoint-initdb.d/01-init.sql
COPY pg_hba.conf /etc/postgresql/recipe-pg_hba.conf
COPY capture.sql /opt/recipe/capture.sql
COPY --chmod=755 capture-state.sh /usr/local/bin/capture-state

RUN install -d -o postgres -g postgres -m 700 /snapshot
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/bin/sh
set -eu

snapshot_tmp=$(mktemp /snapshot/state.XXXXXX)
trap 'rm -f "$snapshot_tmp"' EXIT HUP INT TERM

psql -X -A -t -v ON_ERROR_STOP=1 -U postgres -d orders \
-f /opt/recipe/capture.sql > "$snapshot_tmp"
mv "$snapshot_tmp" /snapshot/state.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
-- One statement provides a consistent snapshot of all grading inputs.
-- Explicit columns, table names, and ordering keep the artifact deterministic.
SELECT jsonb_build_object(
'schema_version', 1,
'customers', (
SELECT COALESCE(jsonb_agg(to_jsonb(row) ORDER BY id), '[]'::jsonb)
FROM (SELECT id, name FROM public.customers) AS row
),
'orders', (
SELECT COALESCE(jsonb_agg(to_jsonb(row) ORDER BY id), '[]'::jsonb)
FROM (SELECT id, customer_id, total_cents FROM public.orders) AS row
),
'line_items', (
SELECT COALESCE(jsonb_agg(to_jsonb(row) ORDER BY id), '[]'::jsonb)
FROM (
SELECT id, order_id, quantity, unit_price_cents FROM public.line_items
) AS row
)
);
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
CREATE TABLE public.customers (
id integer PRIMARY KEY,
name text NOT NULL
);

CREATE TABLE public.orders (
id integer PRIMARY KEY,
customer_id integer NOT NULL REFERENCES public.customers(id),
total_cents integer NOT NULL CHECK (total_cents >= 0)
);

CREATE TABLE public.line_items (
id integer PRIMARY KEY,
order_id integer NOT NULL REFERENCES public.orders(id),
quantity integer NOT NULL CHECK (quantity > 0),
unit_price_cents integer NOT NULL CHECK (unit_price_cents >= 0)
);

INSERT INTO public.customers VALUES (1, 'Ada'), (2, 'Grace');
INSERT INTO public.orders VALUES
(101, 1, 0), (102, 2, 999), (103, 1, 77), (104, 2, 1500);
INSERT INTO public.line_items VALUES
(1, 101, 2, 1200),
(2, 101, 1, 100),
(3, 102, 1, 2999),
(4, 102, 2, 250),
(5, 104, 3, 500);

CREATE ROLE agent LOGIN PASSWORD 'agent';
REVOKE CREATE ON SCHEMA public FROM PUBLIC;
REVOKE ALL ON DATABASE orders FROM PUBLIC;
GRANT CONNECT ON DATABASE orders TO agent;
GRANT USAGE ON SCHEMA public TO agent;
GRANT SELECT ON public.customers, public.orders, public.line_items TO agent;
GRANT UPDATE (total_cents) ON public.orders TO agent;
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# The trusted collect hook uses a local socket inside the database container.
local all postgres trust
# The agent can only authenticate as the restricted role over the network.
host orders agent all scram-sha-256
host all all all reject
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Harbor supplies main's build context, command, volumes, and resource limits.
services:
main:
environment:
PGHOST: database
PGPORT: "5432"
PGDATABASE: orders
PGUSER: agent
PGPASSWORD: agent
depends_on:
database:
condition: service_healthy

database:
build:
context: ./database
environment:
POSTGRES_DB: orders
# Initialization requires a password. pg_hba.conf rejects remote
# superuser connections, including those using this demo password.
POSTGRES_PASSWORD: cookbook-local-only
command: ["postgres", "-c", "hba_file=/etc/postgresql/recipe-pg_hba.conf"]
healthcheck:
test: ["CMD", "pg_isready", "-h", "127.0.0.1", "-U", "agent", "-d", "orders"]
interval: 2s
timeout: 5s
retries: 30
start_period: 5s
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
The order totals in the PostgreSQL database are inconsistent with their line
items. Repair `public.orders.total_cents` so that every order's total equals
the sum of `quantity * unit_price_cents` for its line items. An order with no
line items must have a total of zero.

Use `psql`; the `PGHOST`, `PGPORT`, `PGDATABASE`, `PGUSER`, and `PGPASSWORD`
environment variables are already configured. Inspect the schema and data
before making changes. Preserve every row and every column other than
`orders.total_cents`. Commit your changes before finishing.
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/bin/bash
set -euo pipefail

psql -X -v ON_ERROR_STOP=1 <<'SQL'
BEGIN;
UPDATE public.orders AS orders
SET total_cents = COALESCE((
SELECT SUM(items.quantity * items.unit_price_cents)
FROM public.line_items AS items
WHERE items.order_id = orders.id
), 0);
COMMIT;
SQL
34 changes: 34 additions & 0 deletions harbor_cookbook/recipes/postgres-snapshot-regrade/task.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
schema_version = "1.1"

# Collect the file from the database service, not the agent container.
artifacts = [{ source = "/snapshot/state.json", service = "database" }]

[task]
name = "cookbook/postgres-snapshot-regrade"
description = "Capture PostgreSQL state, grade it in a separate verifier, and regrade the recorded snapshot."
keywords = ["postgresql", "docker-compose", "artifacts", "separate-verifier", "regrade"]

[metadata]
difficulty = "easy"
category = "example"
tags = ["postgresql", "artifacts", "regrade"]

[verifier]
timeout_sec = 180.0
environment_mode = "separate"

# Harbor stops the main container before running sidecar collect hooks.
# This script uses a local socket and cannot be edited by the agent's DB role.
[[verifier.collect]]
service = "database"
command = "/usr/local/bin/capture-state"
user = "postgres"
timeout_sec = 30.0

[agent]
timeout_sec = 300.0

[environment]
build_timeout_sec = 600.0
cpus = 1
memory_mb = 1024
10 changes: 10 additions & 0 deletions harbor_cookbook/recipes/postgres-snapshot-regrade/tests/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
FROM ubuntu:24.04@sha256:68ee22004a9109b44973313b02ff7b1de250f1de0f5ecb628d73f579aaff33ac

COPY --from=ghcr.io/astral-sh/uv:0.13.5@sha256:32a7bb39b496babbd997b0a39a4d66c126bdaf5661b302bf6f06abda8deca6ff /uv /usr/local/bin/uv
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates \
&& rm -rf /var/lib/apt/lists/*

# Separate verifiers own their tests. No PostgreSQL server or client is needed.
COPY . /tests/
WORKDIR /tests
14 changes: 14 additions & 0 deletions harbor_cookbook/recipes/postgres-snapshot-regrade/tests/test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/bin/bash
set -euo pipefail

uv venv --python 3.13.17 /tmp/recipe-verifier
uv pip install --python /tmp/recipe-verifier/bin/python \
pytest==8.4.1 pytest-json-ctrf==0.3.5

# Dependency failures above are infrastructure errors, not a zero task score.
if /tmp/recipe-verifier/bin/python -m pytest \
--ctrf /logs/verifier/ctrf.json /tests/test_snapshot.py -rA; then
echo 1 > /logs/verifier/reward.txt
else
echo 0 > /logs/verifier/reward.txt
fi
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
"""Grade only the captured database artifact, including on regrade."""

import json
from pathlib import Path

import pytest


@pytest.fixture(scope="module")
def snapshot():
state = json.loads(Path("/snapshot/state.json").read_text())
assert set(state) == {"schema_version", "customers", "orders", "line_items"}
assert type(state["schema_version"]) is int
assert state["schema_version"] == 1
return state


def test_customers_preserved(snapshot):
assert snapshot["customers"] == [
{"id": 1, "name": "Ada"},
{"id": 2, "name": "Grace"},
]


def test_line_items_preserved(snapshot):
assert snapshot["line_items"] == [
{"id": 1, "order_id": 101, "quantity": 2, "unit_price_cents": 1200},
{"id": 2, "order_id": 101, "quantity": 1, "unit_price_cents": 100},
{"id": 3, "order_id": 102, "quantity": 1, "unit_price_cents": 2999},
{"id": 4, "order_id": 102, "quantity": 2, "unit_price_cents": 250},
{"id": 5, "order_id": 104, "quantity": 3, "unit_price_cents": 500},
]


def test_orders_preserved(snapshot):
orders = snapshot["orders"]
assert [row["id"] for row in orders] == [101, 102, 103, 104]
assert [row["customer_id"] for row in orders] == [1, 2, 1, 2]
for row in orders:
assert set(row) == {"id", "customer_id", "total_cents"}
assert all(type(value) is int for value in row.values())


def test_first_order_total(snapshot):
assert snapshot["orders"][0]["total_cents"] == 2500


def test_second_order_total(snapshot):
assert snapshot["orders"][1]["total_cents"] == 3499


def test_empty_order_total(snapshot):
assert snapshot["orders"][2]["total_cents"] == 0


def test_correct_order_total_preserved(snapshot):
assert snapshot["orders"][3]["total_cents"] == 1500
5 changes: 5 additions & 0 deletions harbor_cookbook/recipes/registry.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,11 @@
"description": "Docker Compose task with an additional service the agent interacts with over the Docker network.",
"path": "recipes/multi-container"
},
{
"name": "postgres-snapshot-regrade",
"description": "Capture PostgreSQL sidecar state as an artifact, grade it in a separate verifier, and regrade without the original database.",
"path": "recipes/postgres-snapshot-regrade"
},
{
"name": "mcp-tools",
"description": "Giving agents access to custom MCP tools via a FastMCP server running in a companion container.",
Expand Down