Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions benchmarks/birdbench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,3 +138,24 @@ other flags pass through to `gym eval run`.

`rollouts.jsonl` has one row per rollout (prompt, response, reward, verifier fields like `extracted_sql` and
`failure_reason`).

## Auto Ontology

[Auto Ontology](https://github.com/NVIDIA/auto-ontology) answers a question from a catalog of the database: its tables
and columns, the business terms and definitions compiled over them, and embeddings to search both. Its ReAct agent
searches the catalog, tests queries and submits one. The Auto Ontology agent runs it in Gym, in a sandbox of its own
for each episode, from the catalog image.

### The catalog image

```bash
benchmarks/birdbench/auto_ontology/build_image.sh <models.env> # tag: nemo-gym/bird-catalog:dev
```

The image holds Auto Ontology at the commit `build_image.sh` pins, installed from its own repository and lock file,
the BIRD dev databases and their catalog. The build (`auto_ontology/build_catalog.py`) migrates the catalog with Auto
Ontology's Alembic migrations, then for each database ingests the schema, writes BIRD's column descriptions (from its
`database_description` CSVs, which a SQLite database does not carry), embeds the described catalog and compiles the
semantic layer. `<models.env>` holds the model keys (`DEFAULT_MODELS_API_KEY`, and `*_MODEL` settings where they differ
from Auto Ontology's defaults); it reaches the build as a BuildKit secret, never a layer. The build fails when a model
call failed after Auto Ontology's retries or a database has no vectors, since Auto Ontology logs both and goes on.
48 changes: 48 additions & 0 deletions benchmarks/birdbench/auto_ontology/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# syntax=docker/dockerfile:1.7
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# The Auto Ontology agent's sandbox image for BIRD dev: Auto Ontology at the commit build_image.sh pins, the BIRD dev
# databases, and a catalog Postgres with every database ingested, described and compiled. Build it with
# build_image.sh, which passes the pin, the databases and the model keys; the keys reach the catalog build as a
# BuildKit secret only, never a layer.

FROM pgvector/pgvector:pg17 AS runtime

RUN apt-get update \
&& apt-get install -y --no-install-recommends build-essential ca-certificates curl git \
&& rm -rf /var/lib/apt/lists/*
COPY --from=ghcr.io/astral-sh/uv:0.13.7 /uv /usr/local/bin/uv
ENV UV_PYTHON_INSTALL_DIR=/opt/uv-python \
UV_LINK_MODE=copy \
UV_NO_CACHE=1

# Auto Ontology from its own repository and lock file, as it ships: the package, its vendored engine and the catalog
# schema's Alembic migrations, which the package does not include.
ARG AUTO_ONTOLOGY_REPOSITORY
ARG AUTO_ONTOLOGY_COMMIT
RUN git init -q /opt/auto-ontology \
&& git -C /opt/auto-ontology fetch -q --depth 1 "$AUTO_ONTOLOGY_REPOSITORY" "$AUTO_ONTOLOGY_COMMIT" \
&& git -C /opt/auto-ontology checkout -q FETCH_HEAD
WORKDIR /opt/auto-ontology
RUN uv sync --frozen --no-dev

# The BIRD dev databases, each with its database_description CSVs.
COPY --from=bird . /opt/bird-catalog/databases/

# The catalog's Postgres. PGDATA is outside the base image's VOLUME, so the catalog built below stays in the image.
ENV PGDATA=/var/lib/postgresql/catalog \
POSTGRES_HOST=localhost \
POSTGRES_PORT=5432 \
POSTGRES_USER=postgres \
POSTGRES_PASSWORD=catalog \
POSTGRES_DATABASE=auto_ontology
COPY benchmarks/birdbench/auto_ontology/build_catalog.py /opt/bird-catalog/

# The catalog, in its own stage: `--target runtime` builds the rest without the model keys.
FROM runtime AS catalog
RUN --mount=type=secret,id=models,required=true \
/opt/auto-ontology/.venv/bin/python /opt/bird-catalog/build_catalog.py /run/secrets/models

# Last, so editing it does not rebuild the catalog.
COPY benchmarks/birdbench/auto_ontology/start-catalog /opt/bird-catalog/
3 changes: 3 additions & 0 deletions benchmarks/birdbench/auto_ontology/Dockerfile.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# The build context is the repository root: send only this directory.
*
!benchmarks/birdbench/auto_ontology/
2 changes: 2 additions & 0 deletions benchmarks/birdbench/auto_ontology/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
206 changes: 206 additions & 0 deletions benchmarks/birdbench/auto_ontology/build_catalog.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,206 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Build Auto Ontology's catalog of the BIRD dev databases, inside the catalog image.

python build_catalog.py MODELS_ENV

The Dockerfile runs this with Auto Ontology's Python. MODELS_ENV is a ``KEY=VALUE`` file with the model keys
(``DEFAULT_MODELS_API_KEY``, and ``EMBED_*``, ``NON_REASONING_*`` or ``REASONING_*`` settings where they differ from the
defaults). For each database under ``/opt/bird-catalog/databases`` it:

1. ingests the schema into the catalog (``auto_ontology.catalog.ingest_catalog``);
2. writes BIRD's column descriptions, from the database's ``database_description`` CSVs, into the catalog
(``auto_ontology.dal.datasources.apply_metadata_batch``): a SQLite database carries none, and BIRD ships them apart;
3. embeds the described tables and columns into the data layer's vector store;
4. compiles the semantic layer (``auto_ontology.semantic.compile.run_semantic_compilation``).

The catalog schema comes from Auto Ontology's own Alembic migrations. The build fails when any model call failed after
Auto Ontology's retries, or a database has no vectors: Auto Ontology logs both and goes on, which would leave a catalog
with holes. Last, it writes ``/opt/bird-catalog/.env`` with what an episode needs apart from keys: the source
databases, and the embedding and reranking endpoints and models the catalog was built with.
"""

import csv
import io
import logging
import os
import subprocess
import sys
from pathlib import Path
from typing import Optional


CATALOG_DIR = Path("/opt/bird-catalog")
DATABASES_DIR = CATALOG_DIR / "databases"
AUTO_ONTOLOGY_DIR = Path("/opt/auto-ontology")

LOG = logging.getLogger("build_catalog")


def read_env_file(path: Path) -> dict[str, str]:
values = {}
for line in path.read_text().splitlines():
line = line.strip()
if line and not line.startswith("#") and "=" in line:
key, value = line.split("=", 1)
values[key.strip()] = value.strip()
return values


def _cell(text: Optional[str]) -> str:
return " ".join((text or "").split())


def column_descriptions(description_dir: Path) -> list[dict[str, str]]:
"""BIRD's column descriptions of one database, as ``apply_metadata_batch`` column rows.

BIRD ships one CSV per table, named after it, with ``original_column_name`` (the column's name in the database),
``column_description`` and ``value_description``. A column's description is its ``column_description``, then its
``value_description`` after a comma; a ``value_description`` of "not useful" is an annotator's note and is left
out. Some CSVs are Windows-1252 rather than UTF-8.
"""
rows = []
for csv_path in sorted(description_dir.glob("*.csv")):
try:
text = csv_path.read_text(encoding="utf-8-sig")
except UnicodeDecodeError:
text = csv_path.read_text(encoding="cp1252")
# A quoted cell can span lines, as many of BIRD's value descriptions do.
for record in csv.DictReader(io.StringIO(text, newline="")):
fields = {(key or "").strip().lstrip(""): value for key, value in record.items()}
name = _cell(fields.get("original_column_name")) or _cell(fields.get("column_name"))
if not name:
continue
values = _cell(fields.get("value_description"))
if values.lower() == "not useful":
values = ""
description = ", ".join(part for part in (_cell(fields.get("column_description")), values) if part)
if description:
rows.append({"table_name": csv_path.stem, "column_name": name, "description": description})
return rows


class _FailedModelCalls(logging.Handler):
"""Counts the model calls Auto Ontology gave up on, which it logs as errors and goes on."""

def __init__(self) -> None:
super().__init__(logging.ERROR)
self.messages: list[str] = []

def emit(self, record: logging.LogRecord) -> None:
if record.name.startswith(("auto_ontology.utils.llm_invoke", "auto_ontology.utils.embedding")):
self.messages.append(record.getMessage()[:300])


def _postgres(*args: str) -> None:
subprocess.run(["su", "postgres", "-c", " ".join(args)], check=True)


def _start_postgres() -> None:
pgdata, port = os.environ["PGDATA"], os.environ["POSTGRES_PORT"]
_postgres("initdb", "-D", pgdata, f"--username={os.environ['POSTGRES_USER']}", "--auth=trust", ">/dev/null")
_postgres(
"pg_ctl", "-D", pgdata, "-l", f"{pgdata}/server.log", "-w", "-t", "120",
"-o", f"'-p {port} -c listen_addresses=localhost'", "start", ">/dev/null",
) # fmt: skip
subprocess.run(
["psql", "-h", "localhost", "-p", port, "-U", os.environ["POSTGRES_USER"], "-q", "-c",
f"CREATE DATABASE {os.environ['POSTGRES_DATABASE']}"],
check=True,
) # fmt: skip


def _query(sql: str) -> str:
result = subprocess.run(
["psql", "-h", "localhost", "-p", os.environ["POSTGRES_PORT"], "-U", os.environ["POSTGRES_USER"],
"-d", os.environ["POSTGRES_DATABASE"], "-tAc", sql],
check=True, capture_output=True, text=True,
) # fmt: skip
return result.stdout.strip()


def vector_count(layer: str, database: str) -> int:
"""Rows of ``database`` in the vector store ``layer``; the store creates a layer's table with its first rows."""
if _query(f"SELECT to_regclass('vdb.{layer}') IS NOT NULL") != "t":
return 0
return int(_query(f"SELECT count(*) FROM vdb.{layer} WHERE database_name = '{database}'") or 0)


def build(databases: list[Path]) -> None:
"""Ingest, describe, embed and compile each database. Auto Ontology must be importable and configured."""
from auto_ontology.catalog import ingest_catalog
from auto_ontology.connectors.registry import create_connector
from auto_ontology.dal.datasources import apply_metadata_batch, fetch_tables_and_columns_by_node_ids
from auto_ontology.semantic.compile import run_semantic_compilation
from auto_ontology.utils import get_embed_params
from auto_ontology.utils.embedding import batch_embed
from auto_ontology.utils.embedding_rows import CatalogEmbeddingRowsOp
from auto_ontology.vdb import get_data_vdb
from auto_ontology.vdb.records import write_embedded_rows

for database in databases:
name = database.parent.name
LOG.info("Ingesting %s", name)
connector = create_connector(f"sqlite:///{database}")
try:
tables, _ = ingest_catalog(connector)
finally:
connector.close()
descriptions = column_descriptions(database.parent / "database_description")
apply_metadata_batch(name, [], descriptions)
LOG.info("Described %d columns of %s", len(descriptions), name)
# Embed the catalog as described, not as ingested.
table_ids = [str(value) for value in tables["id"].dropna()]
tables, columns, _ = fetch_tables_and_columns_by_node_ids(table_ids)
embedded = batch_embed(CatalogEmbeddingRowsOp(database_name=name)((tables, columns)), get_embed_params())
store = get_data_vdb(database_name=name, reset=True)
written = write_embedded_rows(store, embedded) if len(embedded) else 0
LOG.info("Embedded %d catalog rows of %s", written, name)
LOG.info("Compiling the semantic layer of %s", name)
run_semantic_compilation(name)


def main(argv: list[str]) -> int:
if len(argv) != 1:
print(__doc__, file=sys.stderr)
return 2
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s")
databases = sorted(DATABASES_DIR.glob("*/*.sqlite"))
if not databases:
raise SystemExit(f"No BIRD databases under {DATABASES_DIR}")
connections = ",".join(f"sqlite:///{database}" for database in databases)
# The keys, then this image's catalog and databases over anything the file says about them.
image = {key: value for key, value in os.environ.items() if key.startswith("POSTGRES_")}
os.environ.update(read_env_file(Path(argv[0])))
os.environ.update(image)
os.environ["CONNECTION_STRINGS"] = connections

_start_postgres()
subprocess.run(
[str(AUTO_ONTOLOGY_DIR / ".venv/bin/alembic"), "upgrade", "head"], cwd=AUTO_ONTOLOGY_DIR, check=True
)
failed = _FailedModelCalls()
logging.getLogger().addHandler(failed)
build(databases)

problems = [f"a model call failed: {message}" for message in failed.messages[:5]]
for database in databases:
name = database.parent.name
for layer in ("data_objects_layer", "semantic_layer"):
if vector_count(layer, name) == 0:
problems.append(f"{name} has no vectors in vdb.{layer}")
if problems:
raise SystemExit("The catalog is incomplete; check the keys and rebuild:\n- " + "\n- ".join(problems))

from auto_ontology.utils.model_config import resolve

settings = [f"CONNECTION_STRINGS={connections}"] + [
f"{role}_{field}={resolve(role, field)}" for role in ("EMBED", "RERANK") for field in ("ENDPOINT", "MODEL")
]
(CATALOG_DIR / ".env").write_text("\n".join(settings) + "\n")
_postgres("pg_ctl", "-D", os.environ["PGDATA"], "-w", "stop", ">/dev/null")
return 0


if __name__ == "__main__":
raise SystemExit(main(sys.argv[1:]))
40 changes: 40 additions & 0 deletions benchmarks/birdbench/auto_ontology/build_image.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#!/usr/bin/env bash
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Build the Auto Ontology agent's sandbox image for BIRD dev: its databases and Auto Ontology's catalog of them.
#
# benchmarks/birdbench/auto_ontology/build_image.sh MODELS_ENV [TAG]
#
# Run it from Gym's environment (it downloads BIRD dev with bird_sql's setup when missing). MODELS_ENV is a KEY=VALUE
# file with the model keys the catalog build uses: DEFAULT_MODELS_API_KEY, and EMBED_* or *_MODEL settings where they
# differ from Auto Ontology's defaults. It reaches the build as a BuildKit secret, so no layer holds it. TAG defaults
# to nemo-gym/bird-catalog:dev, the image the Auto Ontology agent's config names.
#
# The build calls the embedding and compile models for all 11 databases; on Claude Opus 5.5 it takes about a quarter
# of an hour.
set -euo pipefail

# The one Auto Ontology that Gym runs: its main (a630ebd), NVIDIA/auto-ontology#543 (the openai version nemo-gym pins)
# and the ReAct loop's extra tools (NVIDIA/auto-ontology#559), which BIRD-Interact's ask_user needs. Move it to
# NVIDIA/auto-ontology once both merge.
AUTO_ONTOLOGY_REPOSITORY=https://github.com/ffrujeri/auto-ontology.git
AUTO_ONTOLOGY_COMMIT=6137967703a550a2c88860af22e9dc11fc49a64d

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "$HERE/../../.." && pwd)"
MODELS_ENV="${1:?usage: build_image.sh MODELS_ENV [TAG]}"
TAG="${2:-nemo-gym/bird-catalog:dev}"
[[ -f "$MODELS_ENV" ]] || { echo "build_image.sh: no $MODELS_ENV" >&2; exit 1; }

DATABASES="$(cd "$ROOT" && python -c 'from resources_servers.bird_sql.setup_bird_sql import ensure_bird_sql; print(ensure_bird_sql())')"
DATABASES="$(realpath "$DATABASES")"

DOCKER_BUILDKIT=1 docker build \
--file "$HERE/Dockerfile" \
--build-context bird="$DATABASES" \
--build-arg AUTO_ONTOLOGY_REPOSITORY="$AUTO_ONTOLOGY_REPOSITORY" \
--build-arg AUTO_ONTOLOGY_COMMIT="$AUTO_ONTOLOGY_COMMIT" \
--secret "id=models,src=$MODELS_ENV" \
--tag "$TAG" \
"$ROOT"
10 changes: 10 additions & 0 deletions benchmarks/birdbench/auto_ontology/start-catalog
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/usr/bin/env bash
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Start the catalog's Postgres and wait until it takes connections. The Auto Ontology agent runs this at each
# episode's seed, since the sandbox keeps its own entrypoint.
set -euo pipefail
su postgres -c "pg_ctl -D '$PGDATA' -l '$PGDATA/server.log' -w -t 120 \
-o '-p $POSTGRES_PORT -c listen_addresses=localhost' start" >/dev/null
pg_isready -h localhost -p "$POSTGRES_PORT" -t 60
38 changes: 38 additions & 0 deletions benchmarks/birdbench/tests/test_auto_ontology.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""The BIRD catalog image's inputs: Gym's pin of Auto Ontology and BIRD's descriptions."""

import re
from pathlib import Path

from benchmarks.birdbench.auto_ontology.build_catalog import column_descriptions


AUTO_ONTOLOGY_DIR = Path(__file__).resolve().parents[1] / "auto_ontology"


def test_auto_ontology_is_pinned_to_a_full_commit():
build_image = (AUTO_ONTOLOGY_DIR / "build_image.sh").read_text()
repository = re.search(r"^AUTO_ONTOLOGY_REPOSITORY=(\S+)$", build_image, re.MULTILINE)
commit = re.search(r"^AUTO_ONTOLOGY_COMMIT=(\S+)$", build_image, re.MULTILINE)
assert repository is not None and repository.group(1).startswith("https://github.com/")
assert commit is not None and re.fullmatch(r"[0-9a-f]{40}", commit.group(1))


def test_bird_descriptions_join_the_column_and_value_descriptions(tmp_path):
(tmp_path / "frpm.csv").write_text(
"original_column_name,column_name,column_description,data_format,value_description\n"
"CDSCode,CDS code,California school code,text,not useful\n"
'Charter School (Y/N), ,Charter school,integer,"0: no\n1: yes"\n'
",,orphan,,\n"
)
(tmp_path / "schools.csv").write_bytes(
"original_column_name,column_name,column_description,data_format,value_description\n"
"City,,Ciudad — city,text,\n".encode("cp1252")
)

assert column_descriptions(tmp_path) == [
{"table_name": "frpm", "column_name": "CDSCode", "description": "California school code"},
{"table_name": "frpm", "column_name": "Charter School (Y/N)", "description": "Charter school, 0: no 1: yes"},
{"table_name": "schools", "column_name": "City", "description": "Ciudad — city"},
]
Loading