Skip to content

feat(birdbench): build the Auto Ontology catalog image in Gym, pinned once - #6288

Draft
ffrujeri wants to merge 1 commit into
ffrujeri/birdbench-single-agent-turnfrom
ffrujeri/birdbench-auto-ontology-catalog
Draft

ffrujeri wants to merge 1 commit into
ffrujeri/birdbench-single-agent-turnfrom
ffrujeri/birdbench-auto-ontology-catalog

Conversation

@ffrujeri

@ffrujeri ffrujeri commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What

The image that the Auto Ontology agent (#6289) starts each episode's sandbox from, built in Gym from a pinned Auto Ontology:

  • The image (benchmarks/birdbench/auto_ontology/Dockerfile, build_image.sh). It holds Auto Ontology at the commit build_image.sh pins, installed from its own repository and lock file, the BIRD dev databases and a catalog Postgres.
  • The catalog (build_catalog.py). The build:
    1. migrates the catalog with Auto Ontology's Alembic migrations;
    2. ingests each database's schema;
    3. writes BIRD's column descriptions through Auto Ontology's apply_metadata_batch;
    4. embeds the described catalog;
    5. compiles the semantic layer.
  • start-catalog starts the catalog's Postgres on the port an episode gives it.

Why

The agent runs Auto Ontology's own loop against its own catalog, so Gym needs a catalog it can rebuild at a version it controls. Auto Ontology has no input for BIRD's descriptions, which a SQLite database does not carry, so the build writes them through Auto Ontology's own API. This gives the same catalog content that auto-ontology-eval uses.

The catalog stays inside the agent harness's sandbox. The other approach, the bird_sql Resources Server serving Auto Ontology's tools to any harness (#6285), is closed for now. This PR no longer carries its rows, prompt or prepare project.

Notes

Stacked on #6274.

Validation

  • pytest benchmarks/birdbench/tests: passed.
  • pre-commit on the stack's changes: passed.
  • build_image.sh built nemo-gym/bird-catalog:dev in 13 minutes with Claude Opus 5.5, at the earlier pin (a630ebd + docs: remove trl docs #543). All 11 databases were described, embedded and compiled, with no failed model call. The semantic layer has the same row count per database as auto-ontology-eval's image.

Rollout evidence

See #6289: the agent runs on this image. The 200-task BIRD dev results are there.

@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ffrujeri
ffrujeri added this pull request to stack #6286 November 17, 2026 07:32
@ffrujeri ffrujeri added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers feature New capabilities, enhancements, or enablement work labels Nov 17, 2026
@ffrujeri

Copy link
Copy Markdown
Contributor Author

/ok to test cfc89ee

@ffrujeri
ffrujeri force-pushed the ffrujeri/birdbench-auto-ontology-catalog branch from cfc89ee to 7dc1527 Compare November 17, 2026 13:37
@ffrujeri

Copy link
Copy Markdown
Contributor Author

/ok to test 7dc1527

@ffrujeri

Copy link
Copy Markdown
Contributor Author

/ok to test a54b110

… once

The image the Auto Ontology agent starts each episode's sandbox from: Auto Ontology
at the commit build_image.sh pins, installed from its own repository and lock file,
the BIRD dev databases and a catalog Postgres with each database ingested,
described with BIRD's column descriptions, embedded and compiled. The model keys
reach the catalog build as a BuildKit secret only. The build fails when a model
call failed or a database has no vectors, which Auto Ontology logs and goes on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
@ffrujeri
ffrujeri force-pushed the ffrujeri/birdbench-auto-ontology-catalog branch from a54b110 to 1d339d0 Compare November 17, 2026 22:48
@ffrujeri
ffrujeri removed this pull request from stack #6286 November 17, 2026 22:48
@ffrujeri
ffrujeri changed the base branch from ffrujeri/birdbench-catalog-tools to ffrujeri/birdbench-single-agent-turn November 17, 2026 22:48
@ffrujeri
ffrujeri added this pull request to stack #6319 November 17, 2026 22:48
@ffrujeri

Copy link
Copy Markdown
Contributor Author

/ok to test 1d339d0

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers feature New capabilities, enhancements, or enablement work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant