Repository navigation
Conversation
Contributor
Author
|
/ok to test cfc89ee |
ffrujeri
force-pushed
the
ffrujeri/birdbench-auto-ontology-catalog
branch
from
November 17, 2026 13:37
cfc89ee to
7dc1527
Compare
Contributor
Author
|
/ok to test 7dc1527 |
ffrujeri
force-pushed
the
ffrujeri/birdbench-auto-ontology-catalog
branch
from
November 17, 2026 19:27
7dc1527 to
a54b110
Compare
Contributor
Author
|
/ok to test a54b110 |
… once The image the Auto Ontology agent starts each episode's sandbox from: Auto Ontology at the commit build_image.sh pins, installed from its own repository and lock file, the BIRD dev databases and a catalog Postgres with each database ingested, described with BIRD's column descriptions, embedded and compiled. The model keys reach the catalog build as a BuildKit secret only. The build fails when a model call failed or a database has no vectors, which Auto Ontology logs and goes on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
ffrujeri
force-pushed
the
ffrujeri/birdbench-auto-ontology-catalog
branch
from
November 17, 2026 22:48
a54b110 to
1d339d0
Compare
ffrujeri
changed the base branch from
ffrujeri/birdbench-catalog-tools
to
ffrujeri/birdbench-single-agent-turn
November 17, 2026 22:48
This was referenced Nov 17, 2026
Contributor
Author
|
/ok to test 1d339d0 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The image that the Auto Ontology agent (#6289) starts each episode's sandbox from, built in Gym from a pinned Auto Ontology:
benchmarks/birdbench/auto_ontology/Dockerfile,build_image.sh). It holds Auto Ontology at the commitbuild_image.shpins, installed from its own repository and lock file, the BIRD dev databases and a catalog Postgres.build_catalog.py). The build:apply_metadata_batch;start-catalogstarts the catalog's Postgres on the port an episode gives it.Why
The agent runs Auto Ontology's own loop against its own catalog, so Gym needs a catalog it can rebuild at a version it controls. Auto Ontology has no input for BIRD's descriptions, which a SQLite database does not carry, so the build writes them through Auto Ontology's own API. This gives the same catalog content that auto-ontology-eval uses.
The catalog stays inside the agent harness's sandbox. The other approach, the
bird_sqlResources Server serving Auto Ontology's tools to any harness (#6285), is closed for now. This PR no longer carries its rows, prompt or prepare project.Notes
build_image.sh: Auto Ontology's main (a630ebd), Allow openai 2.45 and later, so the package installs next to nemo-gym's main NVIDIA/auto-ontology#543 and Let a caller add tools to a ReAct episode NVIDIA/auto-ontology#559 (extra tools for a ReAct episode, which BIRD-Interact'sask_userneeds). For BIRD dev it runs the same Auto Ontology as the image measured below: docs: version switcher is leading to "Page Not Found" #559 changes nothing unless a caller adds tools. The pin moves to NVIDIA/auto-ontology once both merge.Stacked on #6274.
Validation
pytest benchmarks/birdbench/tests: passed.pre-commiton the stack's changes: passed.build_image.shbuiltnemo-gym/bird-catalog:devin 13 minutes with Claude Opus 5.5, at the earlier pin (a630ebd+ docs: remove trl docs #543). All 11 databases were described, embedded and compiled, with no failed model call. The semantic layer has the same row count per database as auto-ontology-eval's image.Rollout evidence
See #6289: the agent runs on this image. The 200-task BIRD dev results are there.