Skip to content

feat(auto_ontology_agent): run Auto Ontology's ReAct agent as an Agent Server, on BIRD dev - #6271

Closed
ffrujeri wants to merge 1 commit into
ffrujeri/birdbench-single-agent-turnfrom
ffrujeri/auto-ontology-react
Closed

ffrujeri wants to merge 1 commit into
ffrujeri/birdbench-single-agent-turnfrom
ffrujeri/auto-ontology-react

Conversation

@ffrujeri

@ffrujeri ffrujeri commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #6274, which moves birdbench to single_agent_turn with a typed taskset (stack: #6274 → this PR).

What changed

A new Agent Server, responses_api_agents/auto_ontology_agent, runs Auto Ontology's ReAct text-to-SQL agent, and its first pairing is BIRD dev.

  • The agent. Auto Ontology's main answers a question with one ReAct episode (react.loop.run_episode): its model searches Auto Ontology's catalog (retrieve), runs SQL on the source database (run_sql) and submits a query (submit). The agent calls run_episode as Auto Ontology's own ReAct step does, with the settings auto-ontology-eval uses for a graded benchmark (SQL answers only, the question as asked, the evidence apart), in a worker thread capped by max_concurrent_episodes. It answers with the submitted query in a fenced sql block, or names the end reason when there is none; the response's metadata gives the end reason and the tool-call count. A resent activation gets the same reply.
  • The task comes in the seed's agent_context (question, database, evidence). The agent refuses a seed without it, or with a sandbox, a required tool grant or continuation.
  • Model calls. The episode's model is Auto Ontology's reasoning client (get_llm_client) on the policy's Model Server. An httpx transport sends each call through Gym's ServerClient on the activation's rollout route, with the invocation ID as x-session-id, and sums the calls' usage (model_calls.py, from feat(auto_ontology_agent): attribute Auto Ontology's model calls to their rollouts #5648).
  • BIRD dev pairing. benchmarks/birdbench/prepare.py takes agent_context=true and writes rows that also carry the task in their agent context, to their own file (simple_agent refuses a seed with an agent context). benchmarks/birdbench/auto_ontology.yaml puts the agent in simple_agent's place on birdbench's single_agent_turn server (taskset birdbench, from refactor(birdbench): run BIRD dev on single_agent_turn with a typed taskset #6274), with bird_sql grading and no tool grants.
  • Pin. auto-ontology-server at Auto Ontology's main (a630ebd) plus the commit of Allow openai 2.45 and later, so the package installs next to nemo-gym's main NVIDIA/auto-ontology#543 that accepts openai 2.45 (Gym pins it exactly; Auto Ontology pinned 2.44.0). It moves to NVIDIA/auto-ontology once docs: remove trl docs #543 merges. scripts/component-lock-policy.json keeps alembic and sqlalchemy for this agent; AGENT_SESSION_AGENT_TYPES lists it.

Why

Auto Ontology's ReAct results (BIRD dev's first 200 tasks, BEAVER) come from auto-ontology-eval, which runs the agent inside a Resources Server's verify() and discards the policy's own reply. There, the model calls that count never reach Gym's Model Server, so no rollout records them and they cannot train a policy. Here the agent is an Agent Server, its model calls go through the Model Server per rollout, and bird_sql grades its query like any other agent's. Tracking: GYMINT-753.

This is one of two stacks: this one on main for BIRD dev, and the BIRD-Interact stack, whose top PR, #5633, adds the c-Interact mode (Auto Ontology's interactive ask/submit mode) to this agent. #5633 carries this PR's commit until this one merges.

Validation

Check Result
pytest responses_api_agents/auto_ontology_agent/tests (component venv) 36 passed: seed refusals, the episode and its answer, a resent activation, a failed episode in the observations, the runtime against stubbed Auto Ontology functions, the model-call transport (routes, usage, timeouts, unrouted calls), and two checks against Auto Ontology at the pinned commit: its reasoning client sends through the agent's HTTP client, and run_episode, ToolContext and the result types have the names the runtime uses
pytest benchmarks/birdbench/tests 76 passed, 3 skipped, including the agent-context rows
pytest tests/unit_tests/config/test_global_config.py tests/unit_tests/tooling passed
scripts/component_locks.py 317 projects, 0 failed
gym env resolve of benchmarks/birdbench/auto_ontology.yaml resolves: the birdbench taskset routes to the single_agent_turn server, which drives auto_ontology_agent with no tool grants, on the agent-context dataset
Auto Ontology's real ReAct loop under AutoOntologyRuntime, with a scripted chat model and a SQLite database the connector comes from source_database_url, the prompt holds the question and its evidence, the tools are retrieve, run_sql and submit; run_sql and submit then need the catalog (POSTGRES_*), which this check has not
pre-commit run --files on the changed files passed

Rollout evidence

None yet, so this stays a draft. A run needs Auto Ontology's catalog with BIRD dev's 11 databases ingested and compiled, and an embedding endpoint (the doc's results use nemotron-3-embed-1b). run_sql and submit check queries against the catalog, so no episode completes without it.

Compatibility

New agent; prepare.py's default output is unchanged.

@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ffrujeri ffrujeri added area:agent Agent harnesses and Responses API agent behavior feature New capabilities, enhancements, or enablement work labels Nov 17, 2026
@ffrujeri

Copy link
Copy Markdown
Contributor Author

/ok to test 2047c14

@github-actions

Copy link
Copy Markdown
Contributor

…t Server, on BIRD dev

Auto Ontology's main answers a question with one ReAct episode
(react.loop.run_episode): its model searches Auto Ontology's catalog, runs SQL
and submits a query. The agent calls run_episode as Auto Ontology's own ReAct
step does, with auto-ontology-eval's settings for a graded benchmark, and
answers with the submitted query in a fenced sql block.

- The task (question, evidence, database) comes in the seed's agent context.
  birdbench's prepare.py writes it with +prepare_script_args.agent_context=true,
  to its own file, and benchmarks/birdbench/auto_ontology.yaml pairs the agent
  with bird_sql through single_agent_turn.
- The episode's model is Auto Ontology's reasoning client on the policy's Model
  Server. Its calls go through Gym's ServerClient on the activation's rollout
  route, with the invocation ID, and report their usage (model_calls.py).
- Auto Ontology is pinned to its main (a630ebd) plus the commit of
  NVIDIA/auto-ontology#543 that accepts the openai version Gym pins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
@ffrujeri
ffrujeri force-pushed the ffrujeri/auto-ontology-react branch from 2047c14 to 1bcd49f Compare November 17, 2026 02:04
@ffrujeri
ffrujeri changed the base branch from main to ffrujeri/birdbench-single-agent-turn November 17, 2026 02:04
@ffrujeri
ffrujeri added this pull request to stack #6275 November 17, 2026 02:05
@ffrujeri

Copy link
Copy Markdown
Contributor Author

/ok to test 1bcd49f

@ffrujeri
ffrujeri removed this pull request from stack #6275 November 17, 2026 03:10
@ffrujeri

Copy link
Copy Markdown
Contributor Author

Closing: the Auto Ontology Agent Server moves to NVIDIA/auto-ontology-eval, next to the ingestion code and annotations that build the catalog Auto Ontology's ReAct results used. There it runs Auto Ontology in a sandbox per episode, from an image whose catalog the benchmark setup builds. #6274 (birdbench on single_agent_turn) stays: the agent there grades BIRD dev through it.

@ffrujeri ffrujeri closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior feature New capabilities, enhancements, or enablement work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant