Repository navigation
Conversation
Contributor
Author
|
/ok to test 2047c14 |
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-6271-2047c144e138.docs.buildwithfern.com/nemo/gym |
This was referenced Nov 17, 2026
…t Server, on BIRD dev Auto Ontology's main answers a question with one ReAct episode (react.loop.run_episode): its model searches Auto Ontology's catalog, runs SQL and submits a query. The agent calls run_episode as Auto Ontology's own ReAct step does, with auto-ontology-eval's settings for a graded benchmark, and answers with the submitted query in a fenced sql block. - The task (question, evidence, database) comes in the seed's agent context. birdbench's prepare.py writes it with +prepare_script_args.agent_context=true, to its own file, and benchmarks/birdbench/auto_ontology.yaml pairs the agent with bird_sql through single_agent_turn. - The episode's model is Auto Ontology's reasoning client on the policy's Model Server. Its calls go through Gym's ServerClient on the activation's rollout route, with the invocation ID, and report their usage (model_calls.py). - Auto Ontology is pinned to its main (a630ebd) plus the commit of NVIDIA/auto-ontology#543 that accepts the openai version Gym pins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
ffrujeri
force-pushed
the
ffrujeri/auto-ontology-react
branch
from
November 17, 2026 02:04
2047c14 to
1bcd49f
Compare
ffrujeri
changed the base branch from
main
to
ffrujeri/birdbench-single-agent-turn
November 17, 2026 02:04
Contributor
Author
|
/ok to test 1bcd49f |
Contributor
Author
|
Closing: the Auto Ontology Agent Server moves to NVIDIA/auto-ontology-eval, next to the ingestion code and annotations that build the catalog Auto Ontology's ReAct results used. There it runs Auto Ontology in a sandbox per episode, from an image whose catalog the benchmark setup builds. #6274 (birdbench on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #6274, which moves birdbench to
single_agent_turnwith a typed taskset (stack: #6274 → this PR).What changed
A new Agent Server,
responses_api_agents/auto_ontology_agent, runs Auto Ontology's ReAct text-to-SQL agent, and its first pairing is BIRD dev.mainanswers a question with one ReAct episode (react.loop.run_episode): its model searches Auto Ontology's catalog (retrieve), runs SQL on the source database (run_sql) and submits a query (submit). The agent callsrun_episodeas Auto Ontology's own ReAct step does, with the settings auto-ontology-eval uses for a graded benchmark (SQL answers only, the question as asked, the evidence apart), in a worker thread capped bymax_concurrent_episodes. It answers with the submitted query in a fencedsqlblock, or names the end reason when there is none; the response's metadata gives the end reason and the tool-call count. A resent activation gets the same reply.agent_context(question,database,evidence). The agent refuses a seed without it, or with a sandbox, a required tool grant or continuation.get_llm_client) on the policy's Model Server. An httpx transport sends each call through Gym'sServerClienton the activation's rollout route, with the invocation ID asx-session-id, and sums the calls' usage (model_calls.py, from feat(auto_ontology_agent): attribute Auto Ontology's model calls to their rollouts #5648).benchmarks/birdbench/prepare.pytakesagent_context=trueand writes rows that also carry the task in their agent context, to their own file (simple_agentrefuses a seed with an agent context).benchmarks/birdbench/auto_ontology.yamlputs the agent insimple_agent's place on birdbench'ssingle_agent_turnserver (tasksetbirdbench, from refactor(birdbench): run BIRD dev on single_agent_turn with a typed taskset #6274), withbird_sqlgrading and no tool grants.auto-ontology-serverat Auto Ontology'smain(a630ebd) plus the commit of Allow openai 2.45 and later, so the package installs next to nemo-gym's main NVIDIA/auto-ontology#543 that acceptsopenai2.45 (Gym pins it exactly; Auto Ontology pinned 2.44.0). It moves toNVIDIA/auto-ontologyonce docs: remove trl docs #543 merges.scripts/component-lock-policy.jsonkeepsalembicandsqlalchemyfor this agent;AGENT_SESSION_AGENT_TYPESlists it.Why
Auto Ontology's ReAct results (BIRD dev's first 200 tasks, BEAVER) come from auto-ontology-eval, which runs the agent inside a Resources Server's
verify()and discards the policy's own reply. There, the model calls that count never reach Gym's Model Server, so no rollout records them and they cannot train a policy. Here the agent is an Agent Server, its model calls go through the Model Server per rollout, andbird_sqlgrades its query like any other agent's. Tracking: GYMINT-753.This is one of two stacks: this one on
mainfor BIRD dev, and the BIRD-Interact stack, whose top PR, #5633, adds the c-Interact mode (Auto Ontology's interactive ask/submit mode) to this agent. #5633 carries this PR's commit until this one merges.Validation
pytest responses_api_agents/auto_ontology_agent/tests(component venv)run_episode,ToolContextand the result types have the names the runtime usespytest benchmarks/birdbench/testspytest tests/unit_tests/config/test_global_config.py tests/unit_tests/toolingscripts/component_locks.pygym env resolveofbenchmarks/birdbench/auto_ontology.yamlbirdbenchtaskset routes to thesingle_agent_turnserver, which drivesauto_ontology_agentwith no tool grants, on the agent-context datasetAutoOntologyRuntime, with a scripted chat model and a SQLite databasesource_database_url, the prompt holds the question and its evidence, the tools areretrieve,run_sqlandsubmit;run_sqlandsubmitthen need the catalog (POSTGRES_*), which this check has notpre-commit run --fileson the changed filesRollout evidence
None yet, so this stays a draft. A run needs Auto Ontology's catalog with BIRD dev's 11 databases ingested and compiled, and an embedding endpoint (the doc's results use
nemotron-3-embed-1b).run_sqlandsubmitcheck queries against the catalog, so no episode completes without it.Compatibility
New agent;
prepare.py's default output is unchanged.