Repository navigation
Conversation
The Auto Ontology Agent Server needs Gym's agent sessions and sandbox library, which no Gym release has yet (0.7.0 predates them), so pin Gym's main at 5d6429f5 in pyproject.toml and in both servers' requirements. Gym's main makes Ray, wandb and mlflow optional extras, so the lock drops them; no server here enables Ray. Pin Auto Ontology to its main at a630ebd, its ReAct agent, plus the commit of NVIDIA/auto-ontology#543, which accepts the openai version Gym pins (2.45.0). Move to NVIDIA/auto-ontology once #543 merges. test_server_requirements_pin_the_parents_nemo_gym now accepts a full Git commit as an exact pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
…isode Run Auto Ontology's ReAct agent as a NeMo Gym Agent Server, on the policy model, so its rollouts compare with the schema-only arm's and can train. The treatment arm instead runs Auto Ontology in a Resources Server's verify() and discards the policy's output. - responses_api_agents/auto_ontology_agent: at the seed, start a sandbox from the agent image and its catalog Postgres; at the activation, run one ReAct episode there and answer with the submitted query in a sql block; at the close, stop the sandbox. The process imports only Gym. - ontology_sql_eval/gym/react_runner.py: the in-sandbox runner. It calls run_episode as Auto Ontology's ReactSQLAgent does (SQL answers only, evidence apart), with its reasoning client on the policy's Gym Model Server URL, on the rollout's route. - docker/auto_ontology_agent and scripts/build_agent_image.sh: the image, with BIRD dev and a catalog built by main.py --skip-eval. The keys reach that step as a BuildKit secret only. The build fails when a database has no vectors, since main.py logs a missing key and goes on. - The config pairs the agent with schema_only_sql through single_agent_turn, so both arms are graded by the same exec_match. - build_gym_tasks.py --arm react writes rows with the task in their agent_context and no schema dump. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
…n port A sandbox on Docker's bridge could not reach the policy's Model Server on a host whose firewall closes it to the bridge: every episode's first model call timed out, and the run scored only what the grader's SELECT 1 filler matched. The sandboxes now run on the host network. They share loopback, so the agent starts each sandbox's catalog on a random port, tries another when one is taken, and gives the episode its port. Also: - build-catalog fails when a model call fails after Auto Ontology's retries: a 429 burst during the compile otherwise left columns and joins undescribed without failing the build. - The image copies the scripts an episode runs after the catalog, so editing them no longer rebuilds it. Smoke: 5 of the first BIRD dev questions on Opus 5.5, 5 correct, each in 2 or 3 tool calls, every model call captured on its rollout's route. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Felipe Vieira Frujeri <ffrujeri@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a third arm,
react: Auto Ontology's ReAct agent as a NeMo Gym Agent Server, one sandbox per episode, on the policy model. The schema-only arm'sexec_matchgrades it, so it compares directly with the control.Why
The
auto_ontologyarm runs Auto Ontology inside the Resources Server'sverifyand ignores the policy's answer. Thereactarm runs the agent as Gym runs any agent: every model call goes through the policy's Model Server on the rollout's route, so Gym records it and a training run could use it.What changed
responses_api_agents/auto_ontology_agent/). The seed starts a sandbox from the agent image and starts its catalog. The activation runsontology_sql_eval/gym/react_runner.pyin the sandbox, which calls Auto Ontology'srun_episodeas itsReactSQLAgentdoes for a SQL answer. The reply is the submitted query in a ```sql block. The close stops the sandbox.scripts/build_gym_tasks.py --arm reactwrites rows whose agent context holds the question, evidence and database.single_agent_turnruns them.docker/auto_ontology_agent/,scripts/build_agent_image.sh). Auto Ontology at the pinned commit, the BIRD dev databases, and their catalog, built once at image build time: schema, BIRD's column descriptions, embeddings and the semantic layer. The build fails when a model call failed or a database has no vectors; Auto Ontology logs both and goes on.ffrujeri/auto-ontology@e774af9, which is Auto Ontology'smainata630ebd(the ReAct agent) plus Allow openai 2.45 and later, so the package installs next to nemo-gym's main auto-ontology#543 (accepts the openai version nemo-gym pins). Move to NVIDIA/auto-ontology once #543 merges.mainat5d6429f. The agent needs Gym's agent sessions and sandbox library, which 0.7.0 does not have. Move to the next release once it is out.Compatibility
auto_ontologyarm. Ata630ebd,stream_agent_responseruns the ReAct agent plus a format step, not the LangGraph agent. The lock drops LangGraph, MLflow and their dependencies with it. Results of this arm before and after this PR are not comparable.gym env testand the server venvs cannot resolvenemo-gym. The Agent Server README says how to install an editable Gym checkout at the pinned commit instead.Results
The first 200 BIRD dev tasks,
aws/anthropic/eccn-claude-opus-5-5, one run per arm, both graded byexec_match:schema_only(control)reactThe same arms on two more models:
openai/openai/eccn-gpt-6-astraschema_only(control)openai/openai/eccn-gpt-6-astrareactnvidia/nvidia/eccn-nemotron-3.5-lightningschema_only(control)nvidia/nvidia/eccn-nemotron-3.5-lightningreactOn both models the arms differ within noise (Astra 17 and 17 tasks, p = 1.0; Lightning 34 and 25, p ≈ 0.30).
NeMo Gym runs the same agent in NVIDIA-NeMo/Gym#6289 with its own catalog image and grader (
bird_sql): 79.5% with Opus. The two runs differ on 13 tasks (4 and 9), within noise (p ≈ 0.27).Two findings from these runs, not fixed here:
exec_matchputsSELECT 1in place of a missing prediction. It credits a missing answer whenever the gold result is a single1(tasks 0, 16 and 62 in this run).scripts/seed/seed_bird.pyparses BIRD's description CSVs fromraw.splitlines(), which drops the line breaks inside a quoted cell: a multi-line value description has its lines joined with nothing between them.Validation
uv run pytest: 102 passed.ruff check .andruff format --check .: clean.uv lock --check: the lock is current.