Skip to content

Diagram the evaluation flow and where it is going - #71

Draft
cafzal wants to merge 4 commits into
mainfrom
docs/evaluate-diagram
Draft

cafzal wants to merge 4 commits into
mainfrom
docs/evaluate-diagram

Conversation

@cafzal

@cafzal cafzal commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Docs only. Adds a Now Mermaid diagram to the README (how a run works today) and a new docs/architecture.md with Now plus Next (Auto-Ontology's ReAct tools on NeMo Gym), linked from the README, matching the Now/Next labels in NVIDIA/auto-ontology#489.

  • Now — Evaluate: seed a benchmark database, ingest, compile, then run two arms. The control arm gets the schema dump; the treatment arm is the same model through Auto-Ontology's agent. A deterministic verifier compares results with the gold SQL, then the report. Traced to main.py stages and resources_servers/README.md.
    • Draft corrections: the scorer is a deterministic verifier, not a judge (ontology_sql_eval/gym/exec_match.py). The benchmark list matches the README table: BIRD, BEAVER, FDABench-Lite, WWI.
  • Method sentence: every reported difference is paired (same model, identical questions, with vs. without the ontology, one verifier). BIRD-Interact is the stated exception; it runs Auto-Ontology alone.
  • Choosing the answer architecture: there's no runtime switch. The treatment arm runs whatever the pinned auto-ontology-server revision in pyproject.toml has. main pins dd78d06, the fixed LangGraph chain. The branch feat/react-ontology-agent_clean pins the ReAct library branch. The README says how to repin, uv sync, and rebuild with GYM_REBUILD_VENV=1.
  • Next — ReAct tools on NeMo Gym (planned, marked not built): Gym's policy model calls Auto-Ontology's ReAct tools (retrieve, run_sql, submit) directly, and the verifier's score becomes the reward. Nothing in either repo implements this yet; the ReAct agent's limits are written to be usable as an RL environment (react/config.py, react/loop.py on auto-ontology #488). Today the treatment arm's Gym policy call is made and discarded.
  • Tokens per question: TODO placeholder, no numbers.
  • WWI seed script: links scripts/seed/seed_wwi.py and notes that Auto-Ontology's Quickstart uses it.

Companion PR: NVIDIA/auto-ontology#489.

Validation

  • All Mermaid blocks validated with npx -y @mermaid-js/mermaid-cli and rendered in the default and dark themes.
  • The GitHub preview check is still to do: github.com was in a partial outage when this was opened.

The README links Auto-Ontology's docs/architecture.md, which exists once NVIDIA/auto-ontology#489 merges.

Add a Mermaid diagram of today's paired run (seed, ingest, compile, two
arms, deterministic verifier, report) and of the planned ReAct-on-Gym
setup, marked as not built. State the paired-difference method, how to
choose the answer architecture through the pinned Auto-Ontology revision,
link the WWI seed script Auto-Ontology's Quickstart uses, and leave a TODO
for tokens-per-question reporting.

Signed-off-by: Cameron Afzal <cafzal@nvidia.com>
Signed-off-by: Cameron Afzal <cafzal@nvidia.com>
The README keeps the Now diagram and links to the planned ReAct-on-Gym flow,
which moves to docs/architecture.md beside it.

Signed-off-by: Cameron Afzal <cafzal@nvidia.com>
Signed-off-by: Cameron Afzal <cafzal@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant