Repository navigation
Score BEAVER with the full official ex_acc rules in Gym; retire the shard scripts - #45
Merged
Merged
Conversation
The Gym verifier already compared BEAVER results with the official rule. This adds the rest of what the official evaluator does, so the Gym number is comparable to the BEAVER leaderboard: - 10-second query timeout for beaverbench, whatever sql_execution_timeout_s is set to. - Every query that yields no rows is an empty result and is scored: a failed or timed-out prediction, a missing prediction (empty model answer, agent crash), and a failed gold query. Missing predictions keep their no_model_output / unknown_error code, so the health gate still sees an outage. - Accuracy without empty-gold questions: each rollout records gold_result, the metrics carry nonempty_gold/pass@1[...], and compare_arms.py prints a matching row. Gold failures scored as empty are counted in health/gold_failure_scored_as_empty_count. execute_and_compare now takes pred_sql=None for "no prediction" and replaces gold_is_runnable. Other datasets keep the stricter handling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Official BEAVER ex_acc now runs inside the Gym benchmark, so the sharded eval_chatbot pipeline was only left to feed the LLM judge, which BEAVER runs no longer use. Removes run_beaver_shards.sh, resume_beaver.sh, run_beaver_eval_only.sh and, under scripts/, resume_beaver_ranges.sh, run_eval_shard.py, merge_eval_shards.py and backfill_gold_answers.py. The BEAVER dataset README now documents the Gym benchmark run and its scoring; main.py and the main README no longer point at the shard scripts. The LLM judge, judge/beaver.py (the reference the Gym rule is tested against) and eval_chatbot stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
liofnv
approved these changes
Oct 14, 2026
yuvalshkolar
added a commit
that referenced
this pull request
Oct 14, 2026
Resolves the conflict in auto_ontology_sql/app.py's agent-crash path: main (#45) now scores it through _without_prediction (BEAVER's official rules), and this branch records the crash's token usage there. Keep both: record usage first, since _without_prediction responds through _response, which reads carried at call time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Gym verifier already used BEAVER's official row comparison. This PR adds the rest of the official
ex_accevaluator, so BEAVER numbers from the Gym benchmark are comparable to the BEAVER leaderboard. With that in place, the shardedeval_chatbotpipeline was only feeding the LLM judge, which BEAVER runs no longer use, so its scripts are removed.Official BEAVER scoring (
beaverbenchonly)sql_execution_timeout_s(QUERY_TIMEOUTSinexec_match.py).ERRORS_AS_EMPTY): failed or timed-out predictions, missing predictions (empty model answer, agent crash) and failed gold queries.no_model_output/unknown_errorcode, so the health counters andcompare_arms.py's broken-run check still flag an outage.health/gold_failure_scored_as_empty_count.gold_result(nonempty_gold/empty_gold/failed_gold). The metrics carrynonempty_gold/pass@1[avg-of-k]/accuracy, andcompare_arms.pyprints anonempty goldrow.Other datasets are unchanged: a failed prediction is a miss and a failed gold query drops the task. One exception: an empty answer whose gold query is broken is still dropped, while an agent crash keeps
unknown_error.execute_and_comparenow takespred_sql=Noneto mean "no prediction", which replacesgold_is_runnable. Both verifiers score missing predictions through it.Retired
run_beaver_shards.sh,resume_beaver.sh,run_beaver_eval_only.sh,scripts/resume_beaver_ranges.sh,scripts/run_eval_shard.py,scripts/merge_eval_shards.py,scripts/backfill_gold_answers.py.Kept: the LLM judge (
ontology-sql-eval),judge/beaver.py(the reference the Gym rule is tested against) andeval_chatbot.Docs
datasets/beaverbench/README.mdnow documents the Gym benchmark run and the official scoring. It previously described the old eval-then-judge flow.resources_servers/README.mddocuments the BEAVER rules.main.pyand the main README no longer reference the shard scripts.Testing
pytest: 67 passed. New cases cover every combination of gold and prediction outcomes for BEAVER and BIRD, failed-query scoring checked against the officialcompare_results, and the timeout per dataset.compare_arms.pyoutput with fake rollouts.nonempty_goldis the number to compare.🤖 Generated with Claude Code