Skip to content

Score BEAVER with the full official ex_acc rules in Gym; retire the shard scripts - #45

Merged
yuvalshkolar merged 2 commits into
mainfrom
gym/beaver-exec-match
Oct 14, 2026
Merged

yuvalshkolar merged 2 commits into
mainfrom
gym/beaver-exec-match

Conversation

@yuvalshkolar

Copy link
Copy Markdown
Collaborator

Summary

The Gym verifier already used BEAVER's official row comparison. This PR adds the rest of the official ex_acc evaluator, so BEAVER numbers from the Gym benchmark are comparable to the BEAVER leaderboard. With that in place, the sharded eval_chatbot pipeline was only feeding the LLM judge, which BEAVER runs no longer use, so its scripts are removed.

Official BEAVER scoring (beaverbench only)

  • 10 s query timeout, overriding sql_execution_timeout_s (QUERY_TIMEOUTS in exec_match.py).
  • Every query that yields no rows counts as an empty result and is scored (ERRORS_AS_EMPTY): failed or timed-out predictions, missing predictions (empty model answer, agent crash) and failed gold queries.
    • A missing prediction against empty gold scores 1.0, as it does officially. It keeps its no_model_output / unknown_error code, so the health counters and compare_arms.py's broken-run check still flag an outage.
    • A failed gold query stays in the denominator instead of being dropped. These are counted in health/gold_failure_scored_as_empty_count.
  • Accuracy without empty-gold questions: each rollout records gold_result (nonempty_gold / empty_gold / failed_gold). The metrics carry nonempty_gold/pass@1[avg-of-k]/accuracy, and compare_arms.py prints a nonempty gold row.

Other datasets are unchanged: a failed prediction is a miss and a failed gold query drops the task. One exception: an empty answer whose gold query is broken is still dropped, while an agent crash keeps unknown_error.

execute_and_compare now takes pred_sql=None to mean "no prediction", which replaces gold_is_runnable. Both verifiers score missing predictions through it.

Retired

run_beaver_shards.sh, resume_beaver.sh, run_beaver_eval_only.sh, scripts/resume_beaver_ranges.sh, scripts/run_eval_shard.py, scripts/merge_eval_shards.py, scripts/backfill_gold_answers.py.

Kept: the LLM judge (ontology-sql-eval), judge/beaver.py (the reference the Gym rule is tested against) and eval_chatbot.

Docs

  • datasets/beaverbench/README.md now documents the Gym benchmark run and the official scoring. It previously described the old eval-then-judge flow.
  • resources_servers/README.md documents the BEAVER rules.
  • main.py and the main README no longer reference the shard scripts.

Testing

  • pytest: 67 passed. New cases cover every combination of gold and prediction outcomes for BEAVER and BIRD, failed-query scoring checked against the official compare_results, and the timeout per dataset.
  • Checked the verifier paths directly (no-output, agent crash, health counter) and compare_arms.py output with fake rollouts.
  • Not yet run: a real Gym benchmark against the BEAVER MySQL database. Expect the overall BEAVER number to rise compared with earlier Gym runs, because empty-gold questions now score the way they do officially. nonempty_gold is the number to compare.

🤖 Generated with Claude Code

yuvalshkolar and others added 2 commits October 13, 2026 15:03
The Gym verifier already compared BEAVER results with the official rule.
This adds the rest of what the official evaluator does, so the Gym number
is comparable to the BEAVER leaderboard:

- 10-second query timeout for beaverbench, whatever
  sql_execution_timeout_s is set to.
- Every query that yields no rows is an empty result and is scored: a
  failed or timed-out prediction, a missing prediction (empty model
  answer, agent crash), and a failed gold query. Missing predictions keep
  their no_model_output / unknown_error code, so the health gate still
  sees an outage.
- Accuracy without empty-gold questions: each rollout records
  gold_result, the metrics carry nonempty_gold/pass@1[...], and
  compare_arms.py prints a matching row. Gold failures scored as empty
  are counted in health/gold_failure_scored_as_empty_count.

execute_and_compare now takes pred_sql=None for "no prediction" and
replaces gold_is_runnable. Other datasets keep the stricter handling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Official BEAVER ex_acc now runs inside the Gym benchmark, so the sharded
eval_chatbot pipeline was only left to feed the LLM judge, which BEAVER
runs no longer use. Removes run_beaver_shards.sh, resume_beaver.sh,
run_beaver_eval_only.sh and, under scripts/, resume_beaver_ranges.sh,
run_eval_shard.py, merge_eval_shards.py and backfill_gold_answers.py.

The BEAVER dataset README now documents the Gym benchmark run and its
scoring; main.py and the main README no longer point at the shard
scripts. The LLM judge, judge/beaver.py (the reference the Gym rule is
tested against) and eval_chatbot stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yuvalshkolar
yuvalshkolar requested a review from liofnv October 14, 2026 15:10
@yuvalshkolar
yuvalshkolar merged commit da987be into main Oct 14, 2026
2 checks passed
yuvalshkolar added a commit that referenced this pull request Oct 14, 2026
Resolves the conflict in auto_ontology_sql/app.py's agent-crash path:
main (#45) now scores it through _without_prediction (BEAVER's official
rules), and this branch records the crash's token usage there. Keep both:
record usage first, since _without_prediction responds through
_response, which reads carried at call time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants