Search results

📋 Documentation Enhancement Suggestion

Status: Open.

#462 In tatsu-lab/alpaca_eval;

· croviatrust opened on Mar 11, 2026

Unexpected keyword argument "leaderboard_mode" in print_leaderboard() when running alpaca_eval make_leaderboard

Status: Open.

#460 In tatsu-lab/alpaca_eval;

· takahashihiroshi opened on Jan 5, 2026

Error when using DeepSeek as a judge in alpaca_eval

Status: Open.

#455 In tatsu-lab/alpaca_eval;

· AqualCross opened on Sep 29, 2025

dataset scripts no longer supported

Status: Open.

#454 In tatsu-lab/alpaca_eval;

· sunnyych opened on Aug 28, 2025

Cannot get length controlled win rate

Status: Open.

#453 In tatsu-lab/alpaca_eval;

· FocusLiwen opened on Jun 20, 2025

What models are used when GPT-4-1106-preview is inaccessible?

Status: Open.

#452 In tatsu-lab/alpaca_eval;

· fulQuan opened on Jun 10, 2025

How to determine the possibility of another LLM as a judge model?

Status: Open.

#450 In tatsu-lab/alpaca_eval;

· AmberXu98 opened on Jun 6, 2025

When fn_completions is set to huggingface_local_completions, the model will be reloaded for each chunk causing OOM error

Status: Open.

#449 In tatsu-lab/alpaca_eval;

· arushibear opened on Jun 3, 2025

Why is the output “[\n {‘model’: ‘model_1’, ‘rank’: 1},\n {‘model’: ‘model_2’, ‘rank’: 2}\n]” and the final result is “preference”: 2.0, when these two do not correspond? What is the reason for this?

Status: Open.

#447 In tatsu-lab/alpaca_eval;

· FHhui opened on May 12, 2025

"You must manually fix the score pair" Error While Using Custom Evaluator

Status: Open.

#446 In tatsu-lab/alpaca_eval;

· tommycwh opened on May 8, 2025

ValueError: cannot reindex on an axis with duplicate labels

Status: Open.

#445 In tatsu-lab/alpaca_eval;

· Liesy opened on May 1, 2025

when model output is like deepseek-r1 ,wondering if it is better to omit content inside <think>

Status: Open.

#442 In tatsu-lab/alpaca_eval;

· jijivski opened on Mar 22, 2025