Skip to content

Commit aed44c8

Browse files
committed
scripts(eval): give examples to numerical word answers
Some models need extra help.
1 parent cddd6d5 commit aed44c8

1 file changed

Lines changed: 4 additions & 3 deletions

File tree

scripts/evals/eval.py

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -293,12 +293,13 @@ async def run_eval_pipeline(
293293
"You should only call the eval function if:\n"
294294
"- The user explicitly attempts to answer the question, AND\n"
295295
f"- Their answer can be cleanly evaluated using: {eval_config.eval}\n"
296-
"Ignore greetings, comments, non-answers, or requests for clarification."
296+
"Ignore greetings, comments, non-answers, or requests for clarification.\n"
297+
"Numerical word answers are allowed (e.g., 'five' is the same as '5').\n"
297298
)
298299
if eval_config.eval_speaks_first:
299-
system_prompt = f"You are an evaluation agent, be extremly brief. Numerical word answers are allowed. You will start the conversation by saying: '{example_prompt}'. {common_system_prompt}"
300+
system_prompt = f"You are an evaluation agent, be extremly brief. You will start the conversation by saying: '{example_prompt}'. {common_system_prompt}"
300301
else:
301-
system_prompt = f"You are an evaluation agent, be extremly brief. Numerical word answers are allowed. First, ask one question: {example_prompt}. {common_system_prompt}"
302+
system_prompt = f"You are an evaluation agent, be extremly brief. First, ask one question: {example_prompt}. {common_system_prompt}"
302303

303304
messages = [
304305
{

0 commit comments

Comments
 (0)