Pipe through evalspecs w/ git revision, packages, and solver/model/task args for leaderboard

[jbragg]

Prepare for https://github.com/allenai/astabench-issues/issues/277 and similar use

The agenteval.json file gets a new field that looks like the following (haven't tested publishing / updating the hf schema):

  "eval_spec": {
      "solver": "path/to/solvers/perplexity_base.py@perplexity_solver",
      "solver_args": "{}",
      "model": "perplexity/sonar-deep-research",
      "model_args": "{}",
      "task_args": "{\"litqa_args\": [], \"litqa_kwargs\": {}}",
      "revision": {
        "type": "git",
        "origin": "[email protected]:org/repo.git",
        "commit": "abcd123"
      },
      "packages": "{\"inspect_ai\": \"0.3.106\"}"
    }

I would be open to pruning back the piped solver/model and their args, but figured those might be useful as part of a pointer to the agent source / run command. (For now, in the leaderboard viewer I have only explicitly constructed a source url pointing to the repo at the git revision, which doesn't include that information).

UPDATE: I also piped through task_args and packages to get better visibility into whether submissions have been run in a standardized/correct way.

[regan-huff]

Do you know if lb view still works if you mix files with and without these fields in the same dataset? Do we need to keep results from before and after the change separated?

[jbragg]

@regan-huff My understanding is that if we update the schema with only new fields as in this PR, then we don't need to keep files with/out those fields separate, and the ones with missing fields should get None. We could test by updating the schema to include these new fields and see if nothing breaks.

[regan-huff]

ok, I did a test and what I observed is that I can mix the data, but I had to delete the existing readme to get the schema to update with upload_summary_to_hf. Not surprisingly, the new data would not load with the old schema.

(test-agent-eval) 16:01:10 ~/src/agent-eval lb-view-source-url $ agenteval lb view --repo-id allenai/asta-bench-test-results --config 1.0.0-dev1 --split validation
README.md: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.34k/2.34k [00:00<00:00, 2.52MB/s]
(…)aloney-sandwich_2025-07-17T23-01-09.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 38.5k/38.5k [00:00<00:00, 25.7MB/s]
Generating validation split: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 188.49 examples/s]
                              id                                             Agent Agent description User/organization Submission date Logs                                                     Source Openness Agent tooling                                                            LLM base  Overall Overall cost lit cost data cost code cost discovery cost
2025-07-17 23:01:09.064808+00:00                                  baloney-sandwich                               Regan      2025-07-17 None https://github.com/allenai/asta-bench-private/tree/2ac7433   Closed      Standard                                           [gemini/gemini-2.0-flash] 0.115244         None     None      None      None           None
2025-06-09 21:03:14.658669+00:00           Basic ReAct (task tools, report editor)                            miked-ai      2025-06-09 None                                                       None     None          None      [gpt-4o-2024-08-06, meta-llama/Llama-4-Scout-17B-16E-Instruct] 0.012516         None     None      None      None           None
2025-06-09 21:05:49.252775+00:00 Basic ReAct (task tools, report editor w/ submit)                            miked-ai      2025-06-09 None                                                       None     None          None      [gpt-4o-2024-08-06, meta-llama/Llama-4-Scout-17B-16E-Instruct] 0.010538         None     None      None      None           None
2025-06-09 21:04:02.400490+00:00              Basic ReAct (task tools, no editors)                            miked-ai      2025-06-09 None                                                       None     None          None      [gpt-4o-2024-08-06, meta-llama/Llama-4-Scout-17B-16E-Instruct] 0.008756         None     None      None      None           None
2025-06-09 21:02:33.921472+00:00            Basic ReAct (task tools, table editor)                            miked-ai      2025-06-09 None                                                       None     None          None [gpt-4o-mini-2024-07-18, meta-llama/Llama-4-Scout-17B-16E-Instruct] 0.004971         None     None      None      None           None
2025-06-09 21:04:59.642327+00:00  Basic ReAct (task tools, table editor w/ submit)                            miked-ai      2025-06-09 None                                                       None     None          None [gpt-4o-mini-2024-07-18, meta-llama/Llama-4-Scout-17B-16E-Instruct] 0.001100         None     None      None      None           None

[regan-huff]

I'm seeing the init of LeaderboardViewer (which calls EvalResult.model_validate) fail when pointed at existing data when EvalSpec gets moved under TaskResult so I think if the leaderboard picked up this change right now it would break.

What's the rationale for changing the structure of EvalResult?

[jbragg]

I pushed a fix for that error (should have tested, thanks).

What's the rationale for changing the structure of EvalResult?

This PR seeks to serialize the eval_spec information somewhere in EvalResult so that the lb viewer can construct the details about how the agent was run like the source code. eval_specs used to be a field on the main EvalResult, which was previously excluded from serialization. I moved it inside each TaskResult now, since there is a 1:1 mapping (this seemed cleaner than the original implementation in this PR which kept eval_specs as a field on EvalResult, which would need associating with task names and joining).

[regan-huff]

View at:
https://pypi.org/project/agent-eval/0.1.18/
Successfully published version 0.1.18