MMLUSR tasks are broken

[christinaexyou]

Evaluations cannot be ran on MMLUSR tasks because the dataset names do not correspond to the dataset schema on HuggingFace.

Suppose a user wants to run evals on a specific subtask, say, mmlusr_answer_only_anatomy:

python3 -m lm_eval --model dummy --tasks mmlusr_answer_only_anatomy --limit 1

The above command returns the following error:

...
  File "/Users/chrxu/Documents/lm-evaluation-harness/lm_eval/api/task.py", line 822, in __init__
    self.download(self.config.dataset_kwargs)
  File "/Users/chrxu/Documents/lm-evaluation-harness/lm_eval/api/task.py", line 929, in download
    self.dataset = datasets.load_dataset(
  File "/Users/chrxu/Documents/lm-evaluation-harness/.venv/lib/python3.10/site-packages/datasets/load.py", line 2062, in load_dataset
    builder_instance = load_dataset_builder(
  File "/Users/chrxu/Documents/lm-evaluation-harness/.venv/lib/python3.10/site-packages/datasets/load.py", line 1819, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/Users/chrxu/Documents/lm-evaluation-harness/.venv/lib/python3.10/site-packages/datasets/builder.py", line 343, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/Users/chrxu/Documents/lm-evaluation-harness/.venv/lib/python3.10/site-packages/datasets/builder.py", line 570, in _create_builder_config
    raise ValueError(
ValueError: BuilderConfig 'mmlusr_answer_only_anatomy' not found. Available: ['answer_only', 'question_only', 'question_and_answer']

Happy to work on this issue and I already have a draft fix here: christinaexyou@998c7ae

[baberabb]

Hi! Thanks for identifying that issue. And yes, will appreciate a PR!