Skip to content

mmlu_pro

mmlu_pro

MMLU-Pro dataset provider (TIGER-Lab/MMLU-Pro).

Adapted from IPW's mmlu_pro.py dataset loader.

Classes

MMLUProDataset 

MMLUProDataset()

Bases: DatasetProvider

MMLU-Pro multiple-choice benchmark dataset.

Source code in src/openjarvis/evals/datasets/mmlu_pro.py 
32
33
def __init__(self) -> None:
    self._records: List[EvalRecord] = []