Skip to content

simpleqa

simpleqa

SimpleQA dataset provider (basicv8vc/SimpleQA).

Short-answer factual QA benchmark for evaluating factual accuracy.

Classes

SimpleQADataset 

SimpleQADataset()

Bases: DatasetProvider

SimpleQA short-answer factual QA benchmark.

Source code in src/openjarvis/evals/datasets/simpleqa.py 
31
32
def __init__(self) -> None:
    self._records: List[EvalRecord] = []