Skip to content

liveresearch

liveresearch

LiveResearchBench dataset provider — deep research benchmark.

Clones the deep_research_bench repo at runtime and parses query + criteria JSONL files into EvalRecords for use with AgenticRunner.

Reference: https://github.com/Ayanami0730/deep_research_bench Paper: https://arxiv.org/abs/2510.14240

Classes

LiveResearchBenchDataset 

LiveResearchBenchDataset(path: Optional[str] = None)

Bases: DatasetProvider

LiveResearchBench — deep research with 100 expert-curated tasks.

Clones Ayanami0730/deep_research_bench from GitHub (or uses a local path) and parses query + criteria JSONL files into EvalRecords.

Source code in src/openjarvis/evals/datasets/liveresearch.py 
62
63
64
65
def __init__(self, path: Optional[str] = None) -> None:
    self._local_path = Path(path) if path else None
    self._repo_dir: Path = self._local_path or CACHE_DIR
    self._records: List[EvalRecord] = []