liveresearchbench

liveresearchbench ¶

LiveResearchBench dataset provider — Salesforce's checklist-based benchmark.

Loads Salesforce/LiveResearchBench from HuggingFace. Each task has a research question and a set of checklist items used for fine-grained, coverage-based evaluation.

Note: This is the actual LiveResearchBench by Salesforce (arxiv 2510.14240). The existing liveresearch module points at DeepResearchBench (Ayanami0730/deep_research_bench) despite its misleading class name.

Reference: https://github.com/SalesforceAIResearch/LiveResearchBench Paper: https://arxiv.org/abs/2510.14240 Dataset: https://huggingface.co/datasets/Salesforce/LiveResearchBench

Classes¶

LiveResearchBenchDataset ¶

LiveResearchBenchDataset(hf_config: Optional[str] = None, hf_split: Optional[str] = None)

Bases: DatasetProvider

LiveResearchBench — Salesforce's expert-curated deep research benchmark.

Loads tasks from HuggingFace with per-task checklists used for coverage-based evaluation. Tasks span 7 domains (Science/Tech, Business, Health, Law/Governance, Society/Culture, Education, Media).

Source code in src/openjarvis/evals/datasets/liveresearchbench.py

def __init__(
    self,
    hf_config: Optional[str] = None,
    hf_split: Optional[str] = None,
) -> None:
    self._hf_config = hf_config or DEFAULT_HF_CONFIG
    self._hf_split = hf_split or DEFAULT_HF_SPLIT
    self._records: List[EvalRecord] = []

liveresearchbench