Skip to content

livecodebench

livecodebench

LiveCodeBench dataset provider — competitive programming code generation.

Loads problems from the LiveCodeBench HuggingFace dataset (livecodebench/code_generation_lite) for evaluating code generation capability.

Reference: https://livecodebench.github.io/ https://github.com/LiveCodeBench/LiveCodeBench

Classes

LiveCodeBenchDataset 

LiveCodeBenchDataset()

Bases: DatasetProvider

LiveCodeBench competitive programming benchmark.

Loads problems from the LiveCodeBench HuggingFace dataset for evaluating code generation capability on fresh competitive programming problems from LeetCode, AtCoder, and CodeForces.

Source code in src/openjarvis/evals/datasets/livecodebench.py 
80
81
def __init__(self) -> None:
    self._records: List[EvalRecord] = []