Skip to content

paperarena_judge

paperarena_judge

Scorer for PaperArena: MC exact match + LLM judge for CA/OA.

Classes

PaperArenaScorer 

PaperArenaScorer(judge_backend: InferenceBackend, judge_model: str)

Bases: LLMJudgeScorer

Score PaperArena: MC via letter extraction, CA/OA via LLM judge.

Source code in src/openjarvis/evals/core/scorer.py 
31
32
33
def __init__(self, judge_backend: InferenceBackend, judge_model: str) -> None:
    self._judge_backend = judge_backend
    self._judge_model = judge_model