Skip to content

terminalbench_judge

terminalbench_judge

TerminalBench scorer — LLM-as-judge for terminal task evaluation.

Compares predicted terminal output / commands against the expected answer using an LLM judge.

Classes

TerminalBenchScorer 

TerminalBenchScorer(judge_backend: InferenceBackend, judge_model: str)

Bases: LLMJudgeScorer

LLM-as-judge evaluation for TerminalBench terminal tasks.

Source code in src/openjarvis/evals/core/scorer.py 
31
32
33
def __init__(self, judge_backend: InferenceBackend, judge_model: str) -> None:
    self._judge_backend = judge_backend
    self._judge_model = judge_model