scorer

scorer

Abstract base classes for scoring.

Classes

Scorer

Bases: ABC

Base class for all scorers.

Functions

score abstractmethod

score(record: EvalRecord, model_answer: str) -> Tuple[Optional[bool], Dict[str, Any]]

Score a model answer against the reference.

Returns (is_correct, metadata) where is_correct may be None if scoring could not be determined.

Source code in src/openjarvis/evals/core/scorer.py

@abstractmethod
def score(
    self, record: EvalRecord, model_answer: str,
) -> Tuple[Optional[bool], Dict[str, Any]]:
    """Score a model answer against the reference.

    Returns (is_correct, metadata) where is_correct may be None
    if scoring could not be determined.
    """

LLMJudgeScorer

LLMJudgeScorer(judge_backend: InferenceBackend, judge_model: str)

Bases: Scorer

Base for scorers that need an LLM to judge answers.

Source code in src/openjarvis/evals/core/scorer.py

def __init__(self, judge_backend: InferenceBackend, judge_model: str) -> None:
    self._judge_backend = judge_backend
    self._judge_model = judge_model