Skip to content

daily_digest

daily_digest

daily_digest scorer — phrase match, ordering, and checklist evaluation.

Tier 1 (phrase match): Check each must-mention item against model output. Tier 1 (ordering): Check that top-priority items appear in first half of response. Tier 2 (checklist): Binary checklist for structure, accuracy, and actionability.

Score: (items_mentioned/total) * 0.5 + ordering_score * 0.3 + checklist * 0.2

Classes

DailyDigestScorer 

DailyDigestScorer(judge_backend=None, judge_model: str = '')

Bases: Scorer

Score daily digest output by coverage, ordering, and quality.

Source code in src/openjarvis/evals/scorers/daily_digest.py 
80
81
82
83
84
def __init__(
    self, judge_backend=None, judge_model: str = "",
) -> None:
    self._judge_backend = judge_backend
    self._judge_model = judge_model

Functions