Skip to content

doc_qa

doc_qa

doc_qa scorer — fact match, citation check, and checklist evaluation.

Tier 1 (fact match): Check if model answer contains each required fact. Tier 1 (citation check): Verify document citations match expected sources. Tier 2 (checklist): Binary checklist for grounding and accuracy.

Score: (facts_found/total) * 0.5 + (citations_correct/citations_made) * 0.3 + checklist * 0.2

Classes

DocQAScorer 

DocQAScorer(judge_backend=None, judge_model: str = '')

Bases: Scorer

Score document QA output by fact coverage, citations, and quality.

Source code in src/openjarvis/evals/scorers/doc_qa.py 
100
101
102
103
104
def __init__(
    self, judge_backend=None, judge_model: str = "",
) -> None:
    self._judge_backend = judge_backend
    self._judge_model = judge_model

Functions