browser_assistant

browser_assistant ¶

browser_assistant scorer — exact match, semantic checklist, and source check.

Tier 1 (exact match): For exact facts, normalize and match numbers/names/versions. Tier 2 (semantic checklist): For semantic facts, binary yes/no via LLM judge. Tier 2 (quality checklist): No fabricated numbers, sources cited, answers question. Tier 1 (sources): Check if any URL or explicit reference is mentioned.

Score: (exact/total_exact) * 0.35 + (semantic/total_semantic) * 0.35 + quality_checklist * 0.15 + sources_cited * 0.15

Classes¶

BrowserAssistantScorer ¶

BrowserAssistantScorer(judge_backend=None, judge_model: str = '')

Bases: Scorer

Score web research output by fact accuracy and sourcing.

Source code in src/openjarvis/evals/scorers/browser_assistant.py

def __init__(
    self, judge_backend=None, judge_model: str = "",
) -> None:
    self._judge_backend = judge_backend
    self._judge_model = judge_model

browser_assistant