Skip to content

taubench

taubench

TauBench scorer — wraps tau2-bench's evaluation results.

Since TauBench runs its own simulation loop (agent + user simulator + tools + evaluation), the scorer simply reads the reward that was computed during task execution and stored in record.metadata.

Classes

TauBenchScorer 

TauBenchScorer(judge_backend: Any = None, judge_model: str = '')

Bases: Scorer

TauBench scorer — reads pre-computed rewards from tau2-bench.

The actual evaluation (DB state checks, action matching, communication checks, NL assertions) is done by tau2-bench's evaluator during simulation. This scorer extracts the result.

Source code in src/openjarvis/evals/scorers/taubench.py 
26
27
28
def __init__(self, judge_backend: Any = None, judge_model: str = "") -> None:
    self._judge_backend = judge_backend
    self._judge_model = judge_model