tracker

tracker

ResultTracker ABC for external experiment tracking.

Classes

ResultTracker

Bases: ABC

Abstract base class for experiment result trackers.

Lifecycle: on_run_start -> on_result (per sample) -> on_summary -> on_run_end.

Functions

on_run_start abstractmethod

on_run_start(config: RunConfig) -> None

Called once before evaluation begins.

Source code in src/openjarvis/evals/core/tracker.py

@abstractmethod
def on_run_start(self, config: RunConfig) -> None:
    """Called once before evaluation begins."""

on_result abstractmethod

on_result(result: EvalResult, config: RunConfig) -> None

Called after each sample is evaluated.

Source code in src/openjarvis/evals/core/tracker.py

@abstractmethod
def on_result(self, result: EvalResult, config: RunConfig) -> None:
    """Called after each sample is evaluated."""

on_summary abstractmethod

on_summary(summary: RunSummary) -> None

Called after all samples are evaluated with aggregate stats.

Source code in src/openjarvis/evals/core/tracker.py

@abstractmethod
def on_summary(self, summary: RunSummary) -> None:
    """Called after all samples are evaluated with aggregate stats."""

on_run_end abstractmethod

on_run_end() -> None

Called at the very end of a run for cleanup.

Source code in src/openjarvis/evals/core/tracker.py

@abstractmethod
def on_run_end(self) -> None:
    """Called at the very end of a run for cleanup."""