coding_assistant scorer — test-based evaluation of bug fixes.
Extracts fixed code from model output, runs the test suite, and computes:
- fix_rate: fraction of originally-failing tests now passing
- regressions: count of originally-passing tests that broke
Classes
CodingAssistantScorer(judge_backend=None, judge_model: str = '')
Bases: Scorer
Score coding assistant bug fixes by test execution.
Source code in src/openjarvis/evals/scorers/coding_assistant.py
| def __init__(self, judge_backend=None, judge_model: str = "") -> None:
pass
|