Skip to content

coding_task

coding_task

Coding task scorer — test pass rate + structural validation.

Extracts the function/class from model output and runs test cases to determine correctness.

Classes

CodingTaskScorer 

CodingTaskScorer(judge_backend=None, judge_model: str = '')

Bases: Scorer

Score coding tasks by running test cases against model output.

Source code in src/openjarvis/evals/scorers/coding_task.py 
103
104
105
106
def __init__(self, judge_backend=None, judge_model: str = "") -> None:
    # Accept same constructor args as LLMJudgeScorer for compatibility
    # but don't need the judge for this scorer
    pass