security_scanner

security_scanner ¶

security_scanner scorer — vulnerability detection evaluation.

Tier 1: Pattern-match model output against vulnerability manifest. Tier 2: Binary checklist for severity correctness and fix quality.

Score formula: (vulns_found/total) * 0.6 + (severity_correct/found) * 0.2 + max(0, 1 - FP/3) * 0.1 + checklist_score * 0.1

SecurityScannerScorer(judge_backend=None, judge_model: str = '')

Bases: Scorer

Score security scanner output by vulnerability detection accuracy.

Source code in src/openjarvis/evals/scorers/security_scanner.py

def __init__(self, judge_backend=None, judge_model: str = "") -> None:
    self._judge_backend = judge_backend
    self._judge_model = judge_model