Skip to content

security_scanner

security_scanner

security_scanner dataset — 30 vulnerable code projects for security auditing.

Each task presents a small Python project with planted vulnerabilities. The agent must identify vulnerabilities, classify severity, and recommend fixes.

Difficulty tiers: - easy (10): single obvious vulnerability (hardcoded secret, SQL injection) - medium (10): multiple vulnerabilities requiring deeper analysis - hard (10): subtle vulnerabilities mixed with safe patterns (false positive traps)

Classes

SecurityScannerDataset 

SecurityScannerDataset()

Bases: DatasetProvider

30 vulnerable code projects for security auditing evaluation.

Source code in src/openjarvis/evals/datasets/security_scanner.py 
687
688
def __init__(self) -> None:
    self._records: List[EvalRecord] = []