Skip to content

regression

regression

Per-cluster regression detection for the benchmark gate.

After an edit is applied, the gate compares the before and after BenchmarkSnapshots. If any cluster's score dropped by more than max_regression, the edit is rejected.

See spec §7.3.

Classes

RegressionResult dataclass 

RegressionResult(has_regression: bool, regressed_clusters: dict[str, float] = dict())

Result of a regression check.

Attributes
regressed_clusters class-attribute instance-attribute 
regressed_clusters: dict[str, float] = field(default_factory=dict)

Cluster id → negative delta for clusters that regressed beyond threshold.

Functions

regression_check 

regression_check(before: BenchmarkSnapshot, after: BenchmarkSnapshot, max_regression: float = 0.05) -> RegressionResult

Check if any cluster regressed beyond the threshold.

PARAMETER DESCRIPTION
before

Benchmark snapshot before the edit.

TYPE: BenchmarkSnapshot

after

Benchmark snapshot after the edit.

TYPE: BenchmarkSnapshot

max_regression

Maximum allowed per-cluster score drop (default 0.05).

TYPE: float DEFAULT: 0.05

RETURNS DESCRIPTION
RegressionResult

has_regression is True if any cluster dropped more than max_regression. regressed_clusters maps cluster ids to their negative deltas.
Source code in src/openjarvis/learning/distillation/gate/regression.py 
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
def regression_check(
    before: BenchmarkSnapshot,
    after: BenchmarkSnapshot,
    max_regression: float = 0.05,
) -> RegressionResult:
    """Check if any cluster regressed beyond the threshold.

    Parameters
    ----------
    before :
        Benchmark snapshot before the edit.
    after :
        Benchmark snapshot after the edit.
    max_regression :
        Maximum allowed per-cluster score drop (default 0.05).

    Returns
    -------
    RegressionResult
        ``has_regression`` is True if any cluster dropped more than
        ``max_regression``. ``regressed_clusters`` maps cluster ids
        to their negative deltas.
    """
    regressed: dict[str, float] = {}

    for cluster_id, before_score in before.cluster_scores.items():
        after_score = after.cluster_scores.get(cluster_id, 0.0)
        delta = after_score - before_score
        if delta < -max_regression:
            regressed[cluster_id] = delta

    return RegressionResult(
        has_regression=bool(regressed),
        regressed_clusters=regressed,
    )