Skip to content

external_adapter

external_adapter

Adapt external agent-dataset records into TraceStore rows.

LLM-guided spec search reads student traces from a SQLite TraceStore via its search/get tools. This adapter reuses that pipeline for external agent corpora (e.g. ADP, ToolOrchestra, GeneralThoughts): each EvalRecord from a provider becomes a synthetic Trace so the teacher proposer can reason over it without a live student run on the corpus task.

These synthetic traces carry feedback=0.5 by default (no ground truth). Tool calls are empty. The source_name tag in each trace's metadata is how multi-source corpora are filtered downstream.

Classes

Functions

write_external_records_as_traces 

write_external_records_as_traces(store: TraceStore, records: Iterable[EvalRecord], *, source_name: str, feedback_score: float = 0.5) -> int

Write each EvalRecord to store as a synthetic Trace.

The LLM-guided spec search proposer reads from the TraceStore via its search/get tools; this lets the diagnose phase substitute live student traces with records from pre-existing agent-task corpora (ADP / ToolOrchestra / GeneralThoughts).

Args: store: Destination TraceStore. records: Iterable of EvalRecord (e.g. from a dataset provider). source_name: Label recorded in each trace's metadata["source"] so multi-source setups can filter later. feedback_score: Value for Trace.feedback (0..1). Defaults to 0.5 because external records carry no ground-truth outcome.

Returns: Number of traces written.

Source code in src/openjarvis/learning/distillation/external_adapter.py 
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
def write_external_records_as_traces(
    store: TraceStore,
    records: Iterable[EvalRecord],
    *,
    source_name: str,
    feedback_score: float = 0.5,
) -> int:
    """Write each EvalRecord to `store` as a synthetic Trace.

    The LLM-guided spec search proposer reads from the TraceStore via
    its search/get tools; this lets the diagnose phase substitute live
    student traces with records from pre-existing agent-task corpora
    (ADP / ToolOrchestra / GeneralThoughts).

    Args:
        store: Destination TraceStore.
        records: Iterable of EvalRecord (e.g. from a dataset provider).
        source_name: Label recorded in each trace's metadata["source"]
            so multi-source setups can filter later.
        feedback_score: Value for Trace.feedback (0..1). Defaults to 0.5
            because external records carry no ground-truth outcome.

    Returns:
        Number of traces written.
    """
    written = 0
    for rec in records:
        meta = {"source": source_name, "record_id": rec.record_id}
        if rec.metadata:
            meta.update(rec.metadata)
        trace = Trace(
            trace_id=f"ext-{source_name}-{rec.record_id}",
            query=rec.problem,
            agent="external",
            model="external",
            engine="external",
            steps=[],
            result=rec.reference or "",
            outcome=None,
            feedback=feedback_score,
            metadata=meta,
        )
        store.save(trace)
        written += 1
    return written