Skip to content

environment

environment

Environment provider ABC for benchmarks requiring external environments.

Classes

TaskEnvironmentError

Bases: RuntimeError

A task execution environment failed to start or operate.

Raised when infrastructure backing a task (Docker container, docker compose project, tmux session, recording binaries, ...) breaks. This is a harness/environment failure, not a model failure: runners record it distinctly (QueryTrace.error_kind == "harness_error") so scoring can exclude the sample from resolve-rate instead of silently counting it as a model miss.

EnvironmentProvider

Bases: ABC

Manages an external environment for evaluation benchmarks.

Provides lifecycle management (setup/reset/teardown) and environment-state validation for benchmarks that need Docker containers, ServiceNow instances, or other live systems.

Functions
setup abstractmethod 
setup() -> Dict[str, Any]

Start the environment and return connection info.

Returns a dict with environment-specific connection details (e.g., URLs, ports, credentials).

Source code in src/openjarvis/evals/core/environment.py 
31
32
33
34
35
36
37
@abstractmethod
def setup(self) -> Dict[str, Any]:
    """Start the environment and return connection info.

    Returns a dict with environment-specific connection details
    (e.g., URLs, ports, credentials).
    """
reset abstractmethod 
reset() -> None

Reset environment state between tasks.

Called between records within an episode to restore the environment to a known state.

Source code in src/openjarvis/evals/core/environment.py 
39
40
41
42
43
44
45
@abstractmethod
def reset(self) -> None:
    """Reset environment state between tasks.

    Called between records within an episode to restore
    the environment to a known state.
    """
validate abstractmethod 
validate(record: EvalRecord) -> Tuple[bool, Dict[str, Any]]

Check environment state against expected outcome.

Args: record: The eval record containing the expected state in metadata.

Returns: (is_correct, metadata) where is_correct indicates whether the environment is in the expected state.

Source code in src/openjarvis/evals/core/environment.py 
47
48
49
50
51
52
53
54
55
56
57
58
59
60
@abstractmethod
def validate(
    self,
    record: EvalRecord,
) -> Tuple[bool, Dict[str, Any]]:
    """Check environment state against expected outcome.

    Args:
        record: The eval record containing the expected state in metadata.

    Returns:
        (is_correct, metadata) where is_correct indicates whether
        the environment is in the expected state.
    """
teardown abstractmethod 
teardown() -> None

Stop the environment and release resources.

Source code in src/openjarvis/evals/core/environment.py 
62
63
64
@abstractmethod
def teardown(self) -> None:
    """Stop the environment and release resources."""