terminalbench_v2_1

terminalbench_v2_1

TerminalBench V2.1 dataset provider.

Loads tasks from the terminal-bench-2.1 repo layout (ekellbuch/terminal-bench-2, branch terminal-bench-2.1). Each task lives in a top-level directory containing:

<task_name>/
  task.toml          # metadata + docker image + timeouts
  instruction.md     # the agent prompt
  environment/       # Dockerfile + supporting files (pre-built into task.toml's docker_image)
  solution/          # oracle solve.sh (not used by eval)
  tests/             # test.sh + test_outputs.py (pytest) used by the verifier

Reference: https://github.com/ekellbuch/terminal-bench-2/tree/terminal-bench-2.1

Classes

TerminalBenchV21Dataset

TerminalBenchV21Dataset(repo_url: str = _DEFAULT_REPO, branch: str = _DEFAULT_BRANCH, path: Optional[str] = None, task_ids: Optional[List[str]] = None)

Bases: DatasetProvider

TerminalBench V2.1 dataset (89 Docker-based terminal tasks).

Source code in src/openjarvis/evals/datasets/terminalbench_v2_1.py

def __init__(
    self,
    repo_url: str = _DEFAULT_REPO,
    branch: str = _DEFAULT_BRANCH,
    path: Optional[str] = None,
    task_ids: Optional[List[str]] = None,
) -> None:
    self._repo_url = repo_url
    self._branch = branch
    self._repo_dir: Path = Path(path) if path else _DEFAULT_CACHE
    self._task_ids = task_ids
    self._records: List[EvalRecord] = []

Functions

create_task_env

create_task_env(record)

Return a per-task Docker environment (context manager).

The runner enters this around the agent call so that tools like docker_shell_exec can target the running container.

Source code in src/openjarvis/evals/datasets/terminalbench_v2_1.py

def create_task_env(self, record):
    """Return a per-task Docker environment (context manager).

    The runner enters this around the agent call so that tools like
    ``docker_shell_exec`` can target the running container.
    """
    try:
        from openjarvis.evals.execution.terminalbench_v2_1_env import (
            TerminalBenchV21TaskEnv,
        )
    except ImportError:
        return None
    return TerminalBenchV21TaskEnv(record.metadata)

verify_requirements

verify_requirements() -> List[str]

Check runtime prerequisites (docker, git, tomllib).

Source code in src/openjarvis/evals/datasets/terminalbench_v2_1.py

def verify_requirements(self) -> List[str]:
    """Check runtime prerequisites (docker, git, tomllib)."""
    issues: List[str] = []
    if shutil.which("docker") is None:
        issues.append("docker not found in PATH (required to run TB v2.1 tasks)")
    if shutil.which("git") is None:
        issues.append("git not found in PATH (required to clone TB v2.1 repo)")
    try:
        import tomllib  # noqa: F401
    except ImportError:
        try:
            import tomli  # noqa: F401
        except ImportError:
            issues.append(
                "tomllib (3.11+) or tomli not available — cannot parse task.toml"
            )
    return issues