workarena

workarena

WorkArena++ enterprise workflow benchmark on ServiceNow.

Faithful integration of the original browsergym-workarena package. Tasks are Python classes that run against a live ServiceNow instance via BrowserGym / Playwright — NOT a static JSON dataset.

L1 = 33 atomic tasks (ICML 2024) L2/L3 = 682 composite tasks (NeurIPS 2024)

Source: https://github.com/ServiceNow/WorkArena Requires: pip install browsergym-workarena playwright==1.44.0

Classes

WorkArenaDataset

WorkArenaDataset(level: str = 'l2', n_seed_l1: int = 10, meta_seed: int = 42, headless: bool = True)

Bases: DatasetProvider

WorkArena++ benchmark using the native browsergym-workarena package.

Tasks are enumerated from the installed browsergym-workarena package exactly as in the original benchmark. Each task class is instantiated with a seed by BrowserGym at evaluation time. Scoring uses the task's native validate() method against the live ServiceNow instance — no LLM judge or text matching.

Source code in src/openjarvis/evals/datasets/workarena.py

def __init__(
    self,
    level: str = "l2",
    n_seed_l1: int = 10,
    meta_seed: int = 42,
    headless: bool = True,
) -> None:
    if level not in _VALID_LEVELS and level != "all":
        raise ValueError(
            f"Unknown WorkArena level: {level!r}. "
            f"Choose from: {list(_VALID_LEVELS)} or 'all'"
        )
    self._level = level
    self._n_seed_l1 = n_seed_l1
    self._meta_seed = meta_seed
    self._headless = headless
    self._records: List[EvalRecord] = []
    self._episodes: List[List[EvalRecord]] = []

Functions

create_task_env

create_task_env(record: EvalRecord)

Return a WorkArenaTaskEnv for the given record.

Source code in src/openjarvis/evals/datasets/workarena.py

def create_task_env(self, record: EvalRecord):
    """Return a WorkArenaTaskEnv for the given record."""
    try:
        from openjarvis.evals.execution.workarena_env import (
            WorkArenaTaskEnv,
        )

        return WorkArenaTaskEnv(record.metadata)
    except ImportError:
        return None

verify_requirements

verify_requirements() -> List[str]

Check that all prerequisites for WorkArena evaluation are met.

Source code in src/openjarvis/evals/datasets/workarena.py

def verify_requirements(self) -> List[str]:
    """Check that all prerequisites for WorkArena evaluation are met."""
    issues: List[str] = []

    if not _HAS_WORKARENA:
        issues.append(
            "browsergym-workarena not installed. "
            "Install with: pip install browsergym-workarena"
        )

    try:
        import playwright  # noqa: F401
    except ImportError:
        issues.append(
            "playwright not installed. "
            "Install with: pip install playwright==1.44.0 && playwright install"
        )

    snow_configured = bool(
        os.environ.get("SNOW_INSTANCE_URL")
        or os.environ.get("HUGGINGFACE_HUB_TOKEN")
    )
    if not snow_configured:
        issues.append(
            "ServiceNow instance not configured. "
            "Set SNOW_INSTANCE_URL or authenticate with HuggingFace: "
            "huggingface-cli login (requires gated access to "
            "ServiceNow/WorkArena-Instances)"
        )

    return issues