WebArena

Paid Self-hosted

What WebArena is

WebArena is a free, open-source, self-hosted web environment and benchmark for building autonomous agents. It recreates websites in four popular categories with functionality and data modeled on real-world equivalents, alongside separate tool and knowledge sites. Agents receive high-level natural-language tasks and must turn them into concrete web interactions. Annotated programs validate functional task completion, while evaluations can compare information-seeking answers with annotated references and inspect intermediate states. Page observations can be represented as a URL with a screenshot, HTML DOM tree, or accessibility tree. Demo environments include a social forum, online shopping, content management, collaborative software development, maps, a wiki, calculator, and scratchpad. Docker resources and a pre-installed Amazon Machine Image support hosting. Repository setup requires Python 3.10 or newer, project requirements, and Playwright. Correct evaluation requires users to set up their own websites; the repository also requires an OpenAI API key and instructs resetting after 812 examples.

Who it suits

WebArena suits researchers and developers building or evaluating web agents who can self-host environments and configure the required software.

What it does well

Free and open source
Programmatic task validation
Multiple observation formats
Docker and Amazon Machine Image options
Eight varied demo site types

What to know first

Self-hosting and setup are required
Correct evaluation needs your own websites
Evaluation requires an OpenAI API key
Environment reset required after 812 examples

Verdict

WebArena provides a structured environment for testing agents against realistic web tasks and intermediate states. Its self-hosting, Python, Playwright, and API-key requirements make setup part of using the benchmark.

Facts from WebArena's own pages

Demo environments: The demos include social forum, online shopping, content management, collaborative software development, map, wiki, calculator, and scratchpad sites. · webarena.dev, 5 Oct 2026
Embedded resources: The environment includes tools and knowledge resources as independent websites. · webarena.dev, 5 Oct 2026
Environment: It recreates websites from four popular categories with functionality and data modeled on real-world equivalents, and includes tools and knowledge resources as separate websites. · webarena.dev, 11 Oct 2026
Evaluation: Evaluation checks information-seeking answers against annotated references and can programmatically check intermediate execution states. · webarena.dev, 11 Oct 2026
Evaluation limit: The repository instructs users to reset the environment after evaluating its 812 examples. · github.com, 11 Oct 2026
Evaluation requirement: The repository says to set up your own WebArena websites for correct evaluation and describes the demo sites as browsing-only. · github.com, 5 Oct 2026
Example tasks: Shown tasks include creating a repository and checking the status and arrival time of an order. · webarena.dev, 5 Oct 2026
External model key: The repository’s evaluation instructions require exporting a valid OpenAI API key before launching evaluation. · github.com, 5 Oct 2026
Included sites: The demos include a social forum, online shopping, content management, collaborative software development, a map, a wiki, a calculator, and a scratchpad. · webarena.dev, 11 Oct 2026
License: The WebArena code repository identifies its license as Apache-2.0. · github.com, 11 Oct 2026
Licensing: The WebArena website states that it is licensed under Creative Commons Attribution-ShareAlike 4.0 International. · webarena.dev, 5 Oct 2026
Model use: The repository's example evaluation command uses GPT-3.5 Turbo and requires an OpenAI API key. · github.com, 11 Oct 2026
Observation formats: Agents can observe a page through its URL and content, represented as a screenshot, HTML DOM tree, or accessibility tree. · webarena.dev, 5 Oct 2026
Observation options: Observations include a page URL and content, represented as a screenshot, HTML DOM tree, or accessibility tree. · webarena.dev, 11 Oct 2026
Purpose: WebArena is a standalone, self-hostable web environment and benchmark for building autonomous agents. · webarena.dev, 11 Oct 2026
Related framework: The repository describes AgentLab as supporting parallel experiments through BrowserGym, benchmark integration including VisualWebArena, unified leaderboard reporting, and improved handling of environment edge cases. · github.com, 11 Oct 2026
Research affiliations: The page lists Carnegie Mellon University and Inspired Cognition as author affiliations. · webarena.dev, 5 Oct 2026
Research provenance: The project page lists Carnegie Mellon University and Inspired Cognition affiliations for the paper's authors. · webarena.dev, 11 Oct 2026
Self-hosting: The project provides Docker resources and a pre-installed Amazon Machine Image for hosting the benchmark websites. · github.com, 11 Oct 2026
Setup: The canonical repository README specifies Python 3.10 or newer and installation of its requirements and Playwright browser. · github.com, 5 Oct 2026
Setup requirement: The repository installation instructions specify Python 3.10 or later and Playwright. · github.com, 11 Oct 2026
Simulated websites: It creates websites across four popular categories with functionality and data modeled on real-world equivalents. · webarena.dev, 5 Oct 2026
Task input: The benchmark evaluates agents that turn high-level natural-language commands into web interactions. · webarena.dev, 5 Oct 2026
Tasks: The benchmark evaluates agents that translate realistic natural-language instructions into concrete web interactions. · webarena.dev, 11 Oct 2026

Where WebArena stands on Freedom251

LLM Evaluation Tools#27 of 29