AgentBench

Free plan APILinuxSelf-hosted

What AgentBench is

AgentBench is a free, self-hosted benchmark for evaluating large language models as agents across different environments. Its original release covers operating systems, databases, knowledge graphs, a digital card game, lateral-thinking puzzles, household tasks, web shopping, and web browsing. The repository identifies AgentBench FC as the current version, using function-calling prompts and integrating with AgentRL. FC provides containerized deployments for alfworld, dbbench, knowledgegraph, os_interaction, and webshop, with a documented one-command Docker Compose setup. The original quick start uses an OpenAI API key and gives gpt-3.5-turbo-0613 as an example agent; configuration can select another agent. Local models served through FastChat are supported, while API-only models can connect through an Agent Client interface. New environments can be added by implementing the Task class and configuring YAML. Python 3.9 is recommended, and the original quick start requires Docker. The project uses an Apache-2.0 license.

Who it suits

It suits developers and researchers evaluating agent models across varied tasks. Self-hosting, Docker deployment, YAML configuration, and extensibility fit technical users.

What it does well

Covers eight original environment types
Current FC version supports five tasks
One-command Docker Compose setup documented
Supports local and API-only models
Apache-2.0 licensed and free

What to know first

Webshop needs about 16 GB of RAM
Alfworld can leak memory and disk space
KnowledgeGraph relies on an unstable online service
Original quick start requires Docker

Limits

The webshop environment requires approximately 16 GB of RAM to start. Alfworld may continue consuming memory and disk space until its task worker is restarted.

Getting started

Open the AgentBench GitHub website
Use Python 3.9 for dependency installation
Install Docker for the original quick-start route
Configure an OpenAI API key for the example agent
Run AgentBench FC tasks with the documented Docker Compose command
Change configuration parameters to select another agent

Verdict

AgentBench provides a broad, extensible framework for testing language-model agents, with a newer function-calling version and containerized tasks. Resource demands and documented environment issues should be considered before deployment.

Questions

Is AgentBench free?

Yes. AgentBench uses a free plan and is available for self-hosted deployment.

Which platforms does it support?

Its listed platforms are API, Linux, and self-hosted deployment.

Which tasks are in AgentBench FC?

The function-calling version supports alfworld, dbbench, knowledgegraph, os_interaction, and webshop with containerized deployment.

Can I use local or other API models?

Yes. Local models served with FastChat are supported. API-only models can connect by implementing an Agent Client interface.

Can I add an evaluation environment?

Yes. Implement the Task class and specify the new environment in the YAML configuration.

What license does AgentBench use?

The project uses the Apache-2.0 license.

Facts from AgentBench's own pages

Configuration: The configuration system uses YAML with extended import, default, and overwrite keywords. · github.com, 9 Oct 2026
Current tasks: The current function-calling version supports alfworld, dbbench, knowledgegraph, os_interaction, and webshop tasks. · github.com, 9 Oct 2026
Current version: The repository says its current version is AgentBench FC, which uses function-calling prompts and integrates with AgentRL. · github.com, 9 Oct 2026
Deployment: The project documents a one-command setup for AgentBench FC tasks using Docker Compose. · github.com, 9 Oct 2026
Environments: The original benchmark includes eight environments, including operating systems, databases, knowledge graphs, house-holding, web shopping, and web browsing. · github.com, 9 Oct 2026
Extensibility: Users can add a task environment by implementing the Task class and specifying it in the configuration. · github.com, 9 Oct 2026
FC task coverage: AgentBench FC supports fully containerized deployment for alfworld, dbbench, knowledgegraph, os_interaction, and webshop. · github.com, 9 Oct 2026
Founded: 2023 · github.com, 28 Sept 2026
Hardware limit: The README warns that the webshop environment requires approximately 16 GB of RAM to start. · github.com, 9 Oct 2026
Knowledge graph dependency: The KnowledgeGraph task depends on an online service that the README says is unstable, and it documents local deployment as an option. · github.com, 9 Oct 2026
Known issue: The README warns that the current alfworld implementation leaks memory and disk space until its task worker is restarted. · github.com, 9 Oct 2026
License: The repository is licensed under Apache License 2.0. · github.com, 9 Oct 2026
Maintainer: The THUDM GitHub profile identifies the group as THUKEG and lists FIT Building, Tsinghua University as its location. · github.com, 9 Oct 2026
Model interfaces: The framework supports local models served with FastChat and says models offered only through an API can be connected by implementing an interface in the Agent Client. · github.com, 9 Oct 2026
Model setup: The original quick start configures an OpenAI API key and gives gpt-3.5-turbo-0613 as its example agent. · github.com, 9 Oct 2026
Original environments: The original benchmark covers eight environments: operating system, database, knowledge graph, digital card game, lateral thinking puzzles, household tasks, web shopping, and web browsing. · github.com, 9 Oct 2026
Other agents: The README says users can replace the default agent by changing configuration parameters. · github.com, 9 Oct 2026
Purpose: AgentBench is a benchmark for evaluating the reasoning and decision-making abilities of LLMs acting as agents. · arxiv.org, 9 Oct 2026
Requirements: The README recommends Python 3.9 for dependency installation and requires Docker for the original quick-start setup. · github.com, 9 Oct 2026
Research release: The repository describes AgentBench as an ICLR 2024 work. · github.com, 9 Oct 2026
Resource limits: The current webshop environment requires about 16 GB of RAM to start, and the README warns that alfworld can leak memory and disk space until its task worker is restarted. · github.com, 9 Oct 2026
Support: The project invites users to join its Slack for questions and collaboration, and lists a Google Group for questions about AgentBench FC. · github.com, 9 Oct 2026

Where AgentBench stands on Freedom251

LLM Evaluation Tools#9 of 29