LiveBench

Paid LinuxMacSelf-hostedWebWindows

What LiveBench is

LiveBench is a free, open-source benchmark for evaluating large language models, designed to limit test-set contamination and use objective scoring. The current benchmark covers 23 tasks in seven categories: Reasoning, Coding, Agentic Coding, Mathematics, Data Analysis, Language, and Instruction Following. It refreshes every six months. The leaderboard displays overall and category scores, with expandable model entries for subtask results. Additional views cover quality versus cost, ranked cost, and category profiles. The project says questions have verifiable answers that can be scored automatically without an LLM judge. It seeks to reduce possible contamination using newly released questions and material based on recent datasets, papers, news, and movie synopses. A documented Python command-line pipeline generates model answers, scores them, and displays results; code is public on GitHub and benchmark data is linked on Hugging Face. The README describes OpenAI-compatible API endpoints and inference support for Anthropic, Cohere, Mistral, Together, and Google models. Local inference is unmaintained, and agentic coding evaluation requires Docker.

Who it suits

LiveBench may suit people comparing language models on objective tasks or running evaluations through supported API endpoints. Its documented pipeline also serves users who want to generate, score, and display results themselves.

What it does well

Covers 23 tasks across seven categories.
Refreshes the benchmark every six months.
Questions are scored automatically without an LLM judge.
Public code and benchmark data are available.

What to know first

Local model inference is unmaintained.
Agentic coding evaluation requires Docker.
Task-specific images may take up to 150 GB to store.

Limits

Local model inference is unmaintained, and the project recommends serving local models through an OpenAI-compatible API using vLLM. Agentic coding evaluations require Docker, and their task-specific images may take up to 150 GB to store.

Getting started

Open the LiveBench website to view its benchmark and leaderboard.
Choose a model endpoint; OpenAI-compatible endpoints and listed provider inference support are documented.
Use the documented Python command-line pipeline to generate model answers, score them, and display results.
For local models, serve them through an OpenAI-compatible API using vLLM.
Install Docker before evaluating agentic coding tasks.

Verdict

LiveBench offers a broad, regularly refreshed benchmark with objective scoring and public evaluation tools. Users considering local inference or agentic coding tasks should account for the documented setup limits.

Questions

Is LiveBench free?

Yes. LiveBench is listed as free and open source.

What tasks does LiveBench cover?

The current benchmark describes 23 objective tasks across Reasoning, Coding, Agentic Coding, Mathematics, Data Analysis, Language, and Instruction Following.

Does LiveBench use an LLM judge?

No. Its questions have verifiable objective answers and can be scored automatically without an LLM judge.

Which model providers are supported?

The project documents OpenAI-compatible API endpoints and implemented inference support for Anthropic, Cohere, Mistral, Together, and Google models.

Can I evaluate local models?

Local inference is unmaintained. The project recommends serving models through an OpenAI-compatible API using vLLM.

What does agentic coding evaluation require?

It requires Docker, and storing the task-specific images may take up to 150 GB.

Facts from LiveBench's own pages

Agentic coding requirements: Evaluating agentic coding tasks requires Docker, and the README says task-specific images may require up to 150 GB of storage. · github.com, 9 Oct 2026
Benchmark scope: The current site describes 23 objective tasks across 7 categories, refreshed every six months. · livebench.ai, 4 Oct 2026
Categories: The website lists Reasoning, Coding, Agentic Coding, Mathematics, Data Analysis, Language, and Instruction Following. · livebench.ai, 9 Oct 2026
Contamination controls: The repository README says questions draw on recently released datasets, papers, news articles, and IMDb movie synopses to limit potential contamination. · github.com, 9 Oct 2026
Cost metric: The site defines cost per successful task as (Σ cost ÷ Σ questions ÷ score) × 100 over the selected scope. · livebench.ai, 4 Oct 2026
Current benchmark: The website describes 23 objective tasks across 7 categories, refreshed every six months. · livebench.ai, 9 Oct 2026
Insights: The website provides quality-versus-cost, cost-ranked, and category-profile views. · livebench.ai, 9 Oct 2026
Leaderboard: The leaderboard lets visitors sort overall and category scores, inspect subtask scores, and compare cost per successful task. · livebench.ai, 9 Oct 2026
Local inference: The README describes local model inference as unmaintained and recommends serving models through an OpenAI-compatible API using vLLM. · github.com, 9 Oct 2026
Local models: The README says local model inference is unmaintained and recommends serving models through an OpenAI-compatible API using vLLM. · github.com, 4 Oct 2026
Model compatibility: The README says OpenAI-compatible API endpoints can be used and lists implemented inference support for Anthropic, Cohere, Mistral, Together, and Google models. · github.com, 4 Oct 2026
Model endpoints: The repository documents custom OpenAI-compatible API endpoints and inference support for Anthropic, Cohere, Mistral, Together, and Google models. · github.com, 9 Oct 2026
Open source: The project publishes its code in a public GitHub repository and links to benchmark data on Hugging Face. · github.com, 4 Oct 2026
Product: LiveBench is described as a challenging, contamination-free LLM benchmark. · livebench.ai, 4 Oct 2026
Purpose: LiveBench is an LLM benchmark designed to limit test set contamination and use objective evaluation. · github.com, 9 Oct 2026
Run evaluations: The repository provides a Python script that can generate model answers, score them, and show results; it supports selecting a model and benchmark subset. · github.com, 9 Oct 2026
Scoring: The repository README says questions have verifiable objective ground-truth answers and are scored automatically without an LLM judge. · github.com, 9 Oct 2026
Support: The repository invites model evaluation requests by GitHub issue or email at [email protected]. · github.com, 9 Oct 2026

Where LiveBench stands on Freedom251

LLM Evaluation Tools#26 of 29