OpenCompass

Free plan APILinuxSelf-hostedWindows

What OpenCompass is

OpenCompass is an open-source, self-hosted platform for evaluating large language and multimodal models. It supports more than 100 datasets covering knowledge, reasoning, coding, science, language, long-context work, and safety. Evaluation can use zero-shot, few-shot, or chain-of-thought methods with standard or dialogue-style prompt templates. The project describes distributed evaluation across more than 70 datasets and about 400,000 questions. It supports Hugging Face, API, and custom models, with documented integrations including OpenAI, Gemini, Claude, and Qwen. LMDeploy and vLLM are listed as inference backends, while API workflows include dependencies for OpenAI, Anthropic, Qwen, and ByteDance Volcano Engine. Users configure runs, execute inference and evaluation, and save results as CSV or TXT files; Lark reporting can send status updates to Lark clients. The software supports Linux, Windows, and API use, and can be installed with pip or from source. OpenCompass requires PyTorch 1.13 or later. Basic installation covers most fundamental datasets, while some need extra dependencies or manual setup. The project uses an Apache-2.0 license and supports safety evaluations.

Who it suits

It suits researchers and practitioners comparing language or vision-language models across varied benchmarks. Self-hosted Linux or Windows users can extend models, datasets, task division, and cluster-management strategies.

What it does well

More than 100 supported datasets
Zero-shot, few-shot, and chain-of-thought modes
Supports API, Hugging Face, and custom models
CSV and TXT result output
Safety evaluation dimensions included

What to know first

Requires PyTorch 1.13 or later
Some datasets need extra setup
Several code benchmarks require pyext
Recommended Python 3.10 for those datasets

Limits

Basic installation covers most fundamental datasets, while some require additional dependencies or manual setup. APPS, TACO, and LiveCodeBench code-generation evaluations are unavailable without pyext, with Python 3.10 recommended for those datasets.

Getting started

Install with pip or from source
Use conda to manage the Python environment
Install PyTorch 1.13 or later
Configure a model, dataset, and evaluation method
Run inference and evaluation
Save results as CSV or TXT

Verdict

OpenCompass offers broad model and dataset coverage with configurable evaluation methods and distributed workflows. Check dataset dependencies and the pyext limitation before planning code-generation evaluations.

Questions

Is OpenCompass free?

Yes. It is free and open source under the Apache-2.0 license.

Which systems can run OpenCompass?

It supports API workflows, Linux, Windows, and self-hosted deployment.

Which models can it evaluate?

Documented support includes Hugging Face, API, open-source, and custom models, including OpenAI, Gemini, Claude, and Qwen integrations.

What outputs does it produce?

Evaluation runs can save results as CSV and TXT files. Lark reporting can provide status updates in Lark clients.

Are safety evaluations supported?

Yes. Evaluation dimensions include safety, fairness, legality, harmlessness, ethical considerations, and privacy protection.

What installation requirement matters most?

OpenCompass requires PyTorch 1.13 or later. Some datasets need additional dependencies or manual setup.

Facts from OpenCompass's own pages

API support: The installation guide documents API evaluation and lists dependencies for OpenAI, Anthropic, Qwen, and ByteDance Volcano Engine APIs. · opencompass.readthedocs.io, 4 Oct 2026
Benchmarks: The project describes support for more than 100 datasets covering areas such as knowledge, reasoning, coding, science, language, long context, and safety. · github.com, 4 Oct 2026
Community support: The README links to Discord and WeChat and directs users to GitHub for reporting issues. · github.com, 8 Oct 2026
Compatibility limit: The README says APPS, TACO, and LiveCodeBench code-generation evaluations are unavailable without pyext, and recommends Python 3.10 for those datasets. · github.com, 8 Oct 2026
Dataset limitation: The basic installation supports most fundamental datasets, while some datasets require additional dependencies or manual setup. · opencompass.readthedocs.io, 4 Oct 2026
Datasets: OpenCompass publishes recommended configurations for supported evaluation datasets. · doc.opencompass.org.cn, 8 Oct 2026
Distributed evaluation: The project says it supports task division and distributed evaluation. · github.com, 8 Oct 2026
Evaluation dimensions: Its documentation describes evaluations across language, knowledge, understanding, reasoning, safety, and specialized capabilities. · doc.opencompass.org.cn, 8 Oct 2026
Evaluation methods: It supports objective and subjective evaluation, including model-assisted judging and human-feedback-based judging. · doc.opencompass.org.cn, 8 Oct 2026
Extensibility: The project says users can add models and datasets and customize task division strategies and cluster management support. · github.com, 4 Oct 2026
Inference backends: The README gives LMDeploy and vLLM as accelerated inference backend options. · github.com, 8 Oct 2026
Installation: The project documents installation with pip and from source, and its README recommends using conda to manage the Python environment. · github.com, 8 Oct 2026
Intended users: The project describes CompassHub and CompassKit for researchers and practitioners evaluating language and vision-language models. · github.com, 4 Oct 2026
License: The GitHub repository links an Apache-2.0 license. · github.com, 4 Oct 2026
Model integrations: Documented model integrations include OpenAI, Gemini, Claude, Qwen, and other API models, as well as open-source models. · github.com, 4 Oct 2026
Model support: The README describes support for Hugging Face models, API models, and custom models. · github.com, 8 Oct 2026
Purpose: OpenCompass is a platform and toolkit for evaluating large language models and large vision-language models. · github.com, 8 Oct 2026
Reporting: The quick start says users can enable Lark reporting for status updates in Lark clients. · opencompass.readthedocs.io, 4 Oct 2026
Requirements: The installation guide states that OpenCompass requires PyTorch 1.13 or later. · opencompass.readthedocs.io, 4 Oct 2026
Safety evaluation: Its evaluation dimensions include fairness, legality, harmlessness, ethical considerations, and privacy protection. · doc.opencompass.org.cn, 8 Oct 2026
Scale: The project describes distributed evaluation and a scheme with 70+ datasets and about 400,000 questions. · github.com, 4 Oct 2026
Windows compatibility: A release note says OpenCC was made optional to restore Windows installation compatibility. · github.com, 8 Oct 2026
Workflow and outputs: The quick start describes configuring, running inference and evaluation, and saving results as CSV and TXT files. · opencompass.readthedocs.io, 4 Oct 2026

Where OpenCompass stands on Freedom251

AI LLM Evaluation Tools#23 of 30