OpenCompass

Linux · Self-hosted · API

Freedom report

One barScore 5.2

Free tierA free tier is on its own pricing page
Open codeNo open-source code on record
Runs widely1 of 6 device platforms
DocumentedThinly documented

OpenCompass is ranked #24 of 29 in LLM evaluation tools on Freedom251. It runs on API, Linux, Self-hosted. There is a free plan.

OpenCompass plans and pricing

All plans

OpenCompass Free Open-source software · Python package · Some datasets and inference backends require optional dependencies github.com · 7 Oct 2026

Compared on LLM evaluation tools

Free plan: Yes
Deployment options: self-hosted
Custom metrics: Yes
LLM-as-a-judge: Yes
Safety evaluations: Yes
Human review workflows: Yes

Best OpenCompass alternatives

See all 20

Two bars DeepEval Free tier No. 1Free6.6 Two bars Promptfoo Free tier No. 2Free6.6 One bar Giskard Free tier No. 3Free6.5 One bar Braintrust Free tier No. 4$249/mo6.4 One bar Confident AI Free tier No. 5$200/mo6.4 One bar Galileo Free tier No. 6$100/mo6.4

The record

Purpose: OpenCompass is an LLM evaluation platform for assessing models across datasets covering knowledge, reasoning, coding, science, language, long context, and safety.
Models and datasets: The project describes support for models from providers including OpenAI, Anthropic, Gemini, Qwen, GLM, and DeepSeek, and more than 100 datasets.
Evaluation methods: It supports zero-shot, few-shot, and chain-of-thought evaluations with standard or dialogue prompt templates.
Distributed evaluation: It supports task division and distributed evaluation, including data-parallel evaluation across multiple GPUs.
Experiment tracking: Configuration files record experiments, and the project supports real-time reporting of results.
Extensibility: Users can extend the modular system with models, datasets, task division strategies, and cluster management systems.
Model APIs: OpenCompass supports evaluating API models alongside open-source models and lists API integrations including OpenAI, Gemini, Claude, Qwen, and others.
Inference integrations: The installation documentation lists LMDeploy and vLLM as inference backends and provides an API installation option.
Multimodal support: The project says its VLMEvalKit integration enables multimodal dataset loading, inference through OpenAI-compatible APIs, and evaluation with VLMEvalKit metrics.
Community resources: The project links to CompassHub for benchmark browsing, CompassRank for leaderboards, documentation, Discord, and WeChat.
License: The GitHub repository identifies its license as Apache-2.0.
Installation requirements: The README recommends Python 3.12, while users relying on pyext-backed code evaluation datasets are instructed to use Python 3.10.
Dataset caveat: The documentation says some datasets need extra dependencies, and some agent evaluations may require many dependencies that can conflict with existing environments.
Intended users: The project describes CompassHub and CompassKit as tools for researchers and practitioners evaluating large language and large vision-language models.
Price from: Free

Where it ranks on Freedom251

Best LLM Evaluation Tools in 2026#24 of 29

Is OpenCompass yours?

Claim it for free: prove the domain, then correct facts, plans and screenshots. An editor reviews every change.

Claim it free Promote OpenCompass

Sources

github.com/open-compass/opencompass· checked 7 Oct 2026
opencompass.readthedocs.io/en/latest/get_started/installation.html· checked 7 Oct 2026