OpenCompass

Free plan APILinuxSelf-hosted

What OpenCompass is

OpenCompass is a self-hosted LLM evaluation platform for assessing models across datasets spanning knowledge, reasoning, coding, science, language, long-context tasks, and safety. It supports models from providers such as OpenAI, Anthropic, Gemini, Qwen, GLM, and DeepSeek, along with more than 100 datasets. Evaluations can use zero-shot, few-shot, or chain-of-thought approaches with standard or dialogue prompt templates. Tasks can be divided across workers, including data-parallel evaluation on multiple GPUs. Configuration files capture experiments, and results can be reported in real time. The modular system can be extended with models, datasets, task-division strategies, and cluster-management systems. OpenCompass also evaluates API models and lists LMDeploy and vLLM inference backends, as well as an API installation option. A VLMEvalKit integration supports multimodal datasets, inference through OpenAI-compatible APIs, and VLMEvalKit metrics. It is free open-source software under the Apache-2.0 license. The README recommends Python 3.12, though pyext-backed code evaluation datasets call for Python 3.10.

Who it suits

OpenCompass suits researchers and practitioners evaluating large language or large vision-language models. It is geared toward users who can work with a Python package and self-hosted deployment.

What it does well

Free plan for the open-source Python package
Supports more than 100 datasets
Distributed evaluation across multiple GPUs
Tracks experiments and reports results in real time
Supports safety and multimodal evaluation

What to know first

Some datasets require extra dependencies
Agent evaluations may need many dependencies
Pyext-backed code datasets call for Python 3.10

Limits

OpenCompass is free, with some datasets and inference backends requiring optional dependencies. The README recommends Python 3.12, while pyext-backed code evaluation datasets require Python 3.10.

Plans

Plan	Billed and limits	Price
OpenCompass	Open-source software · Python package · Some datasets and inference backends require optional dependencies	Free

As github.com lists them, read 7 Oct 2026. The maker's own page has the final word.

Getting started

Open the OpenCompass GitHub repository.
Install the Python package in a self-hosted environment.
Use Python 3.12 as recommended by the README.
Use Python 3.10 if relying on pyext-backed code evaluation datasets.
Install optional dependencies needed by selected datasets or inference backends.

Verdict

OpenCompass offers broad model and dataset evaluation options, including distributed and multimodal workflows. Check the Python and optional dependency requirements against your environment before setting it up.

Questions

Is OpenCompass free?

Yes. The OpenCompass plan is free.

Is OpenCompass open source?

Yes. The GitHub repository identifies the license as Apache-2.0.

How is OpenCompass deployed?

It is a Python package intended for self-hosted deployment.

Which Python version does it require?

The README recommends Python 3.12. Users relying on pyext-backed code evaluation datasets are instructed to use Python 3.10.

Does OpenCompass evaluate API models?

Yes. It supports API models alongside open-source models and lists API integrations including OpenAI, Gemini, Claude, and Qwen.

Are there optional dependencies?

Yes. Some datasets need extra dependencies, and some agent evaluations may require many dependencies that can conflict with existing environments.

Facts from OpenCompass's own pages

Community resources: The project links to CompassHub for benchmark browsing, CompassRank for leaderboards, documentation, Discord, and WeChat. · github.com, 7 Oct 2026
Dataset caveat: The documentation says some datasets need extra dependencies, and some agent evaluations may require many dependencies that can conflict with existing environments. · opencompass.readthedocs.io, 7 Oct 2026
Distributed evaluation: It supports task division and distributed evaluation, including data-parallel evaluation across multiple GPUs. · github.com, 7 Oct 2026
Evaluation methods: It supports zero-shot, few-shot, and chain-of-thought evaluations with standard or dialogue prompt templates. · github.com, 7 Oct 2026
Experiment tracking: Configuration files record experiments, and the project supports real-time reporting of results. · github.com, 7 Oct 2026
Extensibility: Users can extend the modular system with models, datasets, task division strategies, and cluster management systems. · github.com, 7 Oct 2026
Founded: 2020 · github.com, 28 Sept 2026
Inference integrations: The installation documentation lists LMDeploy and vLLM as inference backends and provides an API installation option. · opencompass.readthedocs.io, 7 Oct 2026
Installation requirements: The README recommends Python 3.12, while users relying on pyext-backed code evaluation datasets are instructed to use Python 3.10. · github.com, 7 Oct 2026
Intended users: The project describes CompassHub and CompassKit as tools for researchers and practitioners evaluating large language and large vision-language models. · github.com, 7 Oct 2026
License: The GitHub repository identifies its license as Apache-2.0. · github.com, 7 Oct 2026
Model APIs: OpenCompass supports evaluating API models alongside open-source models and lists API integrations including OpenAI, Gemini, Claude, Qwen, and others. · github.com, 7 Oct 2026
Models and datasets: The project describes support for models from providers including OpenAI, Anthropic, Gemini, Qwen, GLM, and DeepSeek, and more than 100 datasets. · github.com, 7 Oct 2026
Multimodal support: The project says its VLMEvalKit integration enables multimodal dataset loading, inference through OpenAI-compatible APIs, and evaluation with VLMEvalKit metrics. · github.com, 7 Oct 2026
Purpose: OpenCompass is an LLM evaluation platform for assessing models across datasets covering knowledge, reasoning, coding, science, language, long context, and safety. · github.com, 7 Oct 2026

Where OpenCompass stands on Freedom251

LLM Evaluation Tools#17 of 29