No. 18 of 30 ·AI LLM Evaluation Tools

LM Evaluation Harness

Linux · Self-hosted · API

Freedom report

Two barsScore 6.3

Free tierA free tier is on its own pricing page
Open codeNo open-source code on record
Runs widely1 of 6 device platforms
DocumentedPlans, terms and facts published

LM Evaluation Harness is ranked #18 of 30 in AI LLM evaluation tools on Freedom251. It runs on API, Linux, Self-hosted. There is a free plan.

LM Evaluation Harness plans and pricing

All plans

LM Evaluation Harness Free MIT-licensed software · model backends and task dependencies may require separate optional installs github.com · 4 Oct 2026

Compared on AI LLM evaluation tools

Deployment: self-hosted

Best LM Evaluation Harness alternatives

See all 20

Two bars Vellum Free tier No. 1$30/mo6.7 Two bars Weights & Biases Free tier No. 2$60/mo6.7 Two bars Evidently AI Free tier No. 3$80/mo6.6 Two bars Promptfoo Free tier No. 4Free6.6 Two bars DeepEval Free tier No. 5Free6.5 One bar Giskard Free tier No. 6Free6.5

The record

Purpose: LM Evaluation Harness is a unified framework for evaluating generative language models across evaluation tasks.
Benchmarks: The project lists over 60 standard academic LLM benchmarks with hundreds of subtasks and variants.
Model support: Supported model options include Hugging Face Transformers, GPT-NeoX, Megatron-DeepSpeed, and vLLM.
API integrations: The harness supports hosted APIs including OpenAI and Anthropic, plus local servers compatible with OpenAI APIs.
Custom evaluation: Users can define custom prompts and evaluation metrics, and can configure tasks with YAML.
Plugins: Model backends, filters, metrics, and aggregations can be registered from external packages through entry points or loaded from a local module.
Reproducibility: The project says its use of publicly available prompts supports reproducibility and comparability between papers.
Installation: The base package provides the core framework while model backends are installed separately as optional extras.
Supported platforms: Documented runtime options include Windows ML and ONNX Runtime providers for CPU, CUDA, DirectML, NPU, ROCm, and other accelerators.
Security and license: The repository provides the software under the MIT License, which states that it is provided “AS IS” without warranty.
Support: The project directs users to open a GitHub issue or join the EleutherAI Discord for support.
Notable limitation: For model APIs that do not provide logits or log probabilities, the documentation limits use to generate-until tasks.
Users: The project says the harness serves as the backend for Hugging Face’s Open LLM Leaderboard and is used by research papers and organizations.
Price from: Free

Where it ranks on Freedom251

Best AI LLM Evaluation Tools in 2026#18 of 30
Best LLM Evaluation Tools in 2026#7 of 29

Is LM Evaluation Harness yours?

Claim it for free: prove the domain, then correct facts, plans and screenshots. An editor reviews every change.

Claim it free Promote LM Evaluation Harness

Sources

github.com/EleutherAI/lm-evaluation-harness· checked 4 Oct 2026
github.com/EleutherAI/lm-evaluation-harness/blob/m· checked 4 Oct 2026