LM Evaluation Harness

What LM Evaluation Harness is

LM Evaluation Harness is a free, open-source framework for evaluating generative language models across evaluation tasks. It brings together more than 60 standard academic benchmarks, with hundreds of subtasks and variants. Supported model options include Hugging Face Transformers, GPT-NeoX, Megatron-DeepSpeed, and vLLM. It can also connect to hosted APIs such as OpenAI and Anthropic, plus local servers compatible with OpenAI APIs. Users can define prompts and evaluation metrics, configure tasks with YAML, and extend the framework with model backends, filters, metrics, and aggregations from external packages or local modules. The project says publicly available prompts support reproducibility and comparison between papers. The core framework is provided in the base package, while model backends and task dependencies may require separate optional installs. It is MIT-licensed for self-hosted use. Documented runtime options include Windows ML and ONNX Runtime providers for CPU, CUDA, DirectML, NPU, ROCm, and other accelerators. For APIs without logits or log probabilities, use is limited to generate-until tasks.

Who it suits

It suits researchers and organizations evaluating generative language models across benchmarks or custom tasks. Its self-hosted design and configurable tasks may fit users who can manage separate backend and task-dependency installs.

Limits

The base package provides the core framework, while model backends and task dependencies may require separate optional installs. APIs without logits or log probabilities are limited to generate-until tasks.

Getting started

1. Open the LM Evaluation Harness GitHub repository.
2. Install the base package for the core framework.
3. Install optional model backends and task dependencies as needed.
4. Choose a supported model option, hosted API, or compatible local server.
5. Configure tasks with YAML and define any custom prompts or metrics.

Verdict

LM Evaluation Harness offers a shared framework for benchmark and custom model evaluations, with a range of model options and extensibility. Users should account for optional installs and the stated limitation for APIs that do not provide logits or log probabilities.