Free and open first
Best LLM Evaluation Tools in 2026
When you evaluate prompts and model outputs, LLM evaluation tools offer different ways to measure results and support review. Compare custom metrics, safety evaluations, and LLM-as-a-judge with human review workflows and prompt versioning. CI/CD integration and deployment options show how each tool can connect with your development setup; free plans and paid-from pricing add cost considerations. DeepEval, Promptfoo, and Giskard are among the entries to consider, alongside Braintrust and Confident AI. Think about which evaluation methods and workflow connections matter for your team, then compare the listed capabilities and plans.
- 29 apps
- Research read 23 Sept 2026
- How the order is worked out
- 1
Arena (formerly Chatbot Arena)
Free plan Open source APISelf-hostedWebArena combines public response comparisons with ranked results across a range of task areas. Users should note its daily usage limit and the stated use of conversations.
- 2
DeepEval
Free plan Open source APILinuxMac+3DeepEval offers a broad toolkit for evaluating AI systems across modalities, metrics, and agent workflows.
- 3
Giskard
Free plan Open source APILinuxSelf-hosted+1 - 4
Promptfoo
Free plan Open source APILinuxMac+3Promptfoo combines prompt evaluation with red-team testing and a broad provider and integration list.
- 5
TruLens
Free plan Open source APISelf-hostedTruLens offers tracing, evaluation, and version comparison for AI applications, with customizable metrics and self-hosted storage options.
- 6
Braintrust
Free plan APISelf-hostedWebBraintrust combines production trace inspection with dataset evaluations and prompt iteration.
- 7
Confident AI
Free plan APISelf-hostedWebConfident AI brings development evaluations, production tracing, alerting, and safety testing into one platform. The free plan has defined usage limits, while paid plans start at $200 per month.
- 8
garak
Free plan LinuxMacSelf-hosted+1garak provides configurable, automated probing for language-model weaknesses across a range of integrations. Its text-only assessment scope is important for anyone evaluating broader model outputs.
- 9
AgentBench
Free plan APILinuxSelf-hostedAgentBench provides a broad, extensible framework for testing language-model agents, with a newer function-calling version and containerized tasks.
- 10
Galileo
Free plan APISelf-hostedWebGalileo combines evaluation, observability, and guardrails with several deployment options and developer interfaces. Check the trace allowance and SSO availability against your deployment needs.
- 11
ARES
Free plan APILinuxSelf-hostedARES offers a defined evaluation method for RAG systems, with support for custom models and local execution.
- 12
HELM
Free plan APISelf-hostedWebHELM brings standardized benchmarks, model connections, evaluation metrics, and inspection tools into a free, self-hosted Python framework.
- 13
EvalPlus
Free plan APILinuxSelf-hostedEvalPlus provides benchmark expansion, efficiency testing, leaderboards, and broad backend support for code-focused model evaluation.
- 14
LM Evaluation Harness
Free plan APILinuxSelf-hostedLM Evaluation Harness offers a shared framework for benchmark and custom model evaluations, with a range of model options and extensibility.
- 15
Inspect AI
Free plan APIExtensionChoose Inspect AI if you need a free, self-hosted framework to evaluate language models across varied task types, with broad provider and agent support.
- 16
Maxim AI
Free plan APISelf-hostedWebMaxim AI brings prompt work, agent evaluation, and production observability into one platform.
- 17
OpenCompass
Free plan APILinuxSelf-hostedOpenCompass offers broad model and dataset evaluation options, including distributed and multimodal workflows.
- 18
Parea AI
Free plan APISelf-hostedWebParea combines LLM evaluation, prompt workflows, observability, and human review. Its free tier has stated team, log, retention, and prompt limits; the Team plan is $150 a month.
- 19
PyRIT
Free plan APILinuxSelf-hosted+1PyRIT combines attack strategies, repeatable scenarios, and configurable scoring for generative AI assessments.
- 20
HarmBench
Free plan APILinuxSelf-hostedHarmBench provides a configurable pipeline for measuring attacks, defenses, and red-teaming performance.
- 21
SWE-bench
Free plan APILinuxMac+3SWE-bench provides datasets and an evaluation harness for checking model-generated patches against repository tests.
- 22
Ragas
Free plan APILinuxSelf-hostedRagas offers a broad evaluation toolkit for LLM applications, from synthetic test data through production monitoring.
- 23
RAGChecker
Free plan Self-hostedChoose RAGChecker if you want an open-source, self-hosted way to measure RAG pipeline, retrieval, and generation performance.
- 24
OpenAI Evals
Free plan Web - 25
DecodingTrust
Paid APISelf-hostedDecodingTrust is a focused toolkit for structured LLM trustworthiness evaluations.
#1Places #1, #2, #3, #4, #5 on this list can be bought by the makers of the software on it. See the price
More in Developer Tools
- LLM Gateway Software34
- LLM Observability Tools26
- LLM Security Tools31
- Load Balancing Software32
- Local Kubernetes Tools23
- Log Management Software111
- Machine Learning Model Monitoring Software20
- Message Broker Software39
- Metrics Monitoring Tools37
- ML Experiment Tracking Software39
- MLOps Platforms29
- Mobile Application Security Testing Software21