Free and open first
Best AI LLM Evaluation Tools in 2026
Choosing an LLM evaluation tool starts with the questions you need to answer about a model or prompt. Compared on evaluation methods, model support, and safety evaluations, options such as Vellum, Weights & Biases, and Evidently AI can be weighed against your assessment needs. Prompt versioning is another criterion when prompt changes matter, while API access and deployment help you compare how a tool may fit into a workflow. The entries also show free-plan availability and paid-from pricing, so you can consider access and cost alongside capabilities. Review the listed criteria in light of what you want to evaluate and how you expect to use the tool.
- 30 apps
- Research read 23 Sept 2026
- How the order is worked out
- 1
DeepEval
Free plan Open source APILinuxMac+3DeepEval offers a broad toolkit for evaluating AI systems across modalities, metrics, and agent workflows.
- 2
Evidently AI
Free plan Open source APILinuxMac+3Evidently AI covers evaluation, tracing, synthetic data and ongoing monitoring across AI and ML workflows. Review plan limits and support levels against the scale and deployment option you need.
- 3
Giskard
Free plan Open source APILinuxSelf-hosted+1 - 4
Langfuse
Free plan Open source APILinuxSelf-hosted+1Langfuse brings tracing, prompt workflows, and evaluation features into one AI engineering platform, with both hosted and self-hosted options.
- 5
LangWatch
Free plan Open source APILinuxMac+2LangWatch brings agent simulation, evaluation, tracing, and deployment options together for teams working toward production.
- 6
Opik
Free plan Open source APILinuxMac+3Opik brings tracing, evaluation, testing, and production monitoring into one platform, with both cloud and self-hosted options.
- 7
Promptfoo
Free plan Open source APILinuxMac+3Promptfoo combines prompt evaluation with red-team testing and a broad provider and integration list.
- 8
Vellum
Free plan Open source AndroidAPIExtension+5Vellum offers a personal assistant with saved context, skills, integrations and approval controls.
- 9
TruLens
Free plan Open source APISelf-hostedTruLens offers tracing, evaluation, and version comparison for AI applications, with customizable metrics and self-hosted storage options.
- 10
Arize Phoenix
Free plan APISelf-hostedWebPhoenix brings tracing, evaluation, prompt inspection, and dataset experiments to AI application teams. It is free and open source, with deployment choices and usage management left to users.
- 11
Braintrust
Free plan APISelf-hostedWebBraintrust combines production trace inspection with dataset evaluations and prompt iteration.
- 12
Confident AI
Free plan APISelf-hostedWebConfident AI brings development evaluations, production tracing, alerting, and safety testing into one platform. The free plan has defined usage limits, while paid plans start at $200 per month.
- 13
Galileo
Free plan APISelf-hostedWebGalileo combines evaluation, observability, and guardrails with several deployment options and developer interfaces. Check the trace allowance and SSO availability against your deployment needs.
- 14
LangSmith
Free plan APILinuxMac+3LangSmith combines agent evaluation with prompt tools and observability, including cost and trajectory tracking.
- 15
ARES
Free plan APILinuxSelf-hostedARES offers a defined evaluation method for RAG systems, with support for custom models and local execution.
- 16
HELM
Free plan APISelf-hostedWebHELM brings standardized benchmarks, model connections, evaluation metrics, and inspection tools into a free, self-hosted Python framework.
- 17
HoneyHive
Free plan APISelf-hostedWebHoneyHive covers trace inspection, evaluations, live monitoring, and version experiments for AI agents. Teams should compare the free plan’s limits with their event volume and retention needs.
- 18
Inspect AI
Free plan APIExtensionChoose Inspect AI if you need a free, self-hosted framework to evaluate language models across varied task types, with broad provider and agent support.
- 19
LM Evaluation Harness
Free plan APILinuxSelf-hostedLM Evaluation Harness offers a shared framework for benchmark and custom model evaluations, with a range of model options and extensibility.
- 20
Maxim AI
Free plan APISelf-hostedWebMaxim AI brings prompt work, agent evaluation, and production observability into one platform.
- 21
NVIDIA NeMo Evaluator
Free plan APILinuxSelf-hostedChoose NeMo Evaluator if you need an open-source way to assess generative AI models, RAG pipelines, or agents across multiple evaluation methods.
- 22
Rhesis AI
Free plan APILinuxSelf-hosted+1 - 23
OpenCompass
Free plan APILinuxSelf-hosted+1OpenCompass offers broad model and dataset coverage with configurable evaluation methods and distributed workflows.
- 24
Weights & Biases
Free plan AndroidAPIiPhone+5Choose Weights & Biases if your work needs experiment tracking alongside tools for evaluating and monitoring AI applications, with registry and deployment options.
- 25
Ragas
Free plan APILinuxSelf-hostedRagas offers a broad evaluation toolkit for LLM applications, from synthetic test data through production monitoring.
#1Places #1, #2, #3, #4, #5 on this list can be bought by the makers of the software on it. See the price