OpenAgent Eval

Free plan APILinuxMacSelf-hostedWindows

What OpenAgent Eval is

OpenAgent Eval is a free, open-source framework for evaluating retrieval-augmented generation systems and AI agents. It runs locally on Linux, macOS, and Windows, or in a self-hosted setup, without telemetry or required network calls. The `oaeval` command-line tool and typed Python SDK let developers place evaluations in test suites. It supports LangChain, LlamaIndex, and custom RAG pipelines. Built-in metrics cover retrieval, generation, performance, and cost, including faithfulness, latency, and token count. Reports can be printed in a terminal or produced as Markdown, HTML, or JSON with failure analysis. Documented model providers include OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, and a mock provider. Retriever integrations include Chroma, Qdrant, Pinecone, Weaviate, FAISS, PGVector, Elasticsearch, BM25, Memory, HTTP, and Mock. Sentence Transformers is the listed built-in embedder. Plugins can add metrics, providers, embedders, and report generators. Mock providers support complete offline runs without credentials. Python 3.11 or later is required, and some integrations need extra dependencies.

Who it suits

It suits developers evaluating RAG pipelines or agents who want local, scriptable tests and reports. Teams using supported retrievers or model providers can extend it through its plugin architecture.

What it does well

Local operation without telemetry or required network calls
CLI and Python SDK for test-suite integration
Metrics cover retrieval, generation, performance, and cost
Reports available in terminal, Markdown, HTML, and JSON
Offline mock providers need no credentials

What to know first

Requires Python 3.11 or later
Some retrievers and embedders need extra dependencies
Built-in embedder listing is limited to Sentence Transformers

Limits

Some retrievers and embedders require extra dependencies such as chromadb, sentence-transformers, faiss-cpu, or qdrant-client. The project offers an [all] package extra for the full dependency set.

Getting started

Install Python 3.11 or later
Install OpenAgent Eval on Linux, macOS, Windows, or a self-hosted system
Run evaluations with the oaeval command-line interface
Embed evaluations with the typed Python SDK
Connect a supported LLM provider, retriever, or custom pipeline
Use mock providers for a local run without network calls or credentials

Verdict

OpenAgent Eval provides local evaluation workflows with broad provider and retriever integrations. Its dependency requirements and Python version should be planned before installation.

Questions

Is OpenAgent Eval free?

Yes. It is free and open source under the Apache License 2.0.

Where can it run?

It runs on Linux, macOS, Windows, or in a self-hosted setup.

Can evaluations run offline?

Yes. Built-in mock providers can complete evaluations locally without network access or credentials.

What programming interface is available?

The project provides the oaeval command-line interface and a typed Python SDK.

What Python version is required?

Installation documentation requires Python 3.11 or later.

Does data leave the laptop?

The project says it runs on the user's machine without telemetry, dashboards, or accounts, and data never leaves the laptop.

Facts from OpenAgent Eval's own pages

Data handling: The project says it runs on the user's machine without dashboards or accounts and that data never leaves the laptop. · openagenthq.github.io, 8 Oct 2026
Embedders: Sentence Transformers is the listed built-in embedder, and custom embedders can be added through provider base classes. · openagenthq.github.io, 7 Oct 2026
Extensibility: A plugin architecture supports custom metrics, providers, and report generators. · openagenthq.github.io, 7 Oct 2026
Framework support: The framework describes itself as compatible with LangChain, LlamaIndex, and custom RAG pipelines. · openagenthq.github.io, 8 Oct 2026
Integrations: Documented LLM providers include OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, and a mock provider. · openagenthq.github.io, 8 Oct 2026
License: The project is licensed under Apache License 2.0. · github.com, 7 Oct 2026
LLM integrations: Documented LLM providers include OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, and a mock provider. · openagenthq.github.io, 7 Oct 2026
Local operation: The project says it runs on the user's machine without telemetry or required network calls. · openagenthq.github.io, 7 Oct 2026
Metrics: Built-in metrics cover retrieval, generation, latency, and token cost. · openagenthq.github.io, 8 Oct 2026
Notable dependency limit: Some retrievers and embedders require extra dependencies, with an `[all]` package extra offered for the full set. · openagenthq.github.io, 8 Oct 2026
Notable limitation: Some retrievers and embedders require extra dependencies, such as chromadb, sentence-transformers, faiss-cpu, or qdrant-client. · openagenthq.github.io, 7 Oct 2026
Offline use: Built-in mock providers can run a complete evaluation locally without network calls or credentials. · github.com, 8 Oct 2026
Purpose: OpenAgent Eval is an open-source, local-first framework for evaluating RAG systems and AI agents. · openagenthq.github.io, 8 Oct 2026
Reports: Reports can be output in terminal, Markdown, HTML, or JSON formats, with built-in failure analysis. · openagenthq.github.io, 8 Oct 2026
Requirements: The installation guide lists Python 3.11 or later as a requirement. · openagenthq.github.io, 8 Oct 2026
Retriever integrations: Documented retrievers include Chroma, Qdrant, Pinecone, Weaviate, FAISS, PGVector, Elasticsearch, BM25, Memory, HTTP, and Mock. · openagenthq.github.io, 7 Oct 2026
Retrievers: Documented retriever providers include Chroma, Qdrant, Pinecone, Weaviate, FAISS, PGVector, Elasticsearch, BM25, Memory, HTTP, and Mock. · openagenthq.github.io, 8 Oct 2026
Support: The project directs users to GitHub Issues for bugs and GitHub Discussions for questions and ideas. · github.com, 8 Oct 2026
Usage: It provides the `oaeval` command-line interface and a typed Python SDK for embedding evaluations in test suites. · openagenthq.github.io, 8 Oct 2026
Ways to use: It provides the `oaeval` command-line interface and a Python SDK for embedding evaluations in test suites. · openagenthq.github.io, 7 Oct 2026

Where OpenAgent Eval stands on Freedom251

AI Agent Evaluation Tools#20 of 29