RAGChecker

What RAGChecker is

RAGChecker is a free, open-source framework for evaluating and diagnosing retrieval-augmented generation (RAG) systems. It provides overall pipeline measurements and diagnostic metrics for retrieval and generation, with fine-grained evaluation based on claim-level entailment operations. Documented retriever metrics include claim recall and context precision; generator metrics include context utilization, noise sensitivity, hallucination, self-knowledge, and faithfulness.

For each query, the required annotation is the ground-truth answer. The input format also accepts generated responses and retrieved context. RAGChecker can run through a command-line interface or Python API, with options to select metric groups or run all metrics. Its documented installation uses pip install ragchecker, and the quick start also requires downloading the spaCy en_core_web_sm model. Extractor and checker models are configurable; a documented example uses AWS Bedrock with Llama 3 70B. The project also documents integration with LlamaIndex and provides a guide for using its released benchmark. RAGChecker is self-hosted and licensed under Apache-2.0.

Who it suits

RAGChecker suits developers and teams who need to evaluate retrieval-augmented generation systems and inspect retrieval and generation metrics separately. It also fits users of LlamaIndex who want to assess applications built with that framework.

Limits

RAGChecker is self-hosted and requires a ground-truth answer annotation for each query. Its quick start also requires downloading the spaCy en_core_web_sm model.

Getting started

1. Install the framework with pip install ragchecker.
2. Download the spaCy en_core_web_sm model for the quick start.
3. Prepare query inputs with ground-truth answers, and optionally generated responses and retrieved context.
4. Configure extractor and checker models.
5. Run it through the CLI or Python API and select metric groups or all metrics.

Verdict

Choose RAGChecker if you want an open-source, self-hosted way to measure RAG pipeline, retrieval, and generation performance. Look elsewhere if you need a hosted deployment or cannot provide ground-truth answers for queries.