The four commonly used RAG evaluation metrics answer different questions: context precision and context recall probe retrieval, while faithfulness and answer relevancy probe generation. Read them together to decide what to inspect—not as interchangeable scores or proof that a system is correct. Their definitions vary by framework, so identify the implementation and its inputs whenever you report a result.
What the four metrics measure
Retrieval-augmented generation (RAG) systems retrieve material and use it to produce an answer. The metrics below test different points in that process. DeepEval explicitly groups contextual precision and recall with retriever measures, and faithfulness and answer relevancy with generator measures; Ragas also lists these measures among its RAG metrics. See DeepEval’s metric overview and the Ragas metric catalogue.
As an Amazon Associate I earn from qualifying purchases.
| Metric | Question it asks | A weak score suggests inspecting | Key limitation |
|---|---|---|---|
| Context precision | Are useful context passages ranked or selected ahead of irrelevant ones? | Ranking, filtering, top-K, chunking, and noise in retrieved context. | Some implementations are reference-based and require an expected answer; definitions differ by framework. |
| Context recall | Did retrieval include the information needed to answer? | Document coverage, query formulation, chunking, indexing, and retrieval depth. | Reference-based evaluation needs labelled target information, and high recall alone does not ensure a useful answer. |
| Faithfulness | Are the answer’s claims supported by the retrieved context? | Unsupported elaboration, generation behavior, or a mismatch between the context and answer. | Support in retrieved text is not proof that the text or answer is true in the real world. |
| Answer or response relevancy | Does the answer address the user’s question? | Prompt and response construction or alignment with the question. | An on-topic answer can still be unsupported, incomplete, or wrong. |
Context precision: useful evidence near the top
Precision concerns the quality and ordering of retrieved context: whether relevant material is prioritized over distracting material. A low result is a reason to inspect the retrieved chunks and ranking, not automatically evidence that the system failed to find the answer at all.
Context recall: whether needed evidence was retrieved
Recall concerns coverage: whether the retrieved set contains the information required to answer. If it is low, check whether the evidence exists in the index and appears in the retrieved results before attributing the failure to answer generation.
#1 Best Overall
Faithfulness: whether claims follow from retrieved evidence
Faithfulness evaluates grounding against the context supplied to the model. DeepEval describes it in terms of claims supported by that context and documents its calculation in its faithfulness guide. A faithful answer can still repeat an error in the source material; the metric does not independently verify claims against reality or a known-correct reference.
Answer relevancy: whether the response addresses the question
Relevancy concerns the relationship between the user’s question and the answer. It does not establish that the answer used evidence properly or covered every necessary detail.
How to interpret metric combinations
Metric patterns are diagnostic clues, not guaranteed explanations of root cause. Use them to choose what to inspect, then verify with examples and controlled changes. DeepEval’s RAG triad guide maps metrics to likely tuning areas, including prompt templates, chunk size, top-K, and embedding models: Using the RAG Triad for RAG evaluation.
Recommended Free Tools
- Low precision with reasonable recall: the system may retrieve the needed evidence but also pass along distracting chunks. Inspect ranking, filtering, and retrieved examples.
- Low recall: investigate query formulation, index coverage, chunk size, and retrieval depth. Confirm whether the necessary evidence was present in the retrieved set before blaming generation.
- High answer relevancy with low faithfulness: the answer may sound responsive while adding claims the retrieved context does not support. Compare its claims with the trace and source passages.
- High faithfulness with low answer relevancy: the response may stay within the evidence yet miss what the user asked. Inspect the prompt and response construction.
- Good averages but poor user outcomes: break results down by query type and inspect individual failures. An aggregate can hide rare but consequential cases.
Reference-based and referenceless evaluation
Check what information a metric requires before comparing scores or designing a test set. DeepEval’s RAG triad—answer relevancy, faithfulness, and contextual relevancy—can be used without an expected output. Its guide distinguishes these from contextual precision and recall, which require a labelled expected answer. Ragas offers multiple metrics and variants, so a shared label does not guarantee the same implementation or inputs across tools. The RAGAS paper provides foundational framing for evaluating retrieval and generation in RAG.
Referenceless evaluation makes it possible to run checks when gold answers are unavailable, but it does not establish correctness by itself. For retrieval coverage, labelled answers or evidence remain useful because they provide a target against which to assess what retrieval included.
A practical evaluation workflow
- Choose the failure you want to detect. Decide whether the priority is missing evidence, distracting evidence, unsupported claims, or answers that do not address the question.
- Build a representative test set. Include varied query types and known failure cases. Keep expected answers or evidence labels where feasible, especially when evaluating retrieval coverage.
- Document the score’s meaning. Record the metric definition and variant, framework and version, judge configuration, and dataset. Without that context, a score is difficult to interpret or compare.
- Inspect examples and judge explanations. Pay particular attention to cases where metrics disagree. Treat an LLM judge as an evaluator that needs validation, not an oracle.
- Set task-specific thresholds and validate changes. Frameworks may let teams configure thresholds, but no universal RAG-quality cutoff follows from that feature. Review important cases with people, change one component at a time, and check whether the outcome improved for the task.
An applied 2026 comparison discusses how metric usefulness can depend on the dataset and criterion, and recommends checking whether a metric actually approximates the criterion being evaluated: Evaluating RAG Metrics in Applied Contexts.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What a score can—and cannot—tell you
These four metrics separate useful concerns: retrieval ranking, retrieval coverage, grounding in supplied context, and responsiveness to the question. A score is meaningful only in light of its implementation, evaluation inputs, and task. Use patterns to locate likely trouble spots; use traces, representative examples, and task-specific validation to determine whether a diagnosis or change is sound.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchQuick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
