Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Evaluate retrieval and the generated answer as two separate stages. First check whether the system returned the evidence needed for the query; then check whether its answer uses that evidence correctly and cites it accurately. A fluent or confident answer does not prove the right documents were retrieved.
What counts as the “right” document?
In retrieval-augmented generation (RAG), a retrieval system or knowledge base selects information in response to a query and supplies it to a generative AI model as context. That is how NIST defines RAG in its glossary.
As an Amazon Associate I earn from qualifying purchases.
For evaluation, “right” means relevant to a particular information need—not merely similar in topic. A result might mention the same product, person, or subject as the question yet omit the passage that actually answers it. Before testing, identify the documents or passages that contain useful evidence and decide what counts as relevant for each query. Microsoft’s retrieval evaluation guidance recommends preparing test queries alongside the text in test documents that addresses them.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallA practical way to test retrieval
- Build a representative query set. Use realistic questions people are likely to ask. For each, label the relevant documents or passages in advance, including the specific evidence needed to answer it. Add unanswerable queries where the corpus contains no adequate answer; these help reveal whether the system returns irrelevant material instead of recognizing the gap. Microsoft recommends testing positive and negative examples.
- Inspect results before looking at the generated answer. Record the passages returned, their order, and whether they contain the expected evidence. This isolates retrieval quality from the model’s ability to turn context into an answer.
- Measure relevance, coverage, and rank. Use more than one metric because each answers a different question. Precision at K measures the share of the top K results judged relevant. Recall at K measures the share of all relevant items found within those top K. Mean Reciprocal Rank (MRR) reflects how highly the first relevant result appears. Context relevance and context coverage offer related ways to assess relevance and whether the ground-truth material was covered; AWS describes these dimensions in its RAG evaluation metrics guidance.
- Check for missing evidence, not just topical matches. A relevant-looking short snippet may still leave out an answer-bearing passage or a crucial fact. Compare retrieved context with the evidence identified for the query and log consequential misses.
- Evaluate the answer separately. Check whether it is correct and complete, whether its claims are supported by retrieved text, and whether its citations point to passages that actually support those claims. AWS distinguishes citation precision—the correctness of cited passages—from citation coverage—how well the response is supported by citations.
- Compare changes on the same cases. When changing indexing, retrieval, or ranking settings, rerun the fixed query set. Report individual failures as well as aggregate scores: averages can conceal a serious miss on an important query. Microsoft advises examining positive and negative query results separately.
Choose measures for the question you need to answer
| Evaluation question | Useful measure | What it tells you |
|---|---|---|
| Are the top results relevant? | Precision at K or context relevance | How pertinent the returned passages are, and how much irrelevant material appears. |
| Did retrieval find enough of the needed evidence? | Recall at K or context coverage | Whether relevant material is missing from the retrieved set. |
| Is useful material ranked near the top? | MRR or another rank-sensitive measure, such as nDCG | How early useful results appear. Microsoft describes MRR; NIST’s TREC reporting includes nDCG and recall as retrieval measures. |
| Does the answer address the query accurately? | Correctness and completeness | Whether the response is accurate and covers what the user asked. |
| Are answer claims grounded in retrieved text? | Faithfulness or groundedness | Whether the response’s claims are supported by the supplied context. |
| Do citations support the claims, and are claims cited? | Citation precision and citation coverage | Whether cited passages are appropriate and how fully citations support the response. |
Pick K based on the task: it is the cutoff for the results being assessed, not a universal setting. Relevance labels, query mix, cutoff, and task all affect what a score means. Do not treat scores from different test sets as automatically comparable, or assume a strong retrieval score proves the answer is true.
#1 Best Overall
Use failures to locate the problem
- Top results are off-topic: investigate retrieval relevance and ranking.
- Some useful passages appear, but key evidence is absent: investigate retrieval coverage and the chosen cutoff.
- The needed evidence is present, but the answer misstates or ignores it: investigate answer correctness and faithfulness.
- A claim has no supporting passage, or a citation points to the wrong text: investigate citation precision and coverage.
- An unanswerable query produces a confident answer from unrelated material: check whether the system can handle cases where the corpus lacks evidence, rather than relying on answer fluency.
What benchmark findings can—and cannot—tell you
NIST’s July 18, 2025 publication, updated September 18, 2025, reports that automatically generated UMBRELA relevance assessments produced rankings highly correlated with fully manual assessments for nDCG@20, nDCG@100, and Recall@100 across 77 runs from 19 teams in the TREC 2024 RAG Track. In that benchmark, LLM assistance did not appear to increase correlation with fully manual assessments. These findings concern run-level rankings in that evaluation; they do not guarantee that automated judgments will be reliable for a different corpus or a particular consequential query. See NIST’s study of relevance assessments.
NIST’s TREC 2025 RAG Track overview describes four tasks: retrieval, augmented generation, retrieval-augmented generation, and relevance judgment. Its support evaluation uses weighted precision to assess how many correct passage citations are present and weighted recall to assess how many answer sentences are supported by passage citations. These are useful examples of separating retrieval and answer-support questions; they do not establish a universal score threshold for “right documents.”
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.A compact evaluation record
For each test query, keep a record that lets you diagnose both successes and misses:
Recommended Free Tools
- The query and whether the corpus should contain an answer.
- The expected relevant documents or answer-bearing passages.
- The top K returned passages and their ranks.
- Relevance and coverage judgments, plus the chosen retrieval measures.
- Whether the generated answer is correct, complete, and supported by the retrieved text.
- Whether each citation supports its associated claim.
- The failure stage, if any, and whether a system change fixed it on the same query.
There is no universal score that demonstrates a RAG system always finds the right documents. A defensible assessment depends on task-specific relevance judgments, separate checks of retrieval and answer support, and review of important per-query failures.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
