Recommended Free Tools
How do I evaluate retrieval in my RAG system? Freeze a representative set of queries and a corpus snapshot, record which chunks retrieval returns and their ranks, then compare those results with relevance judgments using metrics such as Precision@k and Recall@k. Diagnose retrieval first; measure the generated answer separately. A stronger retrieval score does not by itself guarantee a better answer.
What retrieval evaluation measures
Retrieval evaluation measures the search stage: whether the documents or chunks returned for a query are useful, and where relevant items appear in the ranking. It is distinct from evaluating the complete RAG system, which also includes how the model uses retrieved context to produce an answer. Microsoft’s RAG evaluator guidance distinguishes these evaluation targets.
As an Amazon Associate I earn from qualifying purchases.
To measure retrieval directly, you need a reference for what should have been retrieved. For an answerable query, that is ideally a set of relevant documents or chunks. For a query the corpus cannot answer, the expected result is no useful match. If those reference labels are incomplete, recall measures coverage only against the relevant items you have identified—not against every item that might actually be relevant.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteBuild a repeatable retrieval test
1. Freeze the cases and corpus
Choose queries that reflect the way people use the system, including answerable questions and questions that should have no useful match. Record the relevant documents or chunks for answerable queries. Keep the corpus snapshot fixed while comparing retrieval settings; otherwise, a changed index can confound the comparison.
#1 Best Overall
2. Log results and configuration
For every query, save the returned document or chunk identifiers, their ranks and retrieval scores. Also record the settings that could affect the result, such as filters, top-k, hybrid search, or reranking. This makes it possible to trace a metric change back to a specific result list and configuration.
3. Choose metrics for the cost of failure
Start with Precision@k and Recall@k. Add a ranking metric when the order of results matters, or a graded metric when relevance is not simply yes or no. The choice should reflect the errors that matter for your application—not a desire to maximize a single score.
4. Run the full set and inspect failures
Report an aggregate score, but also examine individual misses and noisy result lists. Microsoft’s information-retrieval guidance recommends testing positive and negative examples and averaging their results separately. A single average can hide a configuration that works well on common questions but fails on questions that should be rejected.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →5. Change one retrieval setting at a time
Compare configurations against the same queries, corpus snapshot, judgments, and metrics. Changing one retrieval setting at a time—for example, top-k or reranking—makes the source of a result change easier to identify. Treat this as an experimental practice, not a vendor-mandated protocol.
Which retrieval metric should you use?
Metrics answer different questions about a ranked result list. In the definitions below, k is the number of top results included in the evaluation.
| Metric | What it tells you | Useful when |
|---|---|---|
| Precision@k | The share of the top-k returned items judged relevant. | Irrelevant context is costly, or the top results need to be clean. |
| Recall@k | The share of all known relevant items that appear in the top k. | Missing relevant material could make an answer incomplete. |
| MRR | The average reciprocal rank of the first relevant result. | The first useful result is especially important. |
| MAP@k | Ranking quality across relevant results, rather than only the first relevant result. | You care about the placement of multiple relevant items. |
| DCG@10 | A position-sensitive ranking measure that can preserve graded relevance judgments. | Results have different degrees of relevance and earlier positions matter. Databricks recommends DCG@10 for many applications, but the metric should still fit the task’s error costs. |
Microsoft’s retrieval quality guidance describes these ranking metrics. Precision and recall are not interchangeable: precision describes how clean the returned top results are, while recall describes how much of the known relevant set was found. A system can improve one while worsening the other.
Use context judgments carefully
If you do not have labeled relevant documents, an LLM judge can assess whether retrieved context appears relevant to a query. That provides a different kind of evidence from checking results against labeled relevant chunks. Treat judge scores as estimates, make the evaluator’s inputs and judgment method explicit, and review samples rather than presenting the scores as objective ground truth. Microsoft’s RAG evaluator documentation describes context-focused evaluators.
Some RAG frameworks also define context relevance or precision and context recall. Their exact definitions and evaluator inputs can vary by implementation; RAGAS, for example, documents its available metrics at its metric reference. State whether a score comes from reference labels or a model judge so readers of your results know what was actually assessed.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Keep retrieval scores separate from answer scores
A retrieved list can be relevant while the model still gives an incomplete, incorrect, or poorly targeted response. Conversely, a model may produce a plausible answer despite retrieval missing useful evidence. Retrieval metrics cannot establish whether the generated response is supported by its context or answers the user’s question.
- Faithfulness or groundedness asks whether the answer is supported by the supplied context.
- Answer relevance asks whether the response addresses the query.
- Completeness and correctness assess other qualities of the generated response and should be interpreted alongside the context and task.
Once you have diagnosed retrieval, evaluate responses separately while keeping the retrieved context visible to the evaluator. Microsoft advises combining response metrics because each measures a different aspect, and model responses can vary across runs. See its end-to-end evaluation guidance and evaluation metrics overview.
Read the failure pattern before tuning
- Low precision with reasonable recall: relevant material is present, but the list also contains noise. Inspect the unwanted chunks and test retrieval settings that could reduce irrelevant context.
- Low recall: relevant items are missing from the top results. Check whether the expected chunks are in the indexed corpus and whether the retrieval configuration can surface them.
- Relevant items appear too low: inspect ranks and compare a first-result metric such as MRR or an ordering-sensitive metric such as DCG@10.
- Positive and negative queries behave differently: report their results separately. A system that retrieves something for every query may look productive while failing to recognize queries with no useful corpus match.
- Retrieval looks sound but answers fail: investigate generation and context use with response-level measures rather than changing the prompt on the basis of retrieval scores alone.
These patterns are diagnostic clues, not proof of a particular root cause. Use the saved result lists and query-level examples to verify what failed before changing a setting.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What to report for a useful comparison
A concise evaluation report should let someone reproduce the comparison and understand its limitations. Include:
- the query set and corpus snapshot used;
- the relevance reference and how it was created, including which queries should have no useful match;
- retrieval configuration, including filters, top-k, hybrid search, or reranking where applicable;
- the metrics, their k values, and whether judgments were binary or graded;
- aggregate results plus representative query-level misses and noisy results; and
- whether context judgments came from labeled references or an LLM judge.
Metric names and evaluator APIs are not identical across implementations. For example, Microsoft’s Foundry Local agentic retrieval metrics page labels that feature PREVIEW; its status may change. Consult the relevant implementation’s current documentation before treating a particular evaluator or definition as universal: Microsoft Foundry Local retrieval metrics.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
