The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Evaluate a retrieval-augmented generation (RAG) pipeline in two lanes: score the evidence retrieved for the prompt, then score the answer produced from that evidence. A combined score is useful for a dashboard, but it cannot tell you whether a weak response came from missing context, noisy ranking, or faulty generation.
DeepEval puts the distinction plainly: “A single end-to-end score can’t tell you whether a bad answer came from bad retrieval or bad generation.” Keep the retrieved passages, evaluator inputs, and per-case explanations so every low score becomes a troubleshooting lead rather than an unexplained number.
As an Amazon Associate I earn from qualifying purchases.
Why one RAG score is not enough
A RAG request has at least two separable transformations. The retriever turns a question into a set of context nodes. The generator turns the question and those nodes into an answer. A response can fail at either stage, or at both.
If the corpus contains the answer but retrieval omits it, even a well-prompted model cannot reliably use that evidence. Conversely, retrieval can return the exact passage while the model ignores it, contradicts it, or answers a different question. An end-to-end score collapses those cases into one value and hides the next engineering action.
#1 Best Overall
Keep a combined score if you need a release summary, but never discard the component view. Attach retrieval measurements to the retriever path and answer measurements to generation. Preserve the actual context used for each answer so a groundedness evaluator can inspect what the model saw.
The 2024 EACL demo paper describing Ragas frames evaluation as several abilities rather than a universal scalar: finding relevant and focused context, using that context faithfully, and producing a quality answer. DeepEval’s current RAG documentation uses the same multidimensional approach. Metric names and calculations can differ between framework versions, so treat a label as a contract to verify, not as a guarantee that two implementations are interchangeable.
Separate the questions your evaluators answer
Retrieval metrics
Retrieval metrics assess the material placed in the context window. They do not judge whether the final prose is helpful.
Recommended Free Tools
| Evaluation question | Metric family | What a low result points you toward |
|---|---|---|
| Did retrieval find the evidence needed to answer? | Contextual recall | Missing passages, incomplete corpus coverage, query rewriting problems, or retrieval settings that fail to surface the right chunks. |
| Is the returned context pertinent and focused? | Contextual relevancy | Irrelevant or excessive material, unsuitable chunk boundaries, an overly large top-K, or a noisy query. |
| Were relevant nodes ranked ahead of distracting ones? | Contextual precision | Ordering, similarity calibration, filtering, or reranking behavior. |
These are practical diagnosis directions, not guarantees. For example, low contextual recall can reflect a missing document rather than a faulty vector index; low contextual relevancy can be caused by the query itself. Inspect examples before changing infrastructure.
Rank #2
Answer metrics
Answer metrics evaluate what the generator did with the request and context.
| Evaluation question | Metric family | What to inspect when it is weak |
|---|---|---|
| Is the answer supported by the retrieved material? | Faithfulness | Unsupported claims, contradictions, overconfident extrapolation, or prompts and decoding settings that do not keep the model grounded. |
| Does the answer address the user’s question? | Answer relevancy | Off-topic content, an incomplete response, misunderstood intent, or instructions that do not direct the model to answer the actual question. |
Faithfulness and answer relevancy are different checks. An answer can be faithful to a retrieved passage yet fail to answer the user’s question. It can also be highly relevant in topic while making claims the context does not support.
Record the inputs each metric needs
Scores only have meaning when the evaluator receives the fields its definition expects. Store a case record that lets you reproduce both lanes:
- User input: the exact question after any normalization or rewrite.
- Retrieved context: every node passed to the generator, in the order supplied, with document and chunk identifiers.
- Actual output: the complete generated answer, including citations if your product displays them.
- Expected output or reference answer: a trusted answer when the selected metric or implementation requires one.
- Retrieval trace: query text, top-K, filters, similarity scores, reranker scores, and index or embedding version.
- Generation trace: model version, prompt template, tool calls, temperature or other decoding settings, and token limits.
- Evaluator result: score, reason or explanation, evaluator version, and timestamp.
DeepEval’s faithfulness setup requires the input, actual output, and retrieval context. Its documented contextual precision and contextual recall setups may also require an expected output. Confirm the exact fields in the version you run; do not compare a case evaluated with a reference answer to one evaluated without it as though they were equivalent.
Rank #3
Build evaluation cases that expose failures
Include questions whose answers are distributed across multiple chunks, questions with similar but incorrect documents, short factual questions, multi-part requests, and questions for which the corpus has no supported answer. Label the expected behavior for unsupported questions: for example, refuse, say the information is unavailable, or ask for clarification. Otherwise a useful refusal may be scored as irrelevant simply because the evaluator expected a factual answer.
Keep a human-reviewed sample for calibration. Human labels are not needed for every production request, but they are essential for checking whether an evaluator’s explanations match your team’s definition of “relevant,” “faithful,” and “complete.”
Run an eval-first triage workflow
- Capture one complete trace. Save the question, rewritten query, retrieved nodes, final prompt, answer, and component versions. A score without the context cannot explain a grounding failure.
- Evaluate retrieval before generation. Run contextual recall, relevancy, and precision (or the corresponding metrics in your framework) against the retrieved set. Record per-case reasons, not only aggregate means.
- Evaluate the answer with the same trace. Run faithfulness against the exact context supplied to the model and answer relevancy against the original question. Do not substitute a later, manually assembled context.
- Classify the case by the two views. Mark retrieval as healthy or suspect and generation as healthy or suspect. This creates a short list for manual inspection instead of sending every failure to the prompt team.
- Inspect the smallest useful artifact. For a recall issue, compare the retrieved nodes with the reference evidence. For precision or relevancy issues, inspect ordering, chunk boundaries, filters, and top-K. For faithfulness issues, highlight unsupported answer claims and locate the exact prompt instruction or generation behavior associated with them.
- Change one variable at a time. A reranker, chunk-size change, query rewrite, prompt edit, and model swap can all move the score. Isolate changes so you know which intervention corrected the failure.
- Re-run the same cases and a holdout set. A fix that improves a handful of examples but harms unrelated intents is not a reliable fix. Keep a fixed regression set and a fresh set that was not used to tune the system.
- Set task-specific baselines. The cited documentation defines metric behavior but does not establish universal pass/fail cutoffs. Choose thresholds from your reviewed cases, risk tolerance, and product objective, then version those thresholds with the evaluator.
Read score combinations as triage signals
| Retrieval view | Answer view | Likely investigation path |
|---|---|---|
| Low recall | Low faithfulness or relevancy | First verify corpus coverage and query-to-document matching. The generator may be failing simply because required evidence never arrived. |
| Good recall, low relevancy or precision | Low faithfulness | Check noisy context, chunk size, top-K, metadata filters, and reranking. The needed passage may be present but buried among distractors. |
| Good retrieval on all three dimensions | Low faithfulness | Inspect prompt hierarchy, citation or “use only supplied context” instructions, model behavior, and decoding settings for unsupported claims or contradictions. |
| Good retrieval on all three dimensions | Low answer relevancy | Check intent interpretation, answer format instructions, and whether the model covers every part of a multi-part question. |
| Low retrieval scores | Apparently good answer | Verify whether the answer was produced from memorized or unrelated information. A fluent answer does not prove that retrieval worked. |
| Mixed or contradictory metrics | Mixed | Read the evaluator reasons and the raw trace. Different metrics may use different references or definitions, and aggregate scores can conceal a small set of severe failures. |
The combinations are hypotheses for triage, not causal proof. A low score can have several causes, and evaluator disagreement is itself a signal to review the case and the metric inputs.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsWhat to change for each retrieval failure
Low contextual recall
Confirm that the source of truth actually contains the required fact and that ingestion indexed the relevant edition or permission scope. Then test query normalization, synonyms, metadata filters, embedding model, and top-K. If the answer requires several passages, measure whether all required pieces appear together; retrieving one supporting sentence may not be enough.
Low contextual relevancy
Read the returned nodes, including their surrounding text. Oversized chunks can include unrelated sections, while tiny chunks can remove the qualifier that makes a statement useful. Test chunk boundaries, overlap, top-K, and filters. Query rewriting can improve focus, but it can also remove an important term, so retain both original and rewritten queries in the trace.
Low contextual precision
Compare the order of nodes with their usefulness to the question. Examine similarity calibration, duplicate chunks, metadata boosts, and reranker behavior. If relevant material is consistently below distractors, ranking is the first suspect even when recall is acceptable.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What to change for each answer failure
Low faithfulness
List each factual claim in the answer and mark whether the retrieved context entails it, contradicts it, or says nothing about it. Tighten instructions to distinguish known context from uncertainty, require the model to abstain when evidence is absent, and test whether a different context ordering changes behavior. Keep in mind that a faithfulness evaluator needs the actual retrieval context; evaluating against a human-selected passage can hide what the model really saw.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Low answer relevancy
Check the question as the model received it, especially after query rewriting or conversation-history injection. Require a direct answer before optional explanation, define the expected format for lists or procedures, and test multi-part questions separately. A response can be factually grounded yet still omit the requested decision, step, or constraint.
Best Value
Choose tools by coverage, inputs, and explainability
DeepEval and Ragas both document RAG-oriented evaluation, but a framework choice should follow your pipeline rather than a brand comparison. Compare:
- Metric coverage: whether the tool offers the retrieval and answer dimensions you need.
- Required labels: whether cases need reference answers, retrieved contexts, human judgments, or only the model trace.
- Integration: how easily the evaluator receives your retriever and generator outputs without altering production behavior.
- Cost and latency: evaluator model calls, batch support, retries, and whether you can run a small gate on every change.
- Per-case explainability: reasons, highlighted claims, node-level details, and artifacts that an engineer can act on.
- Version stability: metric names, formulas, and required fields can change; pin versions and record them with each run.
Do not treat identically named metrics from different frameworks as interchangeable. A score is comparable only when the definition, reference data, evaluator model, and input fields are aligned.
Operationalize the two-lane view
Publish separate dashboards for retrieval and generation, with drill-down to the same request ID. Show distributions and failure examples, not just an average. A high mean can hide a small number of safety-critical misses, while a low mean may reflect intentionally difficult queries.
Gate deployments on the dimensions that matter for the product. For a policy assistant, faithfulness and refusal behavior may dominate; for enterprise search, recall and precision may matter more. Keep a combined score only as a summary after those component gates pass.
When a production user reports a bad answer, reproduce the exact trace before changing the prompt. If the retrieved context is wrong, route the issue to ingestion, query processing, ranking, or access-control owners. If context is sound, route it to prompt, model, or generation owners. This ownership split prevents repeated prompt edits for a retriever defect and repeated index tuning for a grounding defect.
Common evaluation mistakes
- Scoring only the final answer: removes the evidence needed to distinguish retrieval from generation.
- Using a different context for evaluation: makes faithfulness results unrelated to the production trace.
- Mixing metric inputs: comparing reference-based and reference-free cases as one population.
- Assuming a universal threshold: no cited source establishes a single cutoff that fits every task, corpus, language, or risk level.
- Optimizing an aggregate alone: can improve one dimension while degrading another.
- Ignoring evaluator explanations: a number cannot tell you whether a failure was a missing passage, an ordering problem, or an unsupported claim.
- Failing to version the evaluator: metric implementation changes can look like a pipeline regression.
Bottom line
An eval-first RAG practice keeps retrieval and generation scores beside the trace that produced them. Use contextual recall, relevancy, and precision to locate evidence and ranking problems; use faithfulness and answer relevancy to judge grounding and responsiveness. Treat every low score as a triage clue, verify the metric’s required inputs, inspect the underlying case, and only then choose a retriever, reranker, prompt, or model change.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
