A green trace shows that instrumented steps completed; it does not show that the right evidence reached the model or that the answer used it correctly. To find the fault, inspect the request and retrieval, compare retrieved passages with the final prompt context, then check the answer claim by claim. Evaluate whether it is correct, relevant, and complete as separate questions.
What a successful trace does—and does not—prove
A trace records a request moving through the steps your system instruments. If retrieval and generation both finish without an execution error, the trace can be green even when the retriever found weak evidence, the prompt left out the best passage, or the model produced an unsupported answer.
As an Amazon Associate I earn from qualifying purchases.
That is why logging only step status is not enough to diagnose answer quality. Databricks’ Introduction to evaluation & monitoring RAG applications, updated June 30, 2026, recommends tracking inputs, outputs, and intermediate steps such as document retrieval. Without those details, a successful trace tells you that the pipeline ran, not why the response failed.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →How to tell whether the failure is retrieval or generation
Start with two checks: did the model receive enough relevant evidence, and do the answer’s claims follow from that evidence? If either check fails, the trace can look healthy while the response is wrong. If both pass, test whether the response actually resolves the user’s question and whether the evidence itself is trustworthy.
#1 Best Overall
| Dimension | Question it answers | Typical symptom | Inspect |
|---|---|---|---|
| Context relevance | Are the retrieved passages about the question? | A well-supported answer about the wrong subject | Query rewrite, filters, indexed corpus, and ranking |
| Context coverage or claim recall | Did retrieval find the evidence needed to answer all parts? | An incomplete answer or a guessed missing detail | Corpus contents, chunk boundaries, filters, and retrieval results |
| Faithfulness | Does each answer claim follow from the supplied context? | Unsupported claims or contradictions | The final prompt context, instructions, and generated response |
| Correctness | Is the answer accurate against a trusted reference? | The answer repeats a stale or incorrect source | Source authority, version or date, and reference answer |
| Answer relevance | Does the response address what was asked? | A true but evasive, irrelevant, or overly broad response | Question interpretation and response scope |
| Completeness | Does it resolve every part of the question? | One requested detail is omitted | Question decomposition, context coverage, and answer structure |
| Citation precision and coverage | Do citations support the claims, and are claims that need citations covered? | Misleading or missing citations | Claim-to-passage mapping and citation rendering |
AWS Bedrock documents metrics including context relevance and coverage for retrieve-only evaluation, and correctness, completeness, faithfulness, citation precision, and citation coverage for retrieve-and-generate evaluation. The RAGAS paper and Amazon Science’s RAGChecker tutorial describe related distinctions, including answer relevance, claim-level support, and claim recall. These dimensions are diagnostic lenses, not universal pass thresholds.
Debug a RAG trace in six steps
-
Reconstruct the exact request
Record the original question, conversation history, any rewritten query, and the metadata filters applied. Capture retrieved document identifiers and text, scores and ranks, reranker output, the assembled context, the prompt sent to the model, the output, and its citations. A trace without these inputs and intermediate results cannot show whether the system searched for the right thing or what evidence it used.
-
Check retrieval before changing the generator
Confirm that the needed document exists in the indexed corpus, was parsed correctly, and was not excluded by a filter. Check whether the search used the right fields and whether the returned chunks contain the answer-bearing details. Measure both relevance—whether passages fit the question—and coverage—whether they contain enough of the required evidence. If a fact is split across chunks, inspect neighboring context as well.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallSpecial offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy. -
Compare retrieved results with the context actually sent
Do not assume every retrieved passage reached the model. Compare the retriever and reranker outputs with the final assembled prompt context. Assembly or prompt logic can truncate, reorder, duplicate, or omit results; irrelevant material can also compete with useful evidence. The RAGAS paper discusses context relevance and the difficulty of exploiting long passages, particularly when useful information is buried within them.
-
Verify the answer one claim at a time
Break the response into atomic, checkable statements. For each, identify the exact supporting passage or mark it unsupported, contradicted, or absent. RAGChecker describes claim extraction and checking against retrieved context. If the context is relevant and adequate but the answer still adds unsupported details, investigate the prompt instructions, model behavior, and output constraints.
-
Check whether the response answered the question
A response can be faithful to its context and still be incomplete, evasive, or off-target. Evaluate relevance to the user’s question and completeness of the requested answer separately from grounding. AWS Bedrock includes completeness and helpfulness among its documented evaluation metrics; RAGAS distinguishes answer relevance from factual grounding.
-
Make the failure reproducible
Build a compact evaluation set from real user questions and known source material. Include variations in wording and complexity, plus cases involving missing evidence, conflicting versions, tables, long documents, exact dates or quantities, and questions that should receive an uncertainty statement or refusal. Keep the set stable while testing changes to retrieval, chunking, prompts, models, or reranking. Google Cloud’s December 19, 2024 guidance recommends representative questions, known-good outputs, repeatable metrics, and changing one variable at a time.
Recommended: Crashes or Glitches? A Free Driver Scan Usually Finds the Culprit →Recommended: Fix Windows Errors and Clear Junk Files in Minutes - Free Scan →Recommended: Update Every Outdated Driver on Your PC in One Scan - Free →Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Read metric combinations as clues, not verdicts
Metric dimensions help narrow the search, but an overall score can conceal a weak component. Salesforce’s documented diagnostic patterns treat high faithfulness with low context relevance as a clue to retrieval trouble, and low faithfulness with high context relevance as a clue to generation or prompt trouble. Use those patterns to choose what to inspect next, then verify against the actual passages and answer.
- Relevant but incomplete context: investigate corpus coverage, filters, chunking, and retrieval recall before rewriting the answer prompt.
- Relevant, adequate context but unsupported claims: inspect the final prompt and generator behavior, then test whether stricter answer constraints help.
- Faithful answer based on a bad source: check the source’s authority and version; grounding does not make a source correct.
- Well-supported answer that misses the request: examine query interpretation and whether the response addresses every part.
Use automated scores carefully
Automated metrics and judges can make evaluation more repeatable and help identify which stage deserves attention. They do not establish a universal accuracy guarantee. Review a sample of failures against the source text, especially when the response contains exact claims, dates, quantities, or evidence that conflicts across documents. Calibrate thresholds to the application rather than treating a score as a universal pass mark.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
