If an Ask-Your-Docs chatbot gives a wrong answer, inspect the evidence it received before changing the model. The cause may be a missing or malformed source, a retrieval setting that hides the right passage, or generation that fails to use good evidence. These seven checks provide an ordered way to separate those problems and test fixes against the same queries.
1. Confirm the answer is in the source
Search the original document or knowledge store for the exact fact the chatbot should have used. Check that it is present, current, and internally consistent. If the source is missing or wrong, no retrieval setting can recover the correct answer from it.
As an Amazon Associate I earn from qualifying purchases.
Also look for obsolete or contradictory material that could compete with the right fact. Salesforce’s knowledge-retrieval troubleshooting guidance recommends checking retrieved content and its source for incorrect or outdated information.
2. Check ingestion and parsing
Verify that the system extracted the document as intended. A PDF may look correct to a person while its text extraction loses reading order, table relationships, headings, or paragraph boundaries. If the source is structured, confirm that related labels and values remain connected after ingestion.
#1 Best Overall
- Compare extracted text with the original, especially tables, multi-column pages, headings, footnotes, and scanned pages.
- Check whether a heading or qualifier was separated from the statement it explains.
- Confirm preprocessing is appropriate for the source format. GOV.UK’s RAG systems overview describes ingestion and preprocessing as distinct steps in the retrieval pipeline.
3. Inspect chunk boundaries and context
Look at the indexed chunks the chatbot can retrieve, not just the original document. A relevant sentence may have been split away from its heading, exception, date, or surrounding explanation. Conversely, a very large chunk may bury the useful passage in unrelated text.
Compare chunk size and splitting method, including section-aware splitting where it suits the corpus. Keep the query and corpus constant while testing a change; GOV.UK notes that chunk size and split method can affect context relevance, but there is no universal best size established for every document collection.
Rank #2
4. Check query wording, filters, and access
Replay the failing question and examine the query sent to retrieval. Ambiguous wording may miss the passage; excessive specificity may also narrow the search unnecessarily. If the application rewrites queries, inspect both the user’s wording and the rewritten form.
Free tools Windows power users keep installed
One-click scans. No signup required.
- Review metadata filters and confirm they do not exclude the relevant document or passage.
- Check permissions under the affected user’s identity. A document the developer can access may be unavailable to the chatbot user.
- Try a clear paraphrase as a diagnostic, while keeping the original query in the evaluation set.
Salesforce’s troubleshooting documentation specifically flags filters and user permissions as possible causes of missing relevant content.
5. Inspect retrieved candidates and ranking
For each failing query, record the retrieved passages, their rank, and their source location. This reveals whether the right evidence was absent, present but ranked too low, or displaced by plausible-looking but incorrect material. Do not infer retrieval quality from the final answer alone.
If the system supports different retrieval strategies, compare vector and hybrid search on the same corpus and query set. A strategy that helps one kind of question may not help another. Change one variable at a time and preserve the retrieved results so you can attribute any change in the answer to a specific difference.
Rank #4
6. Evaluate retrieval with a fixed query set
Create a repeatable test set containing answerable questions, unanswerable questions, and the evidence passages known to support each answerable question. Include the failures that prompted debugging as well as representative questions from the intended use case. Microsoft recommends evaluating positive and negative examples rather than testing only questions with known answers.
Use retrieval metrics to assess search behavior:
- Precision@K: the fraction of the top K retrieved results that are relevant.
- Recall@K: how much of the relevant evidence appears among the top K results.
- Mean reciprocal rank: how highly the first relevant result appears, averaged across queries.
These measures answer different questions: a system can return mostly relevant passages while still missing needed evidence, or include the evidence but rank it too low. Microsoft documents these metrics in its RAG information retrieval guide. Keep the corpus, queries, and K consistent when comparing configurations.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.7. Compare the evidence with the generated answer
If useful evidence was retrieved, inspect whether the model followed it. Check whether the answer makes claims absent from the passages, overlooks a qualifier, combines conflicting sources, or fails to say when the context is insufficient. Salesforce’s guidance recommends examining the retrieved chunks alongside the associated source content; its quality metrics distinguish context precision, faithfulness, and answer relevance.
Evaluate answer quality separately from retrieval. AWS’s RAG evaluation metrics guidance covers dimensions including context relevance and coverage, correctness, completeness, faithfulness, citation precision, and citation coverage. If citations are part of the product, verify that each one supports the claim it accompanies rather than merely pointing to a broadly related document.
For questions the available documents do not answer, provide an explicit insufficient-context response path. Prompt guidance should direct the system to rely on supplied evidence, identify sources clearly when citations are required, and abstain instead of inventing an answer when evidence is inadequate.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →How to compare a proposed fix
Run the same query set against the same corpus before and after the change. Compare the dimensions that correspond to the suspected failure:
| What to compare | What it tells you |
|---|---|
| Relevant-result precision | Whether retrieved passages are useful rather than distracting. |
| Evidence recall or coverage | Whether the passages needed to answer the question are surfaced. |
| Rank of the first useful passage | Whether relevant evidence appears near the top of the results. |
| Answer faithfulness and completeness | Whether the answer follows the evidence and addresses the question without unsupported additions. |
| Citation correctness and coverage | Whether cited passages support the claims and important claims have citations. |
| Token budget and latency | Whether the change fits operational limits as well as quality goals. |
Official guidance does not establish a universally optimal chunk size, top-K value, reranker, or acceptance threshold. Choose those based on the corpus and measured behavior, not a rule of thumb detached from your use case.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
