What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
After updating a knowledge base, rerun a fixed set of representative questions and compare the chatbot’s answers with expected facts. Include questions that should change, control questions that should stay stable, and edge cases where the right response is to acknowledge uncertainty. For a chatbot that uses retrieval-augmented generation (RAG), check that the updated material was ingested and retrieved—not just whether the final answer sounds right.
Build a regression test set before changing the knowledge base
Keep a versioned dataset of questions and expected answers or key facts. n8n describes evaluations as a way to test AI workflows with sample inputs and often expected outputs, and recommends representative examples that reflect production edge cases. Its documentation calls evaluation “a crucial technique for checking that your AI workflow is reliable.” n8n’s evaluation overview explains the approach.
As an Amazon Associate I earn from qualifying purchases.
For each knowledge-base update, record which source documents changed, their versions or dates, and the answers you expect the change to affect. Pair those targeted questions with controls whose answers should remain the same. This helps distinguish the intended change from an unrelated regression.
- Write down each question and its expected answer or required facts.
- Add paraphrases of common questions, not only the wording used in documentation.
- Include cases where the updated material does not support an answer; define the expected uncertainty or fallback response.
- Where useful, note which source or citation should support the answer.
- Keep the dataset versioned so results from different runs can be compared.
There is no single test-set size or pass threshold prescribed by n8n’s documentation. Set both according to the chatbot’s risk and use case; a handful of happy-path questions is unlikely to reveal failures on realistic edge cases.
#1 Best Overall
Check ingestion and retrieval before judging the answer
If the chatbot uses RAG, first confirm that the updated documents reached the vector store and that retrieval can return the relevant, current material. A correct-looking answer does not prove that the chatbot used the new source; it may be relying on another document or on prior model behavior.
n8n’s RAG documentation describes loading and splitting documents, creating embeddings, storing them, and retrieving relevant chunks through an agent or vector-store node. Check the updated source, its metadata, the chunks created from it, and the results returned for your test questions. Use the same embedding model when inserting documents and querying the vector store, as n8n recommends. See n8n’s RAG guide.
Chunking settings depend on the corpus and task. n8n gives 200–500 tokens as an example range for fine-grained retrieval, while noting that larger chunks can preserve context but may add noise. Treat that range as a starting point to test on your own documents, not a universal setting.
Run the same questions and choose an evaluation method
Run the unchanged dataset after re-ingestion, then compare actual answers with the references. For a small development check, inspect answers visually. For larger datasets, metric-based evaluation can help summarize and compare runs. n8n distinguishes these light and metric-based approaches in its evaluation overview and metric-based evaluation guide.
| Approach | Useful when | What it gives you |
|---|---|---|
| Light evaluation | You have a handful of examples during development. | Direct, visual comparison of answers with expected results. |
| Metric-based evaluation | You have a larger dataset, often drawn from production, and want to track runs. | Metrics, score summaries, and case-level execution review. |
Metric-based evaluations can use data tables or Google Sheets as a dataset source. Availability depends on plan and account status: n8n’s documentation lists Cloud Pro and Enterprise and self-hosted Enterprise, and says registered Community and Starter users can use the feature for a single workflow. Check the current guide and your account before relying on a particular setup; product access can change.
Compare the results across more than one score
Use the same test questions and scoring method across runs so that changes are interpretable. Useful evaluation dimensions include:
- Correctness: Are the answer’s supported facts consistent with the reference? n8n documents an AI-based correctness metric on a 1-to-5 scale.
- Helpfulness and coverage: Does the answer address the full question? n8n documents an AI-based helpfulness metric on a 1-to-5 scale.
- Retrieval relevance: Did the workflow retrieve current, relevant source material? n8n gives RAG document relevance as an example custom metric.
- Unsupported claims and abstention: Does the chatbot avoid inventing details or answer cautiously when the knowledge base is silent? Define a human-review rubric or custom metric for this; the documented built-in metrics do not establish that unsupported claims are absent.
- Operational behavior: Did the workflow complete and use the expected tools? n8n documents a tools-used metric, and the Executions view can be filtered by status and workflow.
Metrics can be deterministic or AI-calculated, and n8n notes that evaluations may add latency and cost. String similarity or exact-match checks can be useful for constrained outputs, but similarity alone is not proof of factual correctness: a correct answer may use different wording, and a closely matching answer may still be wrong. Review high-impact factual failures individually. n8n’s metric-based evaluation guide describes its evaluation metrics and result review.
Investigate failures in the execution, then update the baseline
When a test fails or changes materially, open that case’s execution details and trace the answer through the workflow. Check whether the updated source was missing or stale, retrieval returned an irrelevant chunk, prompt or model behavior changed, or the expected answer needs revision because the authoritative knowledge changed.
- Open the failed case from the evaluation results and inspect its execution details.
- In n8n, use Executions to filter by workflow, status, and start time, then load prior execution data when debugging.
- Trace the retrieved documents and chunks, then inspect the downstream prompt, model response, and tool use as relevant to the workflow.
- Fix the underlying issue and rerun the full dataset, not only the failed question.
- Add genuine production failures and edge cases to the dataset; change expected answers only when the authoritative knowledge itself has changed.
- Save the updated dataset and run results as the next baseline.
Execution history is useful for diagnosis, but it is not a durable audit archive: n8n states that deleting a workflow also deletes its execution history. Preserve the evaluation dataset and any long-term reports separately. n8n’s execution documentation covers execution history and debugging.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
