Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →A reliable retrieval-augmented generation (RAG) system is not one search call attached to a language model. It is a chain of stages—ingestion, retrieval, context assembly, and generation—and each can independently prevent the model from receiving or using the right evidence. Start with a simple pipeline, test each stage on representative documents and questions, and add hybrid search, reranking, or agentic behavior only when evaluation identifies a failure those additions can address.
What a RAG pipeline actually does
RAG retrieves relevant material from an external knowledge source and supplies it to a language model as context for answering a question. The system has two connected flows: a preparation flow that turns source material into searchable records, and a query-time flow that finds and presents relevant records to the model.
As an Amazon Associate I earn from qualifying purchases.
| Stage | What happens | What can go wrong |
|---|---|---|
| Ingestion | Documents or other media are processed, divided into chunks, enriched with metadata, embedded where needed, and stored in a search index. | Important content is missing, poorly extracted, split away from its context, or difficult to filter and find. |
| Retrieval | An orchestrator searches the index and returns candidate passages for the query. | The relevant passage is absent from the index, missed by the search method, or buried among less useful results. |
| Context assembly | Selected results are combined with the user’s question and supplied to the language model. | Useful evidence is omitted, duplicated, or crowded out by irrelevant context. |
| Generation | The model produces an answer using the supplied context. | The answer misreads, ignores, or goes beyond the available evidence. |
Microsoft’s RAG architecture guidance describes these ingestion and query-time responsibilities as distinct parts of the system. That separation is practical: when an answer is wrong, the first question should be which stage failed—not simply whether the model needs a different prompt.
How to design the ingestion stage
Choose chunks that preserve useful meaning
Chunking is a content-design decision, not a setting with one correct value for every corpus. A chunk needs enough surrounding information to be understandable and retrievable, without turning into an unwieldy block that mixes unrelated ideas. The right approach depends on source format, document structure, the questions users ask, and the economics of indexing and retrieval.
#1 Best Overall
Microsoft’s guidance discusses sentence-based and fixed-size chunking, custom strategies, layout analysis, and model-assisted approaches. It also calls attention to cleaning, what content to include or exclude, and metadata enrichment. Treat those as options to compare on actual source material. Inspect extracted chunks from representative files: confirm that headings, definitions, tables, and references survive processing in a form the index can retrieve and a model can interpret.
Add context and metadata where they help retrieval
A passage can become ambiguous when separated from its original document. A short excerpt may refer to “the service,” for example, without preserving which service or period the source discussed. Anthropic’s Contextual Retrieval approach addresses this by adding context to chunks before retrieval. It can be worth testing when chunk boundaries strip away important entities or scope, but it adds an indexing step; compare its retrieval results and operating cost with the simpler alternative.
Rank #2
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
Metadata can provide another way to preserve and use context. Titles, summaries, and keywords may be indexed as distinct fields, making them available to search or filtering rather than burying every clue in a chunk’s body text. Validate that the metadata is accurate and useful for the questions at hand.
When to use lexical search, vector search, and reranking
Lexical and semantic retrieval address different kinds of matching. BM25 is a lexical ranking method that can match precise words or phrases. Vector search uses embeddings to find semantic similarity, which can help when a question and its supporting source use different wording. A pipeline can combine them, but adding components is justified only if the target workload benefits.
Rank #3
| Approach | Useful when | Trade-off to evaluate |
|---|---|---|
| Lexical search, such as BM25 | Questions contain exact names, identifiers, or technical phrases that need precise matching. | May not find a passage expressed with substantially different wording. |
| Vector search | A query may use different language from the source while referring to the same idea. | Semantic similarity alone may rank a passage that is conceptually related but not the precise evidence needed. |
| Hybrid retrieval | Both exact-term matches and semantic matches matter for the query set. | Requires merging and deduplicating results and evaluating whether the combined list improves useful coverage. |
| Reranking | The supporting passage appears among retrieved candidates but is not ranked highly enough for context assembly. | Adds latency and cost; a hosted service may also involve sending document content to an external provider. |
One hybrid pattern, described by Anthropic, runs lexical and vector searches, merges and deduplicates their results using rank fusion, then passes a top-K set onward. Microsoft’s retrieval guidance describes a related multistage pattern: retrieve a broader candidate pool, merge result lists—for example, with reciprocal rank fusion—rerank, then truncate to a smaller set for context. These are designs to test, not a required stack.
For reranking, measure whether the desired evidence moves into the final context set and whether answer quality improves enough to justify extra response time and expense. Microsoft’s guidance treats candidate-pool and final-result counts as tuning choices, not universal constants. Include your security and compliance requirements in the decision: check whether document content may be sent to a hosted reranking service.
Rank #4
How to evaluate retrieval separately from answers
A fluent answer is not proof that retrieval worked. OpenAI’s accuracy guidance distinguishes retrieval failures, in which the model receives incorrect or noisy context, from failures in model behavior. If the context is wrong or overloaded, generation cannot reliably repair the input. Evaluation should therefore make it possible to assess both what the system retrieved and what it did with those results.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
- Build a representative test set. Include real source formats and the kinds of queries your system is expected to handle. Retain examples that have caused failures so changes can be checked against them.
- Check the source and its parsing. For a failed answer, confirm that the supporting information exists in the knowledge source and survived extraction and cleaning.
- Inspect chunks and metadata. Look for lost headings, broken tables, missing scope, or inaccurate metadata that could make the evidence hard to retrieve or interpret.
- Inspect retrieval results. Determine whether the supporting passage was returned, whether irrelevant material crowded the results, and whether its ranking was suitable for context assembly.
- Inspect the final answer. If the context contained adequate evidence, check whether the answer used it correctly, remained grounded, and addressed the question completely and relevantly.
- Track outcomes across queries. Document the pipeline settings and results, and compare aggregate behavior as well as individual regression examples when a component changes.
Microsoft recommends evaluating retrieval and end-to-end response measures including groundedness, completeness, utilization, and relevance. These checks answer different questions: whether relevant material was found, whether the model used the supplied material, and whether the result served the user. NIST’s overview of the TREC 2025 RAG track illustrates another way to separate tasks: it describes retrieval, generation given fixed retrieved context, end-to-end RAG, and relevance judgments. Its generation task asks for sentence-level citations to supporting segments. That is a useful model for checking evidence attribution, not a requirement that every application use the same benchmark format.
Best Value
How to choose the simplest change that addresses a failure
OpenAI recommends reaching an accuracy target with simpler methods before adopting more complex RAG or fine-tuning. In practice, first classify the observed failure, then test the least complicated change that plausibly addresses it.
| Observed failure | Change to test |
|---|---|
| The needed source is missing or malformed in the index. | Fix source coverage, extraction, cleaning, chunk boundaries, or metadata before tuning retrieval. |
| Exact names or phrases are missed. | Test lexical search or a hybrid lexical-and-vector candidate list. |
| The query is vague or needs reformulation to search effectively. | Test query translation or decomposition against the original query set. |
| The right evidence is retrieved but ranks too low. | Test reranking and measure whether it improves final-context relevance and grounded answers. |
| A fixed single search cannot answer questions requiring several steps, dynamic source selection, or retrieval combined with actions. | Evaluate whether an agentic retrieval flow fits the workload and whether its added complexity earns its place. |
| The answer mishandles evidence that is already present in a suitable context. | Investigate generation behavior and context assembly rather than adding more retrieval machinery by default. |
Microsoft characterizes standard RAG as a fixed sequence—accept a query, search, assemble context, and call the model—and describes it as suitable when a query maps to one search against one index. Multistep reasoning, runtime query decomposition, dynamic source selection, or combining retrieval with actions may be reasons to consider agentic RAG. Those workload characteristics are a reason to evaluate it, not proof that an agent will improve accuracy.
When RAG may be unnecessary
Retrieval is not automatically the right answer to every knowledge-in-context problem. Anthropic notes that, in its context and prompt-caching discussion, a knowledge base smaller than 200,000 tokens—described there as about 500 pages of material—may be small enough to include in the prompt instead. Treat that as vendor guidance for a particular context, not a universal cutoff: whether full-context input works depends on the model, task, available context capacity, and operating constraints.
Anthropic also reports that its Contextual Retrieval approach can reduce failed retrievals by 49%, and by 67% when combined with reranking. These are Anthropic-reported results, not expected gains for another corpus or system. Measure any proposed change on your own representative query set before relying on a similar improvement.
What “better” means for your pipeline
Compare alternatives on the dimensions that matter to the application: relevance and coverage of retrieved evidence, exact-term matching, grounded answer quality, latency, cost, operational complexity, and privacy or security constraints. No universal chunk size, top-K, embedding model, reranker, or vendor is established as best for every RAG workload. A sound architecture is the simplest one that meets the measured accuracy need, and whose retrieval and generation behavior you can continue to evaluate as its sources and users change.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
