There is no universal millisecond budget for vector search or reranking. Set stage limits from the response time your users need, measure the actual request path under representative load, and decide in advance what the service should do when an optional stage runs out of time. In a sequential RAG request, an early delay leaves less of the end-to-end deadline for later work, including generation.
What should a RAG latency budget protect?
Start with the user-visible objective, not a target copied from another system. For an interactive service, that might be time to first token (TTFT), complete-answer time, or both. TTFT measures when the user first sees output; full response time measures when the answer is complete. One can meet its target while the other misses, so track both if both matter to the experience.
![Computer Networking Bible: [3 in 1] The Complete Crash Course to Effectively Design, Implement and Manage Networks. Including Sections on Security, Performance and Scalability](https://m.media-amazon.com/images/I/41H4YJnxKgL._SL160_.jpg)
As an Amazon Associate I earn from qualifying purchases.
Map the request path your service actually runs. Depending on the design, it can include query rewriting, remote query embedding, vector search, lexical search, rank fusion, reranking, context assembly, and generation. Some stages may run in parallel; others wait for an earlier result. Record which stages are present and which are required before the next stage can start.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →For each request, capture stage durations alongside end-to-end TTFT and full response time. Inspect p50, p95, and p99—or the percentiles used by your service-level objective—rather than relying on an average. Also segment results by query class, candidate count, corpus or index, concurrency, and cold versus warm conditions. A single aggregate can hide a slow query class or a latency spike caused by queueing.
#1 Best Overall
- Measure queueing, network time, and model or database work separately where instrumentation permits.
- Track errors, timeouts, cancellations, fallbacks, partial results, concurrency, and context size with the latency distributions.
- Use an overall request deadline. Set bounded child deadlines for stages so that a child call cannot outlive its parent request.
- Choose and document a headroom policy, then revisit budgets when traffic, models, indexes, or retrieval settings change.
Do not add stage medians and assume their sum will meet a tail-latency objective. A request’s tail can be driven by several stages running slowly on the same request. With fan-out retrieval, completion can depend on the slowest branch that the implementation requires. Benchmark the real query path and its behavior under load, including queueing and saturation.
How should you measure latency by stage?
Instrument spans for every enabled stage and correlate them with the parent request. NVIDIA’s RAG blueprint recommends using span durations to compare slow and fast requests and identify which stage contributes most to latency, such as retrieval versus generation. It names retrieval_time_ms, context_reranker_time_ms, llm_ttft_ms, llm_generation_time_ms, and rag_ttft_ms as example stage and end-to-end metrics; use equivalent names if your deployment differs. See NVIDIA’s Query-to-Answer Pipeline documentation.
- Export histograms for retrieval and reranking as well as TTFT and full-response time.
- Record candidate count and context size with each relevant request so that latency can be compared at similar work levels.
- Separate model or database compute from queueing and network time where possible; otherwise, a slow span may be misdiagnosed.
- Alert on budget exhaustion by stage and track whether a timeout caused a fallback, cancellation, partial result, or failed request.
- Whenever you change approximate-nearest-neighbor settings, filters, candidate count, or reranker model, evaluate retrieval quality and latency together on representative queries.
Use the measured distributions to set stage budgets within the overall deadline. A stage budget is an operational control, not a promise that every request will finish in that time. Review the resulting tail behavior and headroom under representative concurrency, and adjust the allocation if the end-to-end objective is missed.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →When does reranking justify its latency?
Retrieval and reranking solve related but different problems. Retrieval aims to produce a candidate set with adequate recall: relevant evidence needs to appear among the candidates. A reranker then reorders that set, potentially improving which evidence reaches the final context. A reranker adds another stage and may require additional inference, so its relevance gain must justify its latency and resource cost.
Microsoft describes cross-encoder reranking as evaluating the query and candidate text together; compared with simpler independent encodings, it can improve relevance at higher latency. Its Azure guidance states, “Reranking adds more latency to your search pipeline than standard, vector, or hybrid searches.” See Microsoft Learn’s Information-Retrieval Phase guidance.
Whether that cost is worthwhile depends on the candidate set and task. If retrieval already returns a small, relevant set, reranking may change little. If the collection is noisy or varied, or retrieval is deliberately broad to protect recall, a query-aware reranker may improve precision. Test the actual effect using representative queries and relevance judgments, including whether required evidence survives into the final context. A reranker score is a relative ordering signal; do not treat it as a calibrated relevance probability or use an acceptance threshold without calibrating it on local data.
Bound the amount of reranker work
Limit the number of retrieved candidates sent to the reranker, then test how the limit affects both evidence quality and latency. Elastic’s ES|QL reference explicitly recommends a LIMIT around RERANK to control how many documents are processed. That is an implementation-specific way to bound reranker work, not a universal syntax for every retrieval stack. See Elastic’s ES|QL RERANK command reference.
Compare complete retrieval paths
Use the same representative query set when comparing vector-only retrieval, hybrid lexical/vector retrieval with rank fusion, and hybrid retrieval followed by a cross-encoder. Microsoft documents hybrid retrieval using Reciprocal Rank Fusion and discusses cross-encoder reranking as an additional stage. These are alternatives to evaluate, not a universally superior sequence.
- Compare retrieval and answer relevance or recall, including whether required evidence reaches the final context.
- Compare p50, p95, and p99 for retrieval, reranking, TTFT, and complete responses.
- Record candidate count, concurrency, request cost, and model or inference resource use.
- Test behavior during queueing or service saturation, including what remains usable if reranking times out.
How do you prevent a timeout cascade?
A timeout cascade is a useful name for an engineering failure pattern, not a claim that every RAG service has the same propagation behavior. In a sequential request with one finite end-to-end deadline, extra time spent in embedding, search, or reranking reduces the time available for downstream work. Generation can therefore time out even when each component looks healthy when tested against its own isolated timeout.
Implement deadline handling across the actual request path: pass the remaining parent budget to child calls, bound each stage, and propagate cancellation when the request expires. A per-call timeout must not silently become more time than the user-facing request can afford. Distributed traces can help distinguish queueing, network delay, service execution, and model time when diagnosing where the budget was consumed.
Choose the degradation behavior before an incident
Decide what the service should return when an optional stage exhausts its budget. For example, if the initial retrieval ranking is usable, the service may bypass a timed-out reranker and continue with that ranking. If retrieval has produced only some results, it may return partial evidence when the product can safely support that behavior. If neither fallback is acceptable, fail the request explicitly. Validate the choice against the implementation: not every pipeline can cancel cleanly or produce a useful partial result.
Free tools Windows power users keep installed
One-click scans. No signup required.
Reserve enough remaining time for the user-visible response path according to the service objective. That does not mean assigning the same fixed share to generation in every workload; it means deriving child deadlines from the remaining request budget and stopping work when the parent deadline has expired.
Best Value
What published latency figures can—and cannot—tell you
Published numbers are useful for understanding one documented setup, not for setting a portable service-level objective. NVIDIA’s 2025 enterprise RAG guide gives the following example latency shares and scaling thresholds. They are guide-specific sizing examples, not industry-wide targets or recommended budgets for an unrelated workload.
| Stage | Example share of TTFT in NVIDIA’s 2025 guide | Example scaling threshold in the same guide |
|---|---|---|
| LLM | 70%–90% | Not stated for this stage in the cited threshold examples |
| Reranking | 5%–20% | Above 10% of TTFT |
| Embedding | 3%–12% | Above 5% of TTFT |
| Vector database search | 1%–5% | Above 2% of TTFT |
Both the share ranges and thresholds are examples from NVIDIA’s RAG Scaling Guidelines. Do not interpret a scaling threshold as an SLO or as proof that a service is unhealthy below it.
The same NVIDIA guide’s summary gives a Chat baseline example: Milvus under 50 ms, embedding under 30 ms, reranker under 100 ms, LLM prefill around 1,500 ms, and LLM decode around 3,800 ms. These figures describe that guide’s stated Chat baseline; they are not universal per-stage limits. The source is NVIDIA’s Enterprise RAG Retrieval summary.
For a different kind of example, the authors of a 2017 paper report that, on the standard ClueWeb09B collection and 31,000 queries, their hybrid system achieved a maximum query time of 200 ms with a 99.99% response-time guarantee without a significant loss in overall effectiveness. That is a result for the paper’s benchmark and system, not a general RAG guarantee. See Efficient and Effective Tail Latency Minimization in Multi-Stage Retrieval Systems.
Which product-specific timeout details should not become your budget?
Product defaults and controls describe a particular implementation. For example, Elastic documents a 30-second default timeout for ES|QL’s RERANK behavior and a per-call timeout option. That default may change, and it is generally much longer than an interactive RAG stage budget should be; do not copy it as a recommendation for your service. Check the current ES|QL RERANK reference for the product behavior.
Cloudflare’s AI Search documentation says reranking is disabled by default for its AI Search instances and that enabling it adds a step that may increase latency. That is a Cloudflare platform-specific default, not a general default for RAG systems. See Cloudflare’s reranking documentation.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
