Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsLLM inference usually gets sharply slower when incoming work approaches the model server’s practical capacity. Requests then spend longer waiting for a scheduling slot, prompt processing competes with token generation, and memory or batching limits make each decode step less predictable. The right fix depends on where the delay appears: before the first token, between output tokens, or across the whole request.
Start by measuring time to first token (TTFT), per-output-token latency (TPOT or inter-token latency), end-to-end latency, and the number of requests waiting, running, or swapped. Recreate production prompt lengths, output lengths, arrival rates, and concurrency with a serving benchmark. Only then change scheduling, batching, model precision, parallelism, replica count, or serving topology.
As an Amazon Associate I earn from qualifying purchases.
Latency has three different phases
Calling a request “slow” hides the mechanism that needs fixing. An interactive response has a prompt-processing phase, a first-token boundary, and a repeated generation phase. Track latency distributions, especially p95 and p99, rather than relying on an average that can conceal queueing and outliers.
| Metric | What it measures | What a rise commonly suggests |
|---|---|---|
| Time to first token (TTFT) | Elapsed time from request admission until the first generated token is available. | Admission queueing, a large prompt, prefill contention, or a saturated replica. |
| TPOT or inter-token latency | Time used to produce subsequent output tokens. In vLLM’s documented definition, TPOT is calculated from end-to-end latency minus TTFT, divided by output tokens after the first; requests producing at most one output token record zero. | Decode batching pressure, generation concurrency, memory-bandwidth limits, or interference from other phases. |
| End-to-end latency | Total time from request arrival to completion. | A combination of queueing, prefill, decode cadence, and output length. |
| Waiting, running, and swapped requests | Counts of requests that are queued, actively executing, or temporarily moved because available execution or KV-cache capacity is constrained. | Admission pressure, scheduler backlog, or memory pressure. Interpret the counts with latency and token-length data. |
In a deployment using vLLM, the documented metric names include vllm:time_to_first_token_seconds, vllm:request_time_per_output_token_seconds, and request-state counts for RUNNING, WAITING, and SWAPPED. Names and availability can change between installed versions, so check the metrics reference that matches your server.
#1 Best Overall
- [Local AI Inference & 70B Model Ready] Equipped with the AMD Ryzen 7 PRO 8845HS processor, NEXUS is engineered for heavy local AI workloads. With a full-size GPU bay, it runs 70B LLMs natively without an internet connection. Ideal for AI developers and tech enthusiasts who need private environment for coding and model testing.
- [132TB Mass Storage with ZFS Integrity] Features a hybrid storage architecture (3×NVMe + 4×3.5" HDD) supporting up to 132TB. Utilizing the enterprise-grade ZFS file system and ECC memory, it prevents data corruption and bit rot—a must-have for professional photographers and video editors safeguarding 4K/8K RAW footage.
- [OpenClaw-Driven Automation Workflow] The built-in OpenClaw execution layer allows complex automated tasks to be processed locally. Even when offline, your backup schedules and AI file organization continue seamlessly. Say goodbye to monthly cloud subscriptions and high latency.
- [Dual 10GbE & USB4 Ultra-Connectivity] Experience server-class speeds with dual 10GbE ports and a 40Gbps USB4 interface. It enables multi-user real-time collaboration on large project files directly from the NAS, ensuring zero-lag editing for creative studios and production teams.
- [Open-Source ZimaOS for Total Privacy] Running on the fully open-source ZimaOS, NEXUS ensures your data stays physically on-premise with no backdoors. It acts as a "Digital Fortress" for privacy-conscious families and small businesses who demand absolute data sovereignty.
Read the shape before choosing a remedy
- TTFT rises while waiting requests rise: investigate admission pressure, prefill work, scheduler backlog, and replica saturation.
- TTFT stays acceptable but TPOT worsens: investigate decode batch size, active generation count, memory bandwidth, and interference during decode iterations.
- Both rise as concurrency increases: the workload may be at or beyond its saturation point, although prompt-length mix and KV-cache pressure must also be checked.
- Only long prompts are slow: compare input-token lengths and prefill behavior before treating the issue as a global capacity failure.
These patterns are hypotheses, not proof. A single metric cannot identify every bottleneck.
What saturation means in an LLM server
Every model-and-hardware combination has a workload-specific point at which adding request rate stops increasing useful throughput and starts increasing delay. The vLLM metrics documentation describes this as the inflection where the server cannot obtain more throughput from a higher request rate but begins to incur additional latency. The point is not defined by a universal GPU-utilization percentage.
At low load, a request can often enter the next scheduling iteration immediately. As arrivals rise, the scheduler has more prefills and decode iterations to fit into each iteration, while memory must retain the key-value cache for more active sequences. Once those resources are consistently occupied, new requests wait, active requests receive less predictable service, or requests are swapped to make room. Throughput may flatten even while queue time and tail latency continue climbing.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Why averages mislead
Suppose most requests complete quickly but a burst waits behind a long prefill. The mean can look healthy while p99 TTFT violates an interactive objective. Export request-level distributions and separate latency by input-token and output-token buckets. Record arrival timestamps, admission timestamps, first-token timestamps, completion timestamps, and the number of active sequences so that queueing can be distinguished from model execution.
Prefill and decode compete for different resources
The Sarathi-Serve paper presented the two basic phases precisely: prefill processes the entire input prompt and produces the first output token; decode generates the remaining output tokens one at a time. They place different demands on a shared accelerator.
Prefill is prompt-sized work
Longer prompts require more computation before the first token can be emitted. A newly admitted long prompt can consume substantial compute in one scheduling interval, delaying decode iterations for requests that are already streaming. This is why a rise in TTFT for long-input requests does not necessarily mean that token generation itself became slower.
Decode is repeated, batch-friendly work
Decode advances every active sequence by one token per iteration. It benefits from batching many sequences together, but each additional sequence consumes KV-cache memory and competes for memory bandwidth. Increasing the decode batch can improve aggregate tokens per second while making an individual token, and especially its tail latency, less predictable.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Interleaving creates a trade-off
A scheduler that gives a large prefill uninterrupted access to the accelerator can improve prompt-processing efficiency but stall ongoing streams. A scheduler that protects decode cadence can leave prefill work queued and push TTFT upward. Continuous batching, chunked prefills, and admission limits are attempts to balance those objectives; none is optimal for every prompt-length distribution or service-level objective.
Diagnose the spike with a controlled workload
Do not tune from a dashboard showing only total requests per second and GPU utilization. Reproduce the traffic shape that causes the incident and capture the same dimensions that drive work.
- Define the objective. Choose explicit limits such as p99 TTFT, p99 inter-token latency, p99 end-to-end latency, or a combination. If throughput matters, define goodput as the amount of work completed within those limits.
- Measure all three latency phases. Export TTFT, TPOT or inter-token latency, end-to-end latency, and request-state counts. Keep raw request-level samples so that long prompts and long generations can be analyzed separately.
- Describe the traffic mix. Record input-token and output-token distributions, arrival-rate pattern, concurrency, burstiness, cancellation rate, and the number of active sequences. A test with uniform short prompts is not a capacity result for a workload containing long documents.
- Record the serving configuration. Save model and revision, precision, accelerator type and count, serving framework and version, tensor or pipeline parallel settings, maximum sequence limits, batching and concurrency limits, caching behavior, and routing policy.
- Increase load gradually. Sweep request rate or concurrency and plot throughput against p50, p95, and p99 latency. The saturation region is where throughput stops improving while latency rises materially.
- Repeat each candidate change at the same load. Compare tail latency, goodput, token throughput, memory headroom, and model quality where precision changed. A faster median that violates the p99 objective is not an improvement for an interactive service.
Use a serving benchmark, not only a synthetic burst
The vLLM engineering guide describes vllm bench serve as a server-oriented benchmark that simulates request arrivals and measures serving metrics. Its latency-oriented and throughput-oriented modes represent different operating regimes. An all-at-once load test can reveal a maximum burst response, while a steady arrival-rate test is more useful for capacity planning; neither should automatically be presented as production capacity.
Include representative prompt and output lengths, arrival rates, concurrency limits, and model settings in every benchmark record. If your production traffic is bursty, test the burst and the recovery period. If requests stream to users, measure token cadence and cancellation behavior instead of measuring only completion time.
Use tracing carefully
Detailed tracing can locate delays between admission, scheduling, prefill, decode, and network delivery. However, vLLM warns that detailed trace collection can involve costly or blocking operations and may affect performance. Enable it briefly, on a controlled replica or test run, and compare against an untraced baseline.
Fix scheduler and batching interference
Once measurements show that scheduling or phase interference is responsible, tune for the latency objective rather than maximum raw throughput.
Control batch and concurrency limits
Lowering a maximum batch or active-sequence limit can reduce queue depth and improve p99 TTFT or TPOT, at the cost of lower utilization and throughput. Raising it can have the opposite effect. Sweep the setting while holding traffic constant, and retain the highest value that meets your latency objective with memory headroom.
Test continuous or chunked scheduling where supported
Continuous batching admits work as sequences progress instead of waiting for a whole batch to finish. Chunked-prefill approaches divide a large prompt into smaller scheduling units so that decode work is not blocked by one long prefill. Feature names and behavior depend on the serving engine and version; enable only options supported by the exact deployment and verify their effect with the representative benchmark.
Interpret published capacity gains correctly
Sarathi-Serve reported up to 2.6× higher serving capacity for Mistral-7B on one A100 GPU, up to 3.7× for Yi-34B on two A100 GPUs, and up to 5.6× for Falcon-180B with pipeline parallelism, compared with vLLM in the paper’s experiments. Those figures are results for the paper’s models, GPUs, workloads, scheduling policy, and service constraints. They are not guaranteed multipliers for another model or hardware configuration.
Reduce memory pressure and evaluate model optimizations
Memory pressure can force smaller batches, increase swapping, or prevent additional sequences from being admitted. Google Cloud’s GKE guidance discusses quantization, tensor parallelism, and model-memory optimization for GPU-backed serving with vLLM or TGI. Treat each as an experiment with both performance and quality acceptance criteria.
Quantization
Quantization represents weights at lower precision to reduce memory requirements. Depending on the method, model, accelerator, and serving framework, it can improve latency or throughput by reducing memory traffic and allowing more work to fit. It can also reduce accuracy, and not every method is compatible with every framework or kernel. Evaluate task quality on a fixed test set, then compare TTFT, TPOT, throughput, and tail latency under the same request mix.
Tensor and pipeline parallelism
Tensor parallelism divides model computation across accelerators. It may make a model fit or increase available compute, but communication overhead and topology matter. Pipeline parallelism can change capacity and memory placement while introducing its own scheduling behavior. Benchmark the actual accelerator count and interconnect; do not assume that adding a device produces a proportional speedup.
Other memory changes
Check KV-cache allocation, maximum sequence length, reserved memory, and fragmentation before buying hardware. A limit that is far above observed prompt and output lengths can consume capacity without helping real requests. Conversely, an overly restrictive limit can reject legitimate requests or force undesirable truncation. Change one setting at a time and watch swapped-request counts, out-of-memory events, and tail latency.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Scale from observed saturation
Use the measured throughput-versus-latency curve to decide when to add replicas or accelerators. A practical scaling signal combines queueing or request-state growth with TTFT and end-to-end tail latency, rather than relying on a fixed utilization threshold.
What to verify before adding replicas
- Routing balance: confirm that traffic is distributed evenly and that one replica is not receiving longer prompts or more active streams.
- Memory fit: verify that the model, KV cache, runtime, and safety margin fit on every target accelerator.
- Loading behavior: measure model-load time and readiness delay in the target environment. The documented sources do not establish a universal cold-start duration.
- Availability during a burst: a replica that becomes ready after the burst has passed cannot improve that incident.
- Shared dependencies: check tokenization, request gateways, network links, and observability systems so that scaling the GPU does not expose another bottleneck.
When adding capacity, repeat the same arrival-rate sweep. A second replica can improve queueing only if routing, model loading, and downstream services can use it effectively.
When prefill/decode disaggregation is worth testing
Disaggregated serving places prefill and decode on separate workers so that a large prompt is less likely to interrupt ongoing token generation. It can reduce high-tail inter-token latency for some workloads, but it adds a KV-cache transfer path between workers. That transfer can increase TTFT and become the new bottleneck.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Measure the transfer path first
- Measure available bandwidth and transfer latency between the prefill and decode workers under realistic concurrent transfers.
- Compare collocated and disaggregated p50 and p99 TTFT, inter-token latency, end-to-end latency, and goodput.
- Test both long prompts and short prompts, because transfer overhead can dominate small requests.
- Include bursty arrival patterns and worker failures in the operational test, not only a steady-state run.
A configuration-specific vLLM example
In a vLLM project test using two L40S GPUs, collocated p99 inter-token latency rose from 23 ms to 169 ms at 0.4 requests per second, while the disaggregated setup reported 29 ms at that rate. The same report also identified a TTFT penalty from KV-cache transfer. The model, prompt and output sizes, arrival setup, hardware, and transfer implementation were specific to that test, so the numbers are an example of a possible trade-off, not a promise for another deployment. The report also cites a separate result using different AMD hardware and workload assumptions, reinforcing that transfer and scheduling behavior must be measured locally.
When collocation is the better choice
Keep prefill and decode together when traffic is low or highly bursty, when TTFT is the dominant objective, or when the network cannot move KV cache quickly enough. Separation is most defensible when decode-tail interference is the measured problem and the transfer path has enough bandwidth and predictable latency.
Choose the remedy by bottleneck
| Observed bottleneck | First experiments | Main trade-off | Success criteria |
|---|---|---|---|
| TTFT and waiting queue rise with prompt length | Chunked or continuous prefill scheduling; prompt-length admission limits; more prefill capacity. | Protecting decode can leave prefill queued; extra replicas increase cost. | Lower p95/p99 TTFT without unacceptable decode-tail regression. |
| TPOT worsens at high active-sequence counts | Adjust decode batch and concurrency; inspect memory bandwidth and KV-cache headroom. | Smaller batches improve cadence but reduce throughput. | Inter-token p99 and goodput meet the streaming SLO. |
| Swapped requests or memory exhaustion | Reduce active limits or sequence caps; evaluate compatible quantization and memory settings. | Precision changes can affect accuracy; lower limits can reject or delay work. | No avoidable swapping or out-of-memory events at target load, with quality preserved. |
| Throughput plateaus while all latency phases rise | Add replicas or accelerators after checking routing and startup timing. | Hardware and operational cost; scale-out may arrive too late for short bursts. | Higher sustainable goodput and a wider latency-compliant operating range. |
| Decode-tail interference from long prefills | Benchmark prefill/decode disaggregation and KV-cache transfer. | Transfer overhead can increase TTFT and adds network complexity. | Lower decode-tail latency without violating TTFT or reducing goodput. |
Common mistakes that make spikes worse
- Using GPU utilization as the diagnosis: utilization can be high while the real issue is queueing, memory bandwidth, or an imbalanced replica.
- Benchmarking only short prompts: this hides prefill cost and produces an unrealistic capacity estimate.
- Optimizing average latency: interactive users experience tail latency, and a lower mean can coexist with a worse p99.
- Maximizing tokens per second at any cost: larger batches can improve throughput while violating a streaming latency objective.
- Applying a paper’s multiplier to a new deployment: published gains depend on model, hardware, workload, and scheduling details.
- Turning on detailed tracing during the incident: tracing overhead can alter the behavior being diagnosed.
- Disaggregating without measuring KV transfer: the network path can erase the benefit or increase TTFT.
- Changing several variables at once: you lose the ability to identify which change helped and which introduced a regression.
A practical incident runbook
- Confirm the symptom. Plot TTFT, TPOT or inter-token latency, and end-to-end p50, p95, and p99 for the affected interval.
- Check queue and memory state. Compare waiting, running, and swapped requests with active-sequence counts, accelerator memory, and input/output token lengths.
- Classify the phase. A TTFT-only increase points toward admission or prefill; a TPOT-only increase points toward decode; a rise in both suggests capacity or memory pressure.
- Reproduce the traffic shape. Use
vllm bench serveor an equivalent server benchmark with matching arrival pattern, concurrency, prompt lengths, and output lengths. - Apply the smallest relevant change. Tune scheduling for phase interference, memory or precision for fit, parallelism for compute capacity, replicas for sustained saturation, or disaggregation for measured decode interference.
- Validate quality and operations. Check output correctness after precision changes, routing balance after scale-out, transfer reliability after disaggregation, and recovery behavior after bursts.
- Keep the before-and-after record. Store configuration, workload, percentiles, goodput, throughput, and failure counts so the next capacity decision is based on comparable evidence.
Bottom line
Latency spikes are usually a saturation and interference problem, not a single defective latency setting. Separate TTFT from token cadence and end-to-end time, reproduce the real workload, and locate the queue, phase, memory, or transfer bottleneck. Then choose the least complex remedy that meets the service objective: scheduler and batch tuning for interference, quantization or parallelism for memory and compute constraints, scale-out for sustained saturation, and prefill/decode disaggregation only when its KV-cache transfer cost is demonstrably acceptable.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Free tools Windows power users keep installed
One-click scans. No signup required.
