Recommended Free Tools
A local AI model can feel slow for three different reasons: it takes a long time to process your prompt, it waits before showing its first token, or it generates tokens slowly after the answer begins. First identify which delay you have, then check the runtime’s CPU/GPU placement and memory diagnostics. Those checks often reveal whether a setting, model size, context length, or hardware limit is responsible.
Identify which part of inference is slow
Time to first token is the wait from submitting a request until the first piece of the answer appears. It can include prompt processing, model loading, and other startup work. Prompt processing is the work of reading the input context; generation is the work of producing the answer token by token. A model may be quick at one and slow at another, so “tokens per second” alone does not explain every frustrating delay.
As an Amazon Associate I earn from qualifying purchases.
- Long wait before the answer starts: check whether the model is loading, whether your prompt or context is large, and how quickly the runtime processes input.
- Slow progress after the first token: inspect CPU/GPU placement, memory pressure, and—when using llama.cpp—CPU thread settings.
- Slow only on long conversations or large documents: test a shorter context and a representative prompt to see whether the added input is driving the delay.
For useful comparisons, keep the model, quantization, runtime, hardware, context length, and prompt the same while changing one setting at a time. Results from different configurations are not clean benchmarks of a single setting.
Check where the runtime is doing the work
A detected GPU does not necessarily mean the entire model is running on it. Depending on the model and available memory, inference may run on the CPU, GPU, or a split across both. While a request is active, check the runtime’s own status or diagnostics for device placement and memory use.
#1 Best Overall
- Professional AI & Creator Workstation: AMD Radeon AI PRO R9700 GPU with 32GB GDDR6 is engineered for AI development, professional content creation, and compute-intensive workloads.
- Massive 32GB Memory Capacity: 32GB of GDDR6 memory on a 256-bit bus provides ample bandwidth for large AI models, 8K video editing, and complex 3D rendering.
- Advanced RDNA 4 with AI Accelerators: 64 Compute Units with 3rd Gen Ray Tracing and dedicated 2nd Gen AI Accelerators for groundbreaking AI performance and visual computing.
- Professional Blower Cooling: Efficient single blower design exhausts heat directly out of the chassis, ideal for multi-GPU workstation and server configurations.
- Enterprise-Grade Thermal Solution: Vapor chamber heatsink with industrial Honeywell PTM7950 thermal interface material ensures reliable cooling under sustained professional loads.
- Ollama: its FAQ describes how to check whether a model is running on CPU or GPU and how placement relates to available system memory or VRAM.
- llama.cpp: its token generation performance tips describe startup diagnostics for checking GPU offload.
If GPU use is missing or lower than expected, investigate whether the runtime build, backend, and device configuration support your GPU before changing unrelated performance settings.
Make sure the model and context fit
Model weights are only part of the memory requirement. Runtime state and the context being processed also use memory, and longer contexts can increase both memory use and prompt-processing time. If the model is partly placed on the CPU, or a larger context pushes it beyond available VRAM, performance may suffer.
Rank #2
- NVIDIA Volta GV100 Architecture — 4,608 CUDA Cores, 640 1st-Gen Tensor Cores delivering 14 TFLOPS FP32 and 112 TFLOPS deep learning performance for AI training, inference, HPC, and scientific computing workloads
- 32GB HBM2 ECC Memory — 900 GB/s Bandwidth — High-bandwidth memory on a 4096-bit bus with ECC error correction provides the memory capacity and throughput required for the largest AI models, simulations, and datasets
- PCIe 3.0 x16 Interface — 250W TDP — Standard PCIe Gen3 connectivity with passive cooling designed for enterprise rack server deployment in HPE ProLiant, Dell PowerEdge, and Supermicro platforms with adequate chassis airflow
- NVLink — Scale to 96GB Unified Memory — Connect two V100 GPUs via NVLink at 300 GB/s bi-directional bandwidth to scale GPU memory from 32GB to 96GB for larger AI training and HPC workloads
- Multi-Precision Computing — Supports FP64 (7 TFLOPS), FP32 (14 TFLOPS), FP16 (112 TFLOPS) and INT8 precision modes for flexible deployment across training, inference, and scientific simulation workloads
Try these changes one at a time:
- Use a smaller model if the current model cannot fit comfortably in the available memory.
- Try a supported quantized version. Quantization reduces model memory requirements, but it trades off quality and is not established as a universal speed winner. Check its output on the tasks you actually use.
- Reduce the context length to what the job needs, then test again with a representative prompt. Ollama documents context-length configuration in its FAQ; exact memory needs depend on the model and setup.
NVIDIA’s local AI overview also discusses VRAM planning and quantization. A GPU upgrade is worth considering only after diagnostics show that GPU acceleration or memory capacity is the constraint. The memory needed depends on model weights, context, runtime overhead, and other GPU workloads; there is no universal VRAM target that fits every local model.
Test CPU threads if token generation is unexpectedly slow
With llama.cpp, CPU thread settings can affect generation speed even when GPU acceleration is involved. Its guidance recommends trying a thread count of one as a diagnostic. If that improves generation, an oversubscribed thread setting may be hurting performance; the documented next step is to set the count to the number of physical CPU cores.
Rank #3
- Built for Running LLMs Locally: RDNA 4, 128 AI Accelerators, up to 1,531 TOPS (INT4) for fast inference and fine-tuning
- 32GB GDDR6 VRAM for Large AI Models: 256-bit, up to 640GB/s bandwidth, run large language and multi-modal AI models without offloading
- Multi-GPU Scaling for Local AI Clusters: PCIe 5.0 and 2-slot design support dense multi-GPU builds for local AI training and inference clusters
- Diecast Shroud and Backplate: Wave-pattern design cuts memory temperature by up to 16%, keeping clocks steady during long AI training runs
- Phase-Change GPU Thermal Pad: Delivers superior thermal conductivity for consistent performance and longevity under heavy AI loads
- Record the current thread setting and test a short, representative prompt.
- Set the thread count to one and repeat the same test.
- If generation improves, set the count to the physical core count and compare again.
This is a llama.cpp-specific diagnostic, not a universal instruction to use one thread or a fixed thread count in every runtime. See the llama.cpp performance tips for its guidance.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Do not confuse serving throughput with interactive speed
Batching, in-flight scheduling, KV caching, quantization, and speculative decoding can help certain serving workloads, particularly when an accelerator handles multiple requests. But higher throughput across requests does not automatically mean lower latency for one person using a desktop model. These are runtime- and workload-specific techniques, not generic fixes for an individual slow response.
Rank #4
- 24GB GDDR7 ECC Memory: handles large AI, 3D and rendering files smoothly
- Powerful CUDA Compute - 8,960 CUDA cores for fast graphics and computing power
- AI & Ray Tracing Boost - Tensor of the 5th generation and RT cores of the 4th generation
- PCIe 5.0 x16 interface - fast data connection with modern systems
- 4 × DisplayPort 2.1 - Multi-monitor support for professional workflows
For scale, NVIDIA reported roughly 150 tokens per second on an RTX 4090 running Llama 3 8B with 100 input tokens and 100 output tokens in its llama.cpp and RTX article. That is a vendor-reported result under those conditions, not a typical-speed promise or a comparison that predicts performance on other hardware.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →In a separate specialized serving example, NVIDIA reported speculative-decoding throughput speedups of 3.55x, 3.16x, and 2.63x on a single H200 for Llama 3.3 70B, using Llama 3.2 1B, Llama 3.2 3B, and Llama 3.1 8B as draft models, respectively. Those figures concern a specific GPU-serving setup, not consumer-PC inference. The details are in NVIDIA’s TensorRT-LLM speculative decoding article.
Quick Recap
Use this order before spending money
- Classify the delay: note whether it occurs before the first token, while the prompt is being processed, or between generated tokens.
- Inspect runtime diagnostics: confirm CPU/GPU placement and memory use while a request runs.
- Check fit: if placement or memory indicates a constraint, test a smaller model, supported quantization, or shorter context.
- Test the relevant setting: for slow llama.cpp generation, compare thread settings using the same prompt and model.
- Upgrade only for a diagnosed bottleneck: consider a GPU with enough VRAM for your actual model and context if GPU memory or partial CPU placement is the problem. System RAM may matter for CPU inference or system-memory limits, but adding it does not inherently speed up a GPU-bound workload.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
