What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
A local AI model can feel slow for several different reasons: the prompt may take too long to process, the first token may be delayed, generation may produce too few tokens per second, or concurrent requests may compete for memory and compute. The fix depends on which phase is slow. Start by measuring those phases, verify that your intended accelerator is actually active, then tune threads, batching, context size and offload settings one variable at a time.
The command examples below use llama.cpp terminology and flags. Other runtimes expose different settings, so treat the procedure as a diagnostic method rather than a universal command-line recipe.
As an Amazon Associate I earn from qualifying purchases.
What “slow” means in local inference
Do not judge performance only by the time between pressing Enter and seeing a complete answer. Local inference has separate stages, and each points to a different cause.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →| Symptom | What to measure | Likely area to investigate |
|---|---|---|
| Long pause before the first token | Prompt-processing time, time to first token | Prompt length, batch size, backend acceleration, memory pressure |
| First token arrives promptly but text streams slowly | Generation throughput in tokens per second | CPU thread count, GPU offload, thermal or power limits, model size and quantization |
| Short prompts are fine but long conversations crawl | Prompt time as context grows, memory use | Context allocation, cache size, available RAM or VRAM |
| One request is acceptable but several users are slow | Per-request throughput and aggregate throughput under concurrency | Memory bandwidth, scheduler behavior, batching and backend support |
Record the model name and parameter size, quantization, runtime and exact build, operating system, CPU, GPU, available system and graphics memory, context length, prompt size, thread count, GPU-layer setting and any batch parameters. Repeat the same prompt and generation length under comparable conditions. If your runtime reports prompt and generation timings separately, retain both values.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
Measure a baseline before changing settings
Create a reproducible test
Use a fixed prompt, fixed context length and fixed number of generated tokens. Close unrelated workloads, allow the machine to reach its normal operating temperature, and run several repetitions. A single fast or slow run can reflect compilation, cache state, background activity or thermal behavior rather than a setting change.
For llama.cpp, llama-bench is designed for repeated, structured tests. Its reports include the machine, backend, model, configuration and timing or throughput fields. Keep the report with the command line and software version so a later comparison has the same conditions. The README includes a sample record timestamped 2025-04-24; that timestamp identifies the example record, not a current performance target.
Separate the two workloads
- Prompt processing (prefill): the runtime reads the input context and builds the internal state. Long prompts can make this phase dominate time to first token.
- Token generation (decode): the runtime predicts one new token at a time. This is the phase normally represented by a tokens-per-second figure while text is streaming.
Compare like with like. A change that improves prompt processing may leave generation unchanged, and a change that raises generation throughput may not reduce the initial delay.
Use a one-variable test matrix
- Run the baseline three or more times and record median prompt and generation results.
- Change only one setting, such as thread count or GPU-layer offload.
- Repeat the same runs with the same model, prompt, context and output limit.
- Keep a setting only if the improvement is repeatable and does not cause errors, quality problems or unacceptable memory use.
Verify that the intended accelerator is really being used
A high GPU-layer limit or a GPU option in a launcher does not prove that useful work is executing on the GPU. First confirm that the runtime was built with the backend your hardware requires, then inspect startup diagnostics.
CUDA with llama.cpp
For an NVIDIA device, use a llama.cpp executable built with CUDA support and select the intended device. At startup, inspect the log for diagnostics showing layers moved to the GPU and the amount of VRAM used. Those lines are the practical evidence that offload occurred. If they are absent, check the build, device visibility, selected device, driver installation and whether the model plus its working memory can fit.
Do not proceed to fine-tuning thread counts while the program is silently running on the CPU. A CPU fallback can make every later comparison misleading. Rebuild or reinstall the correct backend, restart the process and confirm the diagnostic again.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Metal and SYCL paths
llama.cpp documents Metal support for Mac GPUs and SYCL support for Intel GPU classes, in addition to CUDA for NVIDIA GPUs. The executable, compiler options and device-selection details differ by platform. Confirm the backend shown at startup and verify that the model loads onto the expected device; do not assume a CUDA flag applies to Metal or SYCL.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsOpenVINO devices
The OpenVINO backend documentation covers Intel CPUs, GPUs and NPUs. It reports validation on specified Intel Core Ultra Series 1 and Series 2 systems, while broader model and quantization coverage remains work in progress. Check that your model format, device, runtime build and driver combination is supported before interpreting a speed result.
Tune CPU threads instead of assuming more is better
Generation is not guaranteed to accelerate when more CPU threads are enabled. Threads can contend for cache and memory bandwidth, compete with GPU work or increase scheduling overhead. llama.cpp’s token-generation guidance recommends starting with one thread, doubling the count until a bottleneck appears, then backing down. In its words: If in doubt, start with 1 and double the amount until you hit a performance bottleneck, then scale the number down.
- Run generation with one thread and record tokens per second.
- Try two, four, eight and higher counts that your CPU can provide, keeping every other setting fixed.
- Stop when throughput stops improving, falls, or becomes unstable.
- Select the best repeatable result, often the count just before the bottleneck rather than the maximum available.
The documented llama.cpp example found its best listed result at four threads, but that is not a universal recommendation. It came from a specific 30B four-bit model on an NVIDIA A6000 system with seven physical CPU cores and 32 GB of RAM.
Fix prompt delays with prompt-processing settings
If the first token is slow but subsequent tokens stream at a reasonable rate, investigate prefill rather than generation threads. llama.cpp build documentation says BLAS may improve prompt processing when batch sizes are above 32. It explicitly does not claim a generation-speed improvement.
What to test
- Compare a build with the relevant BLAS support against one without it, using the same model and prompt.
- Test batch sizes on both sides of the documented threshold, including the value appropriate to your memory budget.
- Measure prompt-processing time and time to first token separately from decode tokens per second.
- Watch memory use: a larger batch can improve prefill but may leave less headroom for context and concurrent requests.
Do not report a faster first response as a faster model overall unless generation throughput also changed. For chat workloads with short prompts and long answers, decode may dominate; for document summarization with large prompts, prefill can dominate.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Control context length and memory pressure
Every additional context position consumes memory for the model’s working state. Allocating the maximum context supported by a model when your application uses only a fraction can waste memory and reduce performance. Set a context length that covers the real prompt plus the expected output, with a safety margin, and increase it only when needed.
OpenVINO NPU caveat
The OpenVINO documentation identifies a specific risk on NPU execution: resolving the context to the model’s full training context can cause an out-of-memory failure or degraded performance. It suggests setting an explicit context, with -c 1024 shown as an example workaround. The value 1024 is documentation-specific, not a universal recommendation; choose a size that fits your application and device.
Recognize memory symptoms
- Loading succeeds but generation becomes erratic as the conversation grows.
- The operating system starts swapping or the GPU driver reports allocation failures.
- Throughput falls after increasing context, batch size or the number of simultaneous requests.
- The runtime silently places some work on the CPU because the selected device cannot hold the required layers and cache.
Reduce context or batch size, lower concurrency, or use a smaller model only after confirming which allocation is exhausting memory. Keep model weights, runtime buffers and context cache in the same accounting; free VRAM reported after loading weights is not necessarily enough for a long context.
Free tools Windows power users keep installed
One-click scans. No signup required.
Understand how model size and quantization affect speed
A larger parameter count generally requires more memory movement and computation, while lower-bit quantization reduces storage and bandwidth requirements at a possible quality cost. The available evidence does not establish one universally best quantization or a cross-hardware buying ranking. Compare quantized variants on your own prompts and check that the chosen backend supports the format.
When comparing two models, hold the runtime, context, prompt, output length and device constant. A smaller model that fits entirely on an accelerator can outperform a larger model whose layers spill between CPU and GPU, even if the larger model has a nominally higher accelerator offload setting.
What the llama.cpp benchmark example actually shows
The rolling llama.cpp troubleshooting documentation includes a configuration-specific illustration: a 30B-parameter model in four-bit GGML format, an NVIDIA A6000 with 48 GB of VRAM, seven physical CPU cores and 32 GB of system RAM. Reported outcomes for different thread and GPU-offload flags were 1.7, 5.5, 8.7 and 9.1 tokens per second. The documented -t 4 -ngl 2000000 run reports 9.1 tokens per second.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
| Reported figure | What it represents | How to use it |
|---|---|---|
| 1.7, 5.5, 8.7, 9.1 tokens/s | Different settings on the stated A6000/30B Q4 configuration | Evidence that settings can materially change throughput |
| 48 GB VRAM | Graphics memory in that example | Describes that machine, not a requirement for every 30B model |
| Seven physical CPU cores; 32 GB RAM | Host hardware in that example | Required context for interpreting its results |
These numbers are not expected speeds for another computer. They do not predict the result of a different model, quantization, driver, context length, operating system or runtime build. Reuse the method, not the headline number.
Diagnose common slow-inference patterns
“The GPU option is enabled, but the model is still slow”
- Read the startup log for actual layer-offload and VRAM-use diagnostics.
- Confirm the executable was built with the correct backend and that the intended device is visible.
- Check whether the model and context fit; partial or failed offload can leave expensive layers on the CPU.
- Only then test thread counts and batch settings.
“The first response takes a long time, then generation is fine”
Measure prompt-processing time. Test BLAS-enabled prompt processing and batch sizes above 32 in llama.cpp, while monitoring memory. Keep the context no larger than required. Do not expect these changes to increase decode tokens per second automatically.
“Generation is slow even with a short prompt”
Verify backend use, then run the one-thread, doubling experiment. Check CPU and GPU utilization, thermal throttling and memory pressure during a repeatable test. If the model is partly offloaded or constantly moving data between devices, a smaller model that fits one device may be faster.
“Performance collapses as the chat gets longer”
Compare a short and long context with the same output length. If prompt time and memory use rise sharply, reduce the configured context to the workload’s actual requirement and avoid allocating the model’s full training context by default. On an OpenVINO NPU path, an explicit context setting is specifically documented as a workaround for out-of-memory or degraded performance.
“One user is fast, several users are not”
Benchmark concurrency separately from single-request speed. Multiple requests share memory bandwidth, cache and accelerator capacity; a setting that wins for one stream may lose when requests overlap. Record per-request latency and aggregate tokens per second, and test a concurrency level your application actually needs.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11When a hardware upgrade is justified
Consider new hardware only after the software path is verified. A faster accelerator cannot fix a CPU-only build, an unsupported model format, an allocation that forces device spillover or an oversized context that exhausts memory.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
- Upgrade the accelerator path when the model fits a compatible GPU backend and measured generation is limited by available compute or memory bandwidth.
- Add memory when the model, context cache and runtime buffers do not fit without swapping or failed allocations.
- Choose a smaller or more suitable model when the current model cannot fit the device efficiently or its quality is unnecessary for the task.
- Improve software first when logs show no offload, thread scaling has not been tested, or prompt and generation phases have not been separated.
Compare candidate systems using compatibility with your model and operating system, confirmed backend activation, memory fit, prompt versus generation performance, quantization behavior and reproducible results on your own workload. The documentation available here does not support a universal card recommendation or a guaranteed speedup.
A practical end-to-end checklist
- Write down model, parameter size, quantization, runtime version/build, operating system, CPU, GPU, RAM, VRAM, context and prompt length.
- Run repeated baseline measurements and save structured output from your benchmark tool.
- Label each result as prompt-processing, time-to-first-token, generation throughput or concurrent throughput.
- Confirm the expected backend in startup logs and fix build or device-selection problems.
- Test CPU threads from one upward, doubling until the bottleneck, then select the best stable count.
- If first-token latency is the problem, test BLAS and batch settings in a llama.cpp build, remembering the documented threshold is above 32 for prompt processing.
- Reduce unnecessary context and verify memory headroom for weights, cache and runtime buffers.
- Change one variable per run, repeat it, and retain only improvements that survive the same workload.
- Re-evaluate hardware only after these checks show a genuine compute or memory limit.
Frequently Asked Questions
How can I tell whether llama.cpp is using my GPU?
Check the startup diagnostics for lines reporting GPU-offloaded layers and VRAM use. A configured GPU-layer limit by itself is not proof of active acceleration; the executable must also be built with the appropriate backend and see the intended device.
Will increasing the thread count always make generation faster?
No. llama.cpp recommends starting at one thread, doubling until performance reaches a bottleneck, then reducing the count. The optimum depends on the model, CPU, memory system and whether GPU work is occurring at the same time.
Recommended Free Tools
Does BLAS make every token generate faster?
Not according to llama.cpp’s build documentation. BLAS may improve prompt processing with batch sizes above 32; it does not establish an improvement in token-generation performance.
Why can a large context make an NPU run fail?
OpenVINO documents that resolving an NPU context to the model’s full training context can cause out-of-memory failure or degraded performance. An explicit context setting, such as the documented -c 1024 example, can avoid that allocation when it matches the workload.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
