Why Is My Local Coding Model So Slow? How to Improve Inference Speed

A local coding model can feel slow for four different reasons: it takes too long to load, waits before producing its first token, processes a large prompt slowly, or generates tokens slowly once it starts. Measure those stages separately. Then check whether the model is actually using your GPU, whether CPU threads are oversubscribed, and whether context length is consuming too much memory. Only after those checks should you change the model, quantization, runtime, or hardware.

First identify which part is slow

“Slow” is not one performance problem. A long pause before the model is ready points to loading; a pause after you submit a prompt can reflect prompt processing; slow streaming after the first token is generation speed. A long repository or conversation context can make prompt processing and memory pressure more noticeable even if short prompts feel fine.

#	Preview	Product	Price
1		GMKtec EVO-X2 AI Mini PC Ryzen Al Max+ 395 Superchip 128GB LPDDR5X 2TB SSD	$3,649.99	Buy on Amazon

As an Amazon Associate I earn from qualifying purchases.

Use the same prompt, model file, context setting, runtime, and machine when comparing changes. Record time to first token, prompt-processing rate if your runtime reports it, and decode tokens per second. Also note model quantization, runtime version, and whether the run was cold or the model was already loaded. Without those details, a tokens-per-second number is difficult to compare meaningfully.

Check whether the model is using the GPU

Before adjusting other settings, verify where inference is running. A GPU may be installed but not receiving model layers, or only some layers may be offloaded. The runtime’s startup output or status command can reveal this.

#1 Best Overall

GMKtec EVO-X2 AI Mini PC Ryzen Al Max+ 395 Superchip 128GB LPDDR5X 2TB SSD

• EVOLUTION RYZEN AI MAX+ 395 MINI PC - GMKtec EVO-X2 is the next evolution in AI mini PC Ryzen Strix Halo series. Thanks to AMD Simultaneous Multithreading (SMT) the core-count is effectively doubled, to 32 threads. Ryzen AI Max+ 395 has 64 MB of L3 cache and can boost up to 5.1 GHz, depending on the workload. The Ryzen AI Max+ 395 is currently rated as the "most powerful x86 APU" on the market for AI computing.
• AI NPU with XDNA 2 ARCHITECTURE - Powered by 16 “Zen 5” CPU cores, 50+ peak AI TOPS XDNA 2 NPU and a truly massive integrated GPU driven by 40 AMD RDNA 3.5 CUs, the Ryzen AI MAX+ 395 is a transformative upgrade and delivers a significant performance boost over the competition. The Ryzen AI Max+ 395 excels in consumer AI workloads like the llama.cpp-powered application: LM Studio. Shaping up to be the must-have app for client LLM workloads, LM Studio allows users to locally run the latest language model without any technical knowledge required and unleash their creativity and productivity.
• AMD RADEON 8090S iGPU GAMING PC - The AMD Radeon RX 8060S offers all 40 CUs with up to 2.9 GHz graphics clock and uses the new RDNA 3.5 architecture. The powerful iGPU is positioned between an RTX 4060 and 4070 laptop GPU and therefore enables gaming in FHD at maximum details in most demanding games. The 8060S can also utilize the full 128GB pool, which is perfect for running LLMs such as Deepseek 70B Q8, which runs comfortably on this machine.
• EIGHT CHANNEL LPDDR5X - LPDDR5X is a new ground breaking memory small form factor installed on-board. With blazing speeds up to to 8000MT/s, it runs 1.5x faster than the DDR5 SODIMMs; 90% better performance over DDR5 SODIMMs in video conferencing and photo editing; 30% better performance in productivity apps; 12% better performance in digital content workloads.
• QUAD SCREEN 8K DISPLAY SUPPORT - EVO-X2 AI Mini PC support 4-screen 4K/8K output via HDMI 2.1 (8K@60Hz), DisplayPort 1.4 (4K@60Hz), and dual USB 4 40Gbps Transfer speed (supporting PD3.0/DP1.4/DATA). Ideal for gaming, video editing, and multitasking, it provides expansive and crisp multi-display support.

$3,649.99

Buy on Amazon

llama.cpp

Inspect the startup diagnostics for GPU-offload messages and the number of layers placed on the GPU. The -ngl or --n-gpu-layers option requests GPU layer offload; setting it high requests the maximum possible, subject to available resources. Check the actual startup result rather than assuming the requested value was achieved. See the llama.cpp token-generation performance guidance.

Ollama

Run ollama ps while the model is loaded and inspect the processor field. It reports whether processing is on the GPU, CPU, or split between them. If the result is not what you expected, troubleshoot placement before buying hardware or tuning generation settings. The Ollama FAQ documents this command and its output.

If generation is slow, test CPU thread settings

More CPU threads are not automatically faster. llama.cpp warns that a thread count that is too high can oversaturate the CPU. Its troubleshooting advice is to try one thread, then increase gradually until performance stops improving or a bottleneck appears, and scale back. The -t or --threads option controls this setting. Treat the result as specific to your machine and workload, not a universal optimal thread count.

llama.cpp illustrates why placement and thread count should be tested together. In its project-reported benchmark, a 30B Q4_0 GGUF model ran on an A6000 with 48 GB of VRAM, a seven-physical-core CPU, and 32 GB of RAM. The reported generation rates were:

These are results from that documented setup, not a prediction for another GPU, model, quantization, or runtime. The useful lesson is to test offload and threads rather than assuming either more threads or a GPU setting is working as intended. Source: llama.cpp performance troubleshooting.

Reduce context and memory pressure when they are the bottleneck

Context length affects memory requirements, and larger contexts can make long coding prompts expensive to process. Ollama’s current FAQ describes a default context length of 4096 tokens and configuration options for changing it. A larger context is useful only when the task needs it; try a shorter prompt or smaller context and compare the stages you measured.

For supported Ollama configurations, Flash Attention and K/V cache quantization can reduce cache memory use. Ollama describes q8_0 as using about half the memory of f16 with very small precision loss; q4_0 uses about one quarter of f16 memory, with small-to-medium loss that may be more noticeable at larger contexts. Effects on output quality depend on the model and task, and may be larger with some grouped-query attention layouts. These memory ratios do not guarantee a particular speed increase. Consult the Ollama FAQ for supported settings.

Parallel requests also multiply context allocation in Ollama. If memory fills up when several requests run at once, reduce concurrency or context allocation before concluding that the model itself is too large. Leave memory headroom for the model weights and other runtime needs.

If the delay is before generation, keep the model loaded

Repeated model loading can make the first request after a pause feel slow. Ollama’s FAQ says its default keep-alive period is five minutes; it also documents preloading through an empty request and controls for changing keep_alive. Keeping a model resident can reduce repeated load waits. It does not, by itself, establish that steady-state token generation will be faster. If streaming remains slow after the model is loaded, continue with placement, thread, context, and model checks.

Choose settings for interactive coding or multi-user serving

One person editing code

For interactive work, low latency—especially time to first token and the time between streamed tokens—may matter more than aggregate throughput. Test changes with a representative coding task and the context sizes you actually use. A setting that maximizes total tokens across concurrent requests may make one person’s response feel slower.

Multiple concurrent requests with vLLM on CPU

vLLM’s CPU tuning guide says larger batches usually improve throughput, while smaller batches tend to reduce latency. Start from the defaults and tune on the target platform. The guide also warns that CPU KV-cache memory plus model-weight memory must fit within a NUMA node; otherwise workers can run out of memory. This serving trade-off is distinct from tuning a single-user desktop session. See the vLLM CPU guide.

When to switch model, quantization, runtime, or hardware

If placement and configuration are correct but performance is still inadequate, compare a smaller model or a different quantized checkpoint that fits the device’s memory budget. Keep coding quality in the decision: test representative tasks, not just a short speed prompt. NVIDIA recommends choosing a checkpoint against VRAM and performance needs and evaluating it with a task-specific dataset and human grading. Its current guidance suggests Q4_K_M for llama.cpp and NVFP4 for vLLM or PyTorch; these are vendor recommendations, not universal benchmark findings. Compatibility and quality vary with runtime, GPU, model architecture, and software support. See NVIDIA’s local AI guidance.

Compare candidate setups on the measures that affect your work:

• Time to first token, prompt-processing rate, and decode tokens per second.
• Coding quality on representative tasks.
• Whether all or only some model layers fit on the accelerator.
• Context length and remaining memory headroom.
• Compatibility with your operating system, hardware, model, and runtime.
• For multi-user serving, aggregate throughput and behavior under concurrency.

A GPU change is worth considering when diagnostics show that an available accelerator is not being used, or that too few layers fit in its memory. There is no single GPU recommendation without knowing the model, runtime, and workload. More system RAM can make a larger model load for CPU inference, but it is not a guaranteed way to increase generation speed.

Quick Recap

Bestseller No. 1

GMKtec EVO-X2 AI Mini PC Ryzen Al Max+ 395 Superchip 128GB LPDDR5X 2TB SSD

$3,649.99

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.