Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Most local AI slowdowns and out-of-memory (OOM) errors have a small set of causes: the model is partly or entirely on the CPU, the context window is too large, several requests are multiplying memory use, the model or quantization does not fit, or the GPU is not available to the runtime. Start by recording the model and settings, inspect logs, and run ollama ps (if you use Ollama) to verify placement. Then reduce memory pressure one variable at a time: context length, parallel requests, idle models, model size or quantization, and—when using LocalAI—the number of GPU layers. Finally, verify drivers and container GPU access. This order identifies the actual bottleneck while preserving as much quality and speed as your hardware allows.
What usually makes a local AI app slow or run out of memory?
A local model can be slow even when the application itself is healthy. The most important distinction is whether the model is loading, waiting in a queue, generating on the CPU, or failing because the available memory is insufficient.
As an Amazon Associate I earn from qualifying purchases.
- CPU fallback or split placement: If weights do not fit in VRAM, the runtime may place all or part of the model in system RAM. CPU execution and CPU/GPU transfers are generally much slower than a model that remains on the GPU.
- Excessive context: Context is the conversation, document text, and prompt tokens kept in memory. A larger window consumes more VRAM even before a long answer is generated.
- Parallel requests: Multiple active chats or API calls multiply the working context. A model that fits for one request can fail when several are processed together.
- Model size and quantization: Larger models and higher-precision weights need more memory. A smaller or more aggressively quantized model may fit where the original does not.
- GPU plumbing problems: A missing driver, incorrect permissions, or a container without GPU access can leave the runtime on the CPU.
- Queueing rather than inference: A request can appear slow while it is waiting behind other requests. Check active jobs and queue settings before changing the model.
Identify which category you have before tuning. Changing several settings at once makes it impossible to tell which change solved the problem or caused a quality loss.
How should I capture a baseline before changing settings?
Write down the conditions of one reproducible prompt. This gives you a before-and-after comparison and separates a load-time failure from a generation-time slowdown.
#1 Best Overall
- A-Tech 32GB RAM Kit (2 x 16GB Modules), DDR4 SO-DIMM 260-Pin, 2666MHz / 2667MHz PC4-21300 (PC4-2666V)
- Non-ECC Unbuffered, JEDEC DDR4 Standard 1.2V Operating Voltage
- Compatible with select DDR4 SODIMM capable Laptop, Notebook, Mini PC, and All-in-One (AIO) computer systems. Please verify your system's memory type, form factor, and maximum supported capacity before purchasing
- Not compatible with desktop (DIMM), DDR2, DDR3, DDR5, ECC Registered (RDIMM), ECC Load Reduced (LRDIMM), or ECC Unbuffered (ECC UDIMM) memory types
- Increases available memory capacity to enhance system responsiveness, application performance, and multitasking capabilities.
- Record the model identifier and quantization. Include the exact tag or file, not only a family name. Note whether it is a full-precision, 8-bit, 6-bit, 5-bit, 4-bit, or other quantized build.
- Record context length and prompt size. Note the configured maximum context and roughly how many tokens your test prompt and attached documents contain.
- Record output length. Capture the requested maximum generation length and any displayed tokens-per-second rate.
- Record the failure stage. “Cannot load model” is different from an OOM after a long conversation. A response that starts after a long pause may indicate queueing or model loading rather than generation speed.
- Record hardware use. Note GPU memory, system memory, other GPU applications, and whether one or several requests were active.
Use the same short prompt after each change. Keep a note of processor placement, memory usage, response latency, and output quality. Stop changing settings once the model fits and the response rate is acceptable for your task.
Where do I find the logs and enable useful diagnostics?
Logs often reveal a failed GPU probe, an allocation request that exceeds available memory, or a model being split between devices. Collect logs before restarting repeatedly, because a restart can remove the context needed to explain the failure.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsOllama on Linux with systemd
View the service log with:
journalctl -u ollama --no-pagerFor additional diagnostics, start the service with OLLAMA_DEBUG=1 in its environment, then reproduce the problem. The exact way to set an environment variable depends on how your service is installed; confirm that the variable is present in the service process, not only in an unrelated shell.
Ollama in a container
Use the container name or ID:
docker logs <container-name>Look for messages about device discovery, allocation failures, and model loading. If the log says the GPU is unavailable, changing context length will not repair the underlying device configuration.
Ollama on macOS
Inspect the server log at:
~/.ollama/logs/server.logSearch around the timestamp of the failed request. Compare a successful short prompt with the failing prompt so that you can see whether the failure is tied to model loading or to increased context.
Rank #2
- Boosts System Performance: 16GB DDR4 Pro Series desktop memory RAM kit (2x8GB) that operates at 3200MHz, 3000MHz, or 2666MHz to improve multitasking and system responsiveness for smoother performance
- Easy Installation: Upgrade your desktop RAM with ease—no computer skills required Follow step-by-step how-to guides available at Crucial for a smooth, worry-free installation
- Compatibility Guaranteed: Ensure seamless compatibility with your desktop by using the Crucial System Scanner or Crucial Upgrade Selector—get accurate recommendations for your specific device
- Trusted Micron Quality: Backed by 42 years of memory expertise, this DDR4 RAM is rigorously tested at both component and module levels, ensuring top performance and reliability
- ECC Type = Non-ECC, Form Factor = UDIMM, Pin Count = 288-pin, PC Speed = PC4-25600, Voltage = 1.2V, Rank and Configuration = 1Rx16, 1Rx8 or 2Rx8
How do I check whether Ollama is using the GPU?
Run:
ollama psInspect the PROCESSOR column. Ollama reports states such as 100% GPU, 100% CPU, or a CPU/GPU split. A split load can explain a large performance drop even when a GPU is visible to the operating system. For best performance, keep the model entirely on the GPU when it fits; avoid CPU offloading unless it is the only way to load the model.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWhat ollama ps shows |
What it means | What to do next |
|---|---|---|
| 100% GPU | The model is resident on the GPU. Memory or queue pressure may still limit speed. | Check context length, parallel requests, GPU memory used by other processes, and generation settings. |
| 100% CPU | The model is running from system memory rather than GPU memory. | Verify drivers and GPU access; if the model is too large, use a smaller model or more aggressive quantization. |
| CPU/GPU split | Some weights are offloaded to system memory. | Reduce the model’s memory footprint until it fits in VRAM, or accept the speed trade-off deliberately. |
Do not infer placement from the presence of a GPU in the computer or from a high GPU utilization percentage during one moment. The processor column is the direct check for the loaded model.
How does context length affect memory and speed?
Ollama defines context length as “the maximum number of tokens that the model has access to in memory.” Increasing it raises memory use; lowering it is usually the least destructive first fix for an OOM because it leaves the model weights unchanged.
Ollama documents these context defaults by available VRAM:
| Available VRAM | Documented Ollama default | Important qualification |
|---|---|---|
| Below 24 GiB | 4,096 tokens (4k) | Ollama default, not a universal hardware requirement. |
| 24–48 GiB | 32,768 tokens (32k) | Ollama default, not a guarantee that every model fits. |
| 48 GiB or more | 262,144 tokens (256k) | Ollama default, not a recommendation for every workload. |
Choose the smallest window that covers the task. A short coding question may need only a few thousand tokens, while a document analysis workflow may need more. If you reduce the window too far, older conversation turns or document sections can be dropped, so test with the longest input you routinely send.
Recommended Free Tools
Rank #3
- DDR3 / DDR3L 1333MHz PC3-10600 204-Pin Non-ECC Unbuffered 1.5V / 1.35V CL9 Dual Rank 2Rx8 based 512x8
- Module Size: 16GB KIT(2x8GB Modules) Package: 2x8GB ; JEDEC standard 1.35V, this is a dual voltage piece and can operate at 1.35V or 1.5V
- Module Size: 16GB Package: 2x8GB For Laptop/Notebook, Not for Desktop
- Compatible for Selected Alienware , AOpen , ASRock , ASUS/ASmobile , BCM , Clevo , Dell , DFI , EliteGroup (ECS) , Fujitsu , Gigabyte , HP/Compaq , Intel , Lenovo , MiTAC , MSI , NEC , Panasonic , Samsung , Shuttle , Supermicro , Toshiba , ZOTAC motherboard systems
- Guaranteed – Lifetime warranty from Purchase Date Free technical support
Set context for an Ollama server
Ollama’s documented command-line form is:
OLLAMA_CONTEXT_LENGTH=64000 ollama serveThe 64,000-token value is an example, not a target. Replace it with a value your VRAM can sustain. When using the API, set the request’s num_ctx parameter to the desired context length and keep that value consistent in your benchmark.
Why do several requests cause an OOM when one request works?
Parallel processing multiplies the memory required for context and intermediate data. Ollama documents the relationship as scaling with OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH. A model that fits for one chat can therefore fail when multiple chats, automation jobs, or users are active.
Reduce concurrency before replacing the model
- Stop or pause extra clients and retry the same prompt with one active request.
- Reduce the configured parallel-request count, commonly through the
OLLAMA_NUM_PARALLELenvironment setting used by your deployment. - Unload idle models so their weights and runtime buffers are not competing for VRAM.
- Check queue behavior. Ollama documents a default
OLLAMA_MAX_QUEUEof 512; a large queue can make requests wait even when the model itself is healthy.
Lowering concurrency improves memory headroom but can increase wait time for users who need simultaneous responses. Pick a limit based on the number of real users and the context each request requires, not on the maximum number your client can submit.
What should I change first when the model runs out of memory?
Apply changes in a controlled order, testing the same prompt after each one. The sequence below starts with reversible settings and preserves model quality as long as possible.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →- Lower context length. Reduce the configured context or API
num_ctxto the smallest value that still contains the task. - Reduce parallel requests. Test one active request, then add concurrency gradually until memory remains stable.
- Unload idle models and free VRAM. Close other GPU applications and stop unused model servers. Confirm the freed memory with your platform’s GPU monitor.
- Use a smaller quantized model. Docker’s inference guidance specifically recommends trying a more aggressive quantization for OOM and slow llama.cpp-based inference. Quantization reduces memory, but lower precision can affect answer quality.
- Use fewer GPU layers in LocalAI. LocalAI documents reducing
context_size, loweringgpu_layers, or freeing VRAM held by other processes. Fewer GPU layers can allow a load to succeed while moving more work to the CPU. - Change the model size. If the workload still cannot fit, choose a smaller parameter count rather than repeatedly forcing an unstable configuration.
Keep the previous working configuration recorded. If a quality regression appears after quantization or context reduction, restore the last stable setting and change a different variable instead of stacking multiple compromises.
Rank #4
- Capacity – 32GB RAM KIT (2 x 16GB Modules) Speed up to 2666MHz Non-ECC Unbuffered 260-Pin 1.2V SODIMM.
- Specs – PCB Color (Green or Black) and Rank (1Rx8 or 2Rx8) may vary depending on production batch. Performance and quality remain consistent across all Timetec products.
- Compatibility – Designed for selected DDR4 Laptop, Notebook, Mini PCs, and All-In-One systems(AIO) that support 260-Pin SODIMM memory. NOT compatible with Desktop DIMM slots.
- Installation – Plug-and-Play Upgrade, Quick and Easy to Install, no expertise required (please refer to your system's manual for guidelines).
- Warranty – All Timetec products are high-quality and rigorously tested to meet stringent standards. Backed by Timetec Limited Lifetime Warranty and professional technical support based in the United States.
How do I verify GPU drivers and Docker access?
If ollama ps shows CPU placement or the logs show no usable device, verify the plumbing before tuning the model.
Check the host GPU
On NVIDIA systems, run:
nvidia-smiThe command should list the GPU, driver, memory usage, and running processes. An error, an empty device list, or a driver mismatch points to a host configuration problem rather than an AI-model setting. Ollama’s troubleshooting guidance also calls out NVIDIA UVM, driver versions, AMD device permissions, and GPU-discovery messages in logs.
Check a container’s GPU visibility
Docker recommends confirming that the container runtime is configured for GPU access and that the supported host platform and driver are installed. Run nvidia-smi from an appropriate GPU-enabled test container, then compare that result with the inference container’s logs. If the test container cannot see the device, changing quantization or context inside the application cannot fix it.
For AMD hardware, verify that the container or service user has the required device permissions. For either vendor, restart the inference service only after correcting the host or container configuration so the model is loaded again with the intended device.
How can I tell whether the problem is memory, queueing, or CPU fallback?
Use the symptom pattern and the checks below rather than guessing from response time alone.
Best Value
- [Specs] DDR3L / DDR3 1600MHz PC3L-12800 / PC3-12800 204-Pin Unbuffered Non ECC 1.35V CL11 Dual Rank 2Rx8 based 512x8
- [Size] Module Size: 16GB KIT(2x8GB Modules) Package: 2x8GB
- [Voltage] JEDEC standard 1.35V, this is a dual voltage piece and can operate at 1.35V or 1.5V
- [Compatibility] Compatible with DDR3 Laptop / Notebook PC, Mini PC, All in one Device
- [Color] PCB Color is green
| Symptom | Likely cause | Confirmation | Targeted fix |
|---|---|---|---|
| Model fails immediately while loading | Weights plus runtime buffers exceed available memory, or the GPU is unavailable. | Load-time OOM in logs; CPU or split placement in ollama ps. |
Fix GPU access, reduce model size or quantization, or lower context. |
| Short prompts are fast, long chats fail | Context memory grows beyond the limit. | Failure appears after adding history or documents. | Lower context, trim old turns, or split documents into smaller tasks. |
| One request works, several fail | Parallel processing multiplies effective context. | Failure disappears with one active client. | Lower parallelism and unload idle models. |
| Model answers, but every token is slow | CPU-only or CPU/GPU split placement. | ollama ps shows 100% CPU or a split. |
Repair drivers/container access or fit the model entirely in VRAM. |
| Requests begin only after a long wait | Queueing or a busy server. | Other requests are active; queue settings and logs show waiting. | Reduce concurrency, shorten jobs, or raise capacity only after measuring memory. |
| GPU is visible but the runtime does not use it | Driver, UVM, permissions, or runtime configuration issue. | nvidia-smi works on the host but logs show failed discovery. |
Correct the host/container setup and restart the inference service. |
What are the trade-offs of each fix?
No setting improves every dimension. Choose the intervention that solves the immediate constraint while preserving the part of the workload that matters most.
| Change | Memory relief | Effect on quality | Effect on speed | Compatibility and reversibility |
|---|---|---|---|---|
| Lower context length | Often substantial, especially for long prompts. | Older history or document text may be unavailable. | Can improve stability and reduce prompt processing time. | Usually compatible and easy to reverse. |
| Reduce parallel requests | Directly reduces multiplied working memory. | No change to an individual answer. | May increase waiting time for simultaneous users. | Compatible with most deployments and reversible. |
| Unload idle models or other GPU jobs | Returns occupied VRAM to the active model. | No model-quality change. | May add reload time when switching models. | Highly reversible; operational rather than model-specific. |
| More aggressive quantization | Reduces weight memory and may prevent CPU offload. | Lower precision can reduce accuracy or consistency. | Can be faster when it keeps the model on the GPU. | Requires a compatible model build; replaceable by restoring the prior build. |
| Fewer LocalAI GPU layers | Reduces VRAM allocation. | Model computation is unchanged, but some layers leave the GPU. | Usually slower because more work runs on the CPU. | LocalAI-specific control; easy to adjust and test. |
| Smaller model | Largest and most predictable reduction. | May reduce capability on difficult tasks. | Often faster if it fits fully in VRAM. | Depends on backend and model format; changing back is straightforward. |
What is a reliable troubleshooting workflow for a slow local AI app?
Use this decision path whenever a configuration changes or a new model is installed.
Free tools Windows power users keep installed
One-click scans. No signup required.
- Reproduce with one request. Use a short, fixed prompt and no attached document. This removes queueing and oversized context from the first test.
- Inspect logs. Capture the server or container log at the moment of load and generation.
- Verify placement. Run
ollama ps; record 100% GPU, 100% CPU, or split. - Verify the device independently. Run
nvidia-smion NVIDIA hosts and check AMD permissions where applicable. - Measure a known-good baseline. Record latency, tokens per second if shown, memory use, and output quality.
- Lower context. Repeat the baseline with a smaller context that still fits the task.
- Lower parallelism. Add requests one at a time and watch memory and queue time.
- Change the model footprint. Try a more aggressive quantization, fewer LocalAI GPU layers, or a smaller model only if the previous steps are insufficient.
- Re-test the real workload. A short prompt can fit while your production prompt does not. Test the longest normal document and the expected number of simultaneous users.
How do I keep the configuration stable after it works?
- Save the model name, quantization, context length, parallelism, GPU-layer setting, and driver version with your deployment notes.
- Keep a one-command health check that records
ollama ps, GPU visibility, and recent server logs. - Set an explicit context length rather than relying on a changing default when you deploy across machines with different VRAM capacities.
- Load-test with the maximum expected concurrent requests, not just a single interactive chat.
- Leave headroom for the operating system, display server, and other GPU processes; a configuration that uses every last MiB may fail when another process starts.
- Change one variable at a time and keep the last stable configuration available for rollback.
What should I do first when the app is still slow?
First verify placement with ollama ps and inspect the logs. If the model is CPU-only or split, repair GPU access or reduce the model until it fits; if it is fully on the GPU, lower context and parallelism before changing model quality. This sequence addresses the largest performance causes while keeping every change measurable and reversible.
Frequently Asked Questions
Does 100% GPU placement guarantee fast responses?
No. It confirms where the model is loaded, not how much work is queued or how large the prompt is. A fully GPU-resident model can still spend time processing a very long context or waiting behind parallel requests.
Can I solve a GPU OOM simply by adding system swap?
Swap does not create additional VRAM, so it cannot make a GPU allocation fit. Concentrate on context, concurrency, quantization, model size, or GPU-layer settings; use swap only as an operating-system safeguard, not as an inference-performance fix.
Why does the same model fit on one computer but not another?
Available VRAM, driver state, other GPU processes, context defaults, and parallel-request settings can all differ. Compare those variables explicitly instead of comparing advertised GPU model names alone.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →How do I choose a context length for document analysis?
Start with the smallest value that contains the prompt plus the portion of the document the model must consider, then test the longest normal document. If you need more text than fits, process the document in sections and preserve only the necessary summaries between requests.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
