Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
You can keep an LLM service running with fewer or older GPUs by reducing weight memory with quantization, placing some or all execution on system CPUs, or using a CPU–GPU hybrid runtime. The practical choice depends on the model’s weight size, context and KV-cache demand, available system RAM, the accelerator backend your software supports, and whether you need low latency or high throughput. Start by measuring the hardware you actually have, then test the smallest quantized model that meets your quality target. Treat offloading as a way to make a model fit—not as a guarantee of fast generation.
What is the fastest way to keep inference running when GPU capacity is limited?
Use this order of operations: first try a lower-bit model, then decide whether the remaining workload belongs on one GPU, several GPUs, system RAM, or a mixture. Quantization usually delivers the largest capacity gain with the fewest architectural changes. If the model still does not fit in VRAM, llama.cpp supports CPU-plus-GPU hybrid inference, while a CPU-only deployment can handle low-throughput workloads. vLLM is useful when its current accelerator or CPU support matches your software stack and serving requirements.
As an Amazon Associate I earn from qualifying purchases.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11- Weights do not fit: select a quantized format before changing hardware.
- Weights fit only in system RAM: use CPU inference or hybrid placement.
- Several users need service: evaluate batching, KV-cache capacity, and parallelism in a serving runtime.
- An older accelerator is available: verify its generation, platform, driver, and backend support in the live runtime documentation.
Neither the llama.cpp README nor the vLLM documentation establishes a universal tokens-per-second result for older GPUs. Measure latency and throughput on your own model, prompt lengths, and request mix.
How should I assess the hardware before choosing a runtime?
Record capacity and compatibility separately. A card can have enough VRAM but lack a supported backend, while a supported card can still fail when context, cache, and concurrent requests consume the remaining memory.
#1 Best Overall
- Get NVMe solid state performance with up to 1050MB/s read and 1000MB/s write speeds in a portable, high-capacity drive(1) (Based on internal testing; performance may be lower depending on host device & other factors. 1MB=1,000,000 bytes.)
- Up to 3-meter drop protection and IP65 water and dust resistance mean this tough drive can take a beating(3) (Previously rated for 2-meter drop protection and IP55 rating. Now qualified for the higher, stated specs.)
- Use the handy carabiner loop to secure it to your belt loop or backpack for extra peace of mind.
- Help keep private content private with the included password protection featuring 256‐bit AES hardware encryption.(3)
- Easily manage files and automatically free up space with the SanDisk Memory Zone app.(5). Non-Operating Temperature -20°C to 85°C
Inventory the machine
- GPU model, generation, VRAM, number of cards, and the driver or accelerator software already installed.
- System RAM and, where relevant, how memory is arranged across NUMA nodes.
- CPU architecture, core count, memory bandwidth, and storage free space.
- Target context length, expected concurrent requests, prompt-to-generation ratio, and acceptable first-token latency.
Separate the memory pools
| Resource | What consumes it | What a shortage looks like |
|---|---|---|
| GPU VRAM | Model layers placed on the accelerator, temporary runtime buffers, and GPU-side KV cache | Model load failure, out-of-memory errors, or a need to move layers to CPU memory |
| System RAM | CPU-resident weights, hybrid layers, CPU KV cache, operating-system processes, and runtime overhead | Swapping, failed allocation, or severe latency spikes |
| KV cache | Conversation context and active sequences; its size grows with context length and concurrency | Short prompts work but long prompts or additional users fail |
| Disk | Downloaded model files, quantized variants, source trees, and build artifacts | Download or build stops despite having enough compute memory |
Weight fit is only the first check. Reserve headroom for the runtime, the operating system, context, and simultaneous requests instead of allocating every available byte to model layers.
What does quantization change, and how much quality can I expect?
Quantization stores model weights with fewer bits, reducing the memory required to load them. The llama.cpp project documents formats ranging from 1.5-bit through 8-bit. Lower-bit representations generally create more capacity, but quality can decline, so the correct choice is the lowest representation that passes your task-specific evaluation.
Recommended Free Tools
Use the precision range as a testing ladder
| Representation range documented by llama.cpp | Best use | Decision test |
|---|---|---|
| 1.5–3-bit | Extreme memory pressure when a model otherwise cannot run | Check factual accuracy, instruction following, and safety behavior on a representative test set |
| 4–6-bit | Common compromise between capacity and output quality | Compare answers against a higher-precision reference for your actual workload |
| 7–8-bit | When you need a smaller memory reduction and want to stay closer to the original weights | Confirm the remaining memory is enough for context and concurrent requests |
These ranges describe storage formats, not a promise that every model behaves identically at each level. A coding assistant, summarizer, and retrieval-augmented system can fail in different ways. Test long-context behavior as well as short prompts, because a model that loads successfully can still run out of cache memory during real conversations.
Choose a quantized file that matches the runtime
Use a format explicitly supported by the runtime you intend to operate. Keep the original model or a higher-precision reference available for spot checks, but do not download several large variants until you have confirmed the one your backend can load. Quantization reduces weight memory; it does not remove KV-cache or runtime overhead.
How do I use llama.cpp with one older GPU or several smaller GPUs?
llama.cpp is designed for local inference and documents quantized formats, multiple CPU and GPU backends, and CPU–GPU hybrid inference. That combination lets you run a model that is larger than the VRAM in one card, provided system RAM can hold the CPU-resident portion and the selected backend supports your accelerator.
Rank #2
- Easily store and access 2TB to content on the go with the Seagate Portable Drive, a USB external hard drive
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
- Confirm the backend: identify the GPU API or CPU path available on the machine and select the llama.cpp build that supports it. Do not assume a newer build supports every older card; check the project’s current backend notes.
- Start with a known quantized model: choose a file within the documented quantization range and verify its format before attempting a large download.
- Load the model with conservative context settings: begin with the context length and concurrency your application actually needs, leaving memory headroom for the runtime.
- Place layers on the GPU until VRAM is nearly, but not completely, full: leave room for GPU-side buffers and cache. Put the remaining layers in system RAM through the runtime’s hybrid-inference options.
- Run a representative prompt set: measure first-token latency, generation speed, peak VRAM, peak RAM, and whether long prompts trigger allocation failures.
- Adjust one variable at a time: change GPU layer placement, quantization, context, or concurrency separately so you can identify the cause of a failure or slowdown.
What hybrid inference can and cannot do
Hybrid placement is a capacity technique: the GPU accelerates the layers assigned to it while the CPU handles the rest. It can make a model load when all-GPU placement is impossible, but transfers between CPU and GPU and the slower CPU portion can reduce speed compared with an all-GPU run. The project documentation does not provide a universal speed estimate for a particular older card, so publish or plan with measurements from your own machine.
When several GPUs are available
Treat the cards as a combined but nonuniform resource. Differences in VRAM, interconnect, and backend support can make an apparently large total capacity less useful than one fast, compatible card. Test a small layer-placement change first, and monitor each card separately rather than relying only on a total-memory figure.
Can I run inference entirely on the CPU?
Yes. CPU-only inference is practical when throughput requirements are modest or no suitable accelerator is available, but speed depends heavily on CPU architecture, memory capacity, memory bandwidth, context length, and workload. vLLM documents CPU support and exposes controls for CPU KV-cache allocation and batching because those settings determine how many requests and how much context can fit.
Configure capacity before chasing speed
- Build or install the CPU-supported runtime: follow the current vLLM CPU installation path for your operating system and CPU architecture.
- Allocate CPU KV cache deliberately: leave enough NUMA-node memory for weights, the operating system, and runtime buffers. Too little cache limits context or concurrency; too much can starve the rest of the process.
- Set batch controls for the workload: smaller batches usually favor responsiveness, while larger batches can improve throughput when requests can wait.
- Test prompt lengths and concurrency separately: a configuration that handles one short request may fail with several long contexts.
- Watch for swapping: once the operating system starts paging model or cache data, latency becomes unpredictable. Reduce context, concurrency, or model size before increasing batch limits.
When CPU-only is a sensible choice
- Offline jobs where total completion time matters more than interactive latency.
- Low-volume internal tools with a small number of simultaneous users.
- Development and evaluation, where avoiding unavailable or unsupported GPUs is more important than peak speed.
Do not describe CPU execution as GPU-like performance without measurements. The documented controls exist precisely because hardware and request patterns change the result.
How do I decide whether vLLM is compatible with an older GPU?
Check the current vLLM GPU installation documentation before selecting it. The project lists supported accelerator families and platform requirements, and those requirements can change as software and drivers evolve. A card that worked with an earlier release may need a different build—or may no longer be supported—so verify the exact GPU generation, operating system, driver, and installation path.
Rank #3
- 9.5X faster than hard disk drives with up to 1050MB/s read and 1000MB/s write, for significantly faster transfers
- Super-fast backups and instant storage expansion
- Built-in, double layer graphite sheets quickly conduct heat, enchancing heat dissipation to keep the drive cool, even during blazing fast transfers
- The drive is drop-resistant up to 2 meters for protection, ensuring durability for everyday and office use.
- Plug-and-play compatibility with a 2-in-1 USB-C/USB-A cable to easily use with your PC, Mac, tablet, smartphone, and more
Compatibility checklist
- Is the accelerator family listed in the current GPU installation guide?
- Does your operating system and driver meet the documented platform requirements?
- Does the required quantization method run on that backend?
- Can the card reserve enough VRAM for weights, KV cache, and runtime overhead?
- Will your request rate benefit from vLLM’s batching or parallelism features, or would their overhead hurt a single-user workload?
Validate with a small deployment
Load the intended model with a short context, send one request, then increase context and concurrency while monitoring memory. Record startup success, first-token latency, generation rate, and error messages. Only after that smoke test should you commit to a multi-GPU or production configuration.
Is CPU weight offloading a good fallback?
CPU weight offloading can fit a full-precision large model into limited GPU memory by keeping some weights in system RAM. A vLLM project article dated July 23, 2024, characterized CPU offloading as a last-resort option and warned that its performance was preliminary at that time. Use that description as historical context, not as a current benchmark or a promise of interactive speed.
For a current local CPU/GPU split, llama.cpp’s documented hybrid inference is the clearer starting point. In either runtime, offloading increases the importance of system RAM capacity and CPU–GPU transfer costs. It solves a placement problem; it does not make the model computationally smaller.
How much memory should I reserve for context and concurrent users?
Plan memory in layers: weights, KV cache, runtime overhead, and concurrency. A model that consumes nearly all VRAM at startup leaves no room for a long conversation or a second request. On CPU deployments, vLLM’s documentation specifically ties CPU KV-cache capacity to available NUMA-node memory and explains that batch settings affect throughput and latency.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Use a staged capacity test
- Load one request at the shortest context you support.
- Increase context toward the production maximum and note peak VRAM and RAM.
- Add concurrent requests one at a time until latency or allocation errors become unacceptable.
- Reserve a safety margin for tokenizer buffers, framework overhead, monitoring, and normal operating-system activity.
If long contexts are rare, do not reserve maximum cache for every request. If long contexts are common, reducing the model’s weight precision may be more effective than trying to reclaim the last few megabytes of runtime memory.
Rank #4
- MADE FOR THE MAKERS: Create; Explore; Store; The T7 Portable SSD delivers fast speeds and durable features to back up any endeavor; Build your video editing empire, file your photographs or back up your blogs all in an instant
- SHARE IDEAS IN A FLASH: Don’t waste a second waiting and spend more time doing; The T7 is embedded with PCIe NVMe technology that brings fast read and write speeds up to 1,050/1,000 MB/s¹, making it almost twice as fast as the T5
- ALWAYS MAKE THE SAVE: Compact design with massive capacity; With capacities up to 4TB, save exactly what you need to your drive – from large working files to game data and everything in between
- ADAPTS TO EVERY NEED: Whether using a PC or mobile phone, count on the T7 for extensive compatibility²; It’s a true team player when it comes to heavy-duty application usage or file-saving
- HI RESOLUTION VIDEO RECORDING: Record Ultra High Resolution (4K 60fs) videos directly onto the T7 Portable SSD with your favorite camera or mobile devices; Supports iPhone 15 Pro Res 4K at 60fps video and more³
How much disk space do local model workflows need?
Disk is separate from VRAM and RAM, but it can still block deployment. An NVIDIA llama.cpp playbook updated October 9, 2026, uses about 40 GB of free space for its example model download plus build artifacts. That figure is an example-workflow estimate, not a universal minimum: model size, duplicate quantized files, source trees, and temporary build data can raise the requirement.
Plan storage deliberately
- Keep free space for the model file, a temporary download, and build outputs at the same time.
- Delete unused precision variants after validating the selected file.
- Place model files on a fast local volume when possible, while remembering that storage speed does not increase VRAM or replace system RAM.
- An external SSD can add capacity if your computer has a compatible port and adequate free space is otherwise unavailable; it does not by itself make inference faster.
What are the common failure modes and fixes?
The model fails to load with an out-of-memory error
Lower the weight precision, reduce context or concurrency, move layers to CPU memory, or choose a smaller model. Check both VRAM and system RAM: hybrid placement can simply move the failure from one pool to the other.
One short prompt works but long conversations fail
The KV cache is consuming the remaining memory. Reduce maximum context, lower concurrency, or allocate more CPU cache where the runtime supports it. Test the longest expected prompt rather than relying on startup success.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
The older GPU is detected but the runtime will not start
Recheck the exact accelerator family, driver, operating system, and backend requirements in the current installation guide. Detection alone does not prove that the required kernels or quantization path are supported.
Hybrid inference runs but is too slow
Measure how many layers are on the CPU and whether transfers or CPU computation dominate. Try a smaller or higher-quality quantized model that fits more layers on the GPU, reduce context, or reserve hybrid mode for workloads where capacity matters more than latency.
Best Value
- Solid state performance with up to 800MB/s read speeds in a portable drive. (Based on internal testing; performance may be lower depending on host device, interface, usage conditions and other factors. 1MB=1,000,000 bytes.)
- Back up your content and memories on a storage solution that fits seamlessly into your mobile lifestyle.
- Take it with you on your adventures—up to two-meter drop protection means this durable drive can take a beating. (Based on internal testing.)
- Secure it to your belt loop or backpack for extra peace of mind thanks to the tough rubber hook.
- From SanDisk, a brand professional photographers trust to take on assignments.
CPU serving becomes erratic under load
Inspect NUMA memory pressure and operating-system swapping, then reduce batch size, concurrency, or cache allocation. A smaller steady workload is usually more useful than a nominal configuration that exhausts memory.
Which method should I choose for my situation?
| Situation | First option | Why | Main caution |
|---|---|---|---|
| Weights exceed available VRAM or RAM | Lower-bit quantized model | Reduces weight memory without changing the basic serving architecture | Validate quality on the actual task |
| No suitable GPU, low request volume | CPU inference | Uses available system resources and avoids accelerator compatibility issues | Latency and throughput depend strongly on CPU and cache settings |
| Model exceeds total GPU VRAM but system RAM is available | llama.cpp CPU–GPU hybrid inference | Places some layers on the GPU and the rest in RAM | Transfers and CPU layers can make it slower than all-GPU execution |
| Supported accelerator and multi-user serving workload | vLLM GPU serving | Offers documented quantization, batching, and parallelism features | Confirm current accelerator and platform support first |
| Full-precision model barely fits with limited GPU memory | CPU weight offload | Can make the placement possible when VRAM alone is insufficient | The July 2024 project article called performance preliminary and a last resort |
What is a practical rollout plan during a shortage?
Make capacity and quality decisions in separate stages so a shortage does not turn into an untestable production change.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minute- Define acceptance tests: include factuality, instruction following, safety behavior, maximum context, first-token latency, and sustained throughput.
- Establish a baseline: run the smallest existing deployment and record peak VRAM, RAM, latency, and concurrency.
- Test quantization: move down the available precision ladder and keep the first format that passes the acceptance tests.
- Test placement: if the model still does not fit, compare all-GPU, CPU-only, and hybrid layouts with the same prompts and context.
- Validate the runtime: for vLLM, verify current accelerator and platform support; for llama.cpp, verify the selected backend and hybrid options.
- Stress the real workload: increase context and concurrent users until you reach the service target, not merely until the model answers one prompt.
- Document the fallback: record the model file, quantization, runtime version, backend, cache settings, batch limits, and measured results so another machine can reproduce the deployment.
The practical playbook is therefore simple: quantize first, use CPU or hybrid placement when capacity still falls short, and select a serving runtime only after checking the exact older hardware against current support requirements. Keep quality tests and workload measurements alongside memory figures, because a model that fits is not automatically a model that serves well.
Frequently Asked Questions
Can I combine different quantization levels in one deployment?
Use one model format that your chosen runtime officially supports rather than mixing files manually. If you need a different memory target, test another complete quantized variant and compare it against the same quality and context checks.
Does reducing context length improve performance as well as memory use?
It can reduce KV-cache pressure and leave more memory for active layers, which may prevent swapping or allocation failures. The effect on latency depends on prompt length and batching, so measure with the request pattern you expect in production.
How can I tell whether an old GPU is worth keeping for inference?
Run a small compatibility and load test with the exact runtime, model format, driver, and context settings you plan to use. Keep the card if it delivers acceptable quality and latency within a stable memory envelope; theoretical VRAM alone is not enough.
Free tools Windows power users keep installed
One-click scans. No signup required.
Should I store models on an external SSD?
An external SSD is useful when local capacity is the bottleneck and the computer has a compatible port. It adds room for model files and build artifacts, but it does not add VRAM, system RAM, or compute throughput.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
