There is no universal winner. For local large-language-model inference, an integrated GPU can reduce first-token latency or increase generation speed when its runtime backend is well supported, while a CPU can be the more dependable choice when the model fits in RAM and GPU acceleration is unavailable. The deciding factors are model size and quantization, available shared memory, memory bandwidth, context length, runtime support and how you measure performance.
Evaluate time to first token (including prompt processing) separately from steady-state generation speed. A machine that feels quick to begin answering may produce later tokens slowly, or the reverse. Use the same model, precision, prompt, context, token limit and software versions when comparing CPU and integrated-GPU modes.
As an Amazon Associate I earn from qualifying purchases.
What the comparison actually means
A CPU-only run executes the model’s matrix operations and other work on general-purpose processor cores. An integrated GPU (iGPU) uses graphics-compute units built into the same processor package. Unlike a discrete graphics card, an iGPU normally has no dedicated high-capacity VRAM pool: it reserves part of system memory and shares the memory controller with the CPU and operating system.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →That shared design creates a trade-off rather than a simple advantage. The GPU can access a larger pool than a small discrete-VRAM allocation, and data does not have to cross a separate PCIe device boundary. However, the operating system, applications, model weights, runtime buffers and key-value (KV) cache all compete for the same finite capacity and bandwidth. A model file that appears to fit on disk may still fail to load once these other allocations are included.
#1 Best Overall
- System Compatibility Note: This 2-slot card measures 271 x 112 x 39 mm and requires a single 12V-2x6-pin power connector. Please verify chassis and PSU compatibility before purchase.
- Dedicated Support: Please contact us directly through Amazon for any product questions or assistance you may require.
- Professional Intel Arc Pro B70 GPU: Built on the Intel Xe2-HPG architecture, it features 32 Xe cores and 256 XMX engines, designed to accelerate AI, rendering, and complex visualization workloads.
- Massive 32GB GDDR6 VRAM: Equipped with 32GB of high-speed GDDR6 memory on a 256-bit bus, running at 19 Gbps, which allows for handling large AI models and complex datasets locally.
- High-Performance Engine Clock: Delivers an engine clock of 2540 MHz, providing the compute power needed for demanding professional applications and AI inference.
Shared memory: the iGPU’s main advantage and limit
Apple’s MLX documentation describes the benefit clearly: “Operations in MLX can run on either the CPU or the GPU without needing to move memory around.” In a unified-memory system, the same physical pool is visible to both processors, avoiding explicit copies between CPU RAM and GPU VRAM. That can simplify scheduling and let a GPU work on models that would exceed a small discrete card’s VRAM.
Shared memory is not free memory. A 24GB machine does not provide 24GB exclusively to model weights: macOS or another operating system, the inference process, prompt tokens, KV cache and background applications consume part of it. Long contexts increase KV-cache use, and larger batch sizes increase temporary workspace. Leave headroom rather than sizing a purchase to the exact byte count of a quantized file.
Capacity and bandwidth are separate constraints. Capacity determines whether the model can load; bandwidth influences how quickly weights and cache data can be read during generation. An iGPU with more memory available but lower bandwidth can still be slower, while a high-bandwidth design may be unusable if the model does not fit.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Why first-token latency and generation speed diverge
Inference has at least two user-visible phases:
- Prompt processing and time to first token (TTFT): the runtime evaluates the supplied context and prepares the first output token. This phase performs substantial parallel computation.
- Decode or steady-state generation: the runtime produces one token at a time, repeatedly reading model weights and updating the KV cache.
In Apple’s tested M5 workload, the company says, “In LLM inference, generating the first token is compute-bound, and takes full advantage of the Neural Accelerators.” It separately states, “Generating subsequent tokens is bounded by memory bandwidth, rather than by compute ability.” Those statements describe Apple’s framework and tested configuration, not a law for every processor, model or runtime. Tom’s Hardware similarly explains decode as repeatedly streaming model weights from memory.
Report both measurements. A prompt-heavy application such as document summarization may be dominated by TTFT, while an interactive chat session with short prompts and long answers may be dominated by decode throughput. A single tokens-per-second number hides that distinction.
Rank #2
- NVIDIA Volta GV100 Architecture — 4,608 CUDA Cores, 640 1st-Gen Tensor Cores delivering 14 TFLOPS FP32 and 112 TFLOPS deep learning performance for AI training, inference, HPC, and scientific computing workloads
- 32GB HBM2 ECC Memory — 900 GB/s Bandwidth — High-bandwidth memory on a 4096-bit bus with ECC error correction provides the memory capacity and throughput required for the largest AI models, simulations, and datasets
- PCIe 3.0 x16 Interface — 250W TDP — Standard PCIe Gen3 connectivity with passive cooling designed for enterprise rack server deployment in HPE ProLiant, Dell PowerEdge, and Supermicro platforms with adequate chassis airflow
- NVLink — Scale to 96GB Unified Memory — Connect two V100 GPUs via NVLink at 300 GB/s bi-directional bandwidth to scale GPU memory from 32GB to 96GB for larger AI training and HPC workloads
- Multi-Precision Computing — Supports FP64 (7 TFLOPS), FP32 (14 TFLOPS), FP16 (112 TFLOPS) and INT8 precision modes for flexible deployment across training, inference, and scientific simulation workloads
Model fit comes before speed
Weights, runtime overhead and KV cache
Model parameters are only the first allocation. The runtime needs space for quantized or full-precision weights, temporary activations, tokenizer and scheduler state, and the KV cache. The cache grows with context length, number of layers, attention dimensions and batch size. Different runtimes may allocate different workspace sizes, so a model that fits in one backend can fail in another.
Use the runtime’s reported memory usage and a failed-load test, not just the download size. Keep enough free memory for the operating system and other applications, especially on shared-memory machines.
Published M5 examples
Apple’s 2026 MLX test on a MacBook Pro with 24GB of unified memory reported these particular configurations:
| Workload | Precision | Reported memory used | What the figure means |
|---|---|---|---|
| Qwen3-8B | BF16 | 17.46GB | Measured for Apple’s stated model and runtime configuration; it is not a universal 8B-model requirement. |
| Qwen3-30B-A3B | 4-bit | 17.31GB | Measured for Apple’s stated mixture-of-experts model and runtime configuration; active experts and runtime overhead affect the result. |
These results illustrate why parameter count alone is misleading: a larger, quantized mixture-of-experts model can occupy a similar footprint to a smaller BF16 model. Read the exact model variant, context and runtime settings before applying any published number to your own system.
Quantization changes the capacity and quality equation
Quantization stores parameters at lower numerical precision, reducing the model footprint and often making a previously impossible model fit. Apple benchmarks both native and 4-bit configurations in its MLX work. Lower precision is not a guaranteed speedup: dequantization overhead, kernel support, memory layout and the specific model all matter.
Rank #3
- Powered by Radeon AI PRO R9700 - Supercharge you workflow with the cutting-edge RDNA 4 Architecture and 2nd-gen AI Accelerators.
- 32GB GDDR6 with 256-bit memory bus - Tackle larger, more complex projects without limits.
- PCIe Gen 5 - Unlock lightning-fast data transfers with PCIe Gen 5 support.
- GIGABYTE TURBO Fan Cooling System - Indented metal cover and blower fan increase airflow intake, while the vapor chamber, all copper heat sink, and metal frame offer efficient heat dissipation. Optimized airflow design allows for easy multi-GPU scalability.
- Double Ball Bearing Fan - Delivers superior heat resistance and rotational efficiency for better performance and a longer lifespan compared to conventional sleeve fans.
Quality also varies by model and quantization method. Treat “4-bit” as a storage format, not a promise of identical quality or throughput across runtimes. Compare the exact quantized file, context length and backend you intend to use. If quality is more important than fitting the largest possible model, a smaller higher-precision model may be preferable.
Software support can decide the practical winner
Backend availability
An integrated GPU is useful only when the inference runtime can execute supported kernels on it. A visible GPU in the operating system does not prove that the model is using it. Check startup logs, device-selection flags and runtime diagnostics for the selected backend, then watch utilization while generating tokens.
Apple’s MLX research is specific to Apple silicon. A 2025 preprint comparing MLX, MLC-LLM, Ollama, llama.cpp and PyTorch MPS shows why runtime choice is a first-class variable, but its results should not be transplanted to unrelated processors or software versions: the comparative study.
Operating-system and version requirements
Apple reports that MLX works on Apple silicon generally, while access to the M5 Neural Accelerator performance described on its page requires macOS 26.2 or later. Version requirements and backend capabilities change, so verify the current runtime and operating-system documentation before buying or upgrading. A benchmark made with one driver or framework version may not reproduce after an update.
How to benchmark CPU and integrated-GPU modes on your machine
A controlled local test is more useful than comparing unrelated online headline numbers. Use this procedure:
Recommended Free Tools
Rank #4
- System Compatibility Note: 2-slot card, 271x112x39mm, single 8-pin power, 200W TDP. Verify chassis clearance and PSU capacity before purchase.
- Dedicated Support: Please contact us directly through Amazon for any product questions or assistance you may require.
- 24GB GDDR6 on 192-Bit Bus: Massive 24GB memory with 456 GB/s bandwidth – ideal for LLMs, AI inference, 3D rendering, and generative design.
- Intel Xe2-HPG Architecture: Built on Intel's next-gen architecture with 20 Xe cores and 160 XMX engines for AI acceleration (197 INT8 TOPS).
- PCIe 5.0 Support: PCI Express 5.0 x16 interface for maximum bandwidth with the latest workstation platforms.
- Choose one exact model. Record the model family, revision, quantization format and file checksum if available. Do not compare an 8B 4-bit model on one device with a 7B BF16 model on another.
- Fix the prompt and context. Save a representative prompt and specify the total input-token count. Include a short-prompt case and a long-context case if your workload varies.
- Fix the output length. Set a defined maximum generated-token count. Discard warm-up output and measure several repeated runs rather than timing a single response.
- Record software versions. Note the operating system, runtime, model-loader version, GPU driver or Metal/Vulkan stack, and command-line flags. Keep power mode, cooling profile and background applications consistent.
- Run CPU-only first. Explicitly select the CPU backend and verify in logs that no GPU layers are being used. Record model-load time, peak memory, TTFT or prompt-processing rate, and steady-state tokens per second.
- Run the integrated-GPU backend. Select the documented GPU backend, verify device selection, and repeat the identical prompt and token limit. Record failures to load, peak shared-memory use, TTFT and decode speed.
- Test concurrency separately. If you serve multiple users, measure two or more simultaneous requests. Throughput under concurrency can differ substantially from single-stream latency because requests compete for memory bandwidth and cache space.
- Repeat after a cold and warm start. A cold run includes model loading and compilation; a warm run measures inference after kernels and files are resident. Report which one you are quoting.
- Keep a result table. Include median and range, not only the fastest run. A useful row contains backend, TTFT, decode tokens per second, peak memory, context length, generated tokens and any errors.
This method also reveals whether a faster-looking setup is merely producing a first-token improvement while decoding at the same speed, or whether it is silently falling back to the CPU.
When CPU-only is the sensible choice
- The model fits comfortably in system RAM, including the intended context and KV cache.
- Your runtime has no reliable iGPU backend, or the available backend is unstable after driver updates.
- Your workload is small, occasional or latency-tolerant, so the setup cost of GPU acceleration is not justified.
- You need predictable behavior across model families and do not want to tune device placement.
- The iGPU would consume memory needed by the operating system or force aggressive context limits.
CPU-only inference is a reasonable practical option when response speed meets your requirement. The available evidence does not establish one universal CPU-versus-iGPU ratio, so measure your own model rather than assuming that an iGPU will always be faster.
When the integrated GPU is worth using
- The runtime reliably selects the iGPU and provides optimized kernels for your processor generation.
- You have a compute-heavy prompt-processing workload where parallel GPU execution reduces TTFT.
- Your system has enough shared memory for the model, KV cache and applications with comfortable headroom.
- You run long responses and testing shows higher decode throughput, not just a lower first-token time.
- You prefer a single unified memory pool over a discrete card’s smaller VRAM capacity.
Confirm the benefit with measurements. An iGPU that is present but unsupported can add no speed at all, and a backend that falls back for some operators may perform worse than a well-optimized CPU path.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Hybrid CPU-and-GPU offload is a tuning problem
Many runtimes can place some layers or expert weights on the GPU and keep the rest on the CPU. This is not one fixed “hybrid mode”: the split changes memory pressure, PCIe or shared-memory traffic, kernel scheduling and load balance. More GPU layers are not automatically faster if transfers or memory contention dominate.
Free tools Windows power users keep installed
One-click scans. No signup required.
An Intel benchmark repository provides a concrete illustration on a Core Ultra 9 285K system paired with an Arc A770. For a Qwen3.6-35B-A3B run, the author reports an increase from 6.5 tokens per second to 22.1 tokens per second after tuning the CPU-offload setting. The result is an enthusiast-reported configuration using a discrete GPU, not an integrated-GPU-versus-CPU head-to-head result. Its lesson is methodological: test several offload settings instead of forcing all expert weights onto the CPU. See the benchmark repository for the exact setup.
Best Value
- Professional AI & Creator Workstation: AMD Radeon AI PRO R9700 GPU with 32GB GDDR6 is engineered for AI development, professional content creation, and compute-intensive workloads.
- Massive 32GB Memory Capacity: 32GB of GDDR6 memory on a 256-bit bus provides ample bandwidth for large AI models, 8K video editing, and complex 3D rendering.
- Advanced RDNA 4 with AI Accelerators: 64 Compute Units with 3rd Gen Ray Tracing and dedicated 2nd Gen AI Accelerators for groundbreaking AI performance and visual computing.
- Professional Blower Cooling: Efficient single blower design exhausts heat directly out of the chassis, ideal for multi-GPU workstation and server configurations.
- Enterprise-Grade Thermal Solution: Vapor chamber heatsink with industrial Honeywell PTM7950 thermal interface material ensures reliable cooling under sustained professional loads.
On an iGPU, CPU offload still consumes the same shared memory pool, so moving layers may change placement without freeing physical memory. Use runtime diagnostics to confirm which layers are resident on each device and watch for swapping.
What the published bandwidth and speed figures do—and do not—show
Apple’s M5 versus M4 measurements
Apple reports 153GB/s of memory bandwidth for the tested M5 MacBook Pro system versus 120GB/s for its M4 comparison system. It reports up to 4x faster time to first token on tested LLM workloads when using M5 GPU Neural Accelerators relative to its M4 baseline, and 19–27% higher generation performance on the tested architectures. “Up to 4x” is a maximum from those tests, not a general CPU-versus-iGPU multiplier; the 19–27% range likewise applies to Apple’s stated workloads and configurations.
Why bandwidth alone cannot rank platforms
Tom’s Hardware lists 546GB/s rated bandwidth for its tested M4 Max system and reports model-dependent decode differences against other unified-memory platforms, including Nvidia GB10 and AMD Ryzen AI Max+ 395. The review explicitly cautions that rated bandwidth does not directly predict delivered tokens per second. Kernel efficiency, processor architecture, model sparsity, quantization, cache behavior and software support can outweigh a paper bandwidth advantage. Read the full comparison at Tom’s Hardware.
Interpreting three useful case studies
Apple M5 and MLX
Apple’s 2026 page is valuable for explaining phase behavior, unified-memory operation, quantization and Neural Accelerator support. It is a vendor benchmark on Apple hardware and MLX, so its speedups should not be generalized to Windows or Linux iGPUs, other runtimes or CPU-only implementations. The 24GB memory examples are configuration-specific measurements, not a universal calculator.
Intel benchmark repository
The reported 6.5-to-22.1-token-per-second change demonstrates the importance of offload tuning, but it uses a Core Ultra 9 285K with a discrete Arc A770 and a particular model and software stack. It cannot answer how every Intel iGPU compares with a CPU. Use it as a reminder to verify placement and test multiple splits.
Independent unified-memory comparison
The Tom’s Hardware comparison shows that even systems with substantial rated bandwidth can have different model-dependent results. It supports measuring the complete workload rather than selecting hardware from a bandwidth number alone.
Troubleshooting common failures
The model will not load
- Check free system memory immediately before launch and include the KV-cache reservation for your intended context.
- Reduce context length or batch size, close background applications, or choose a smaller or more aggressively quantized model.
- Confirm that the runtime is not reserving a second copy of the weights for an unsupported GPU backend.
The iGPU is present but speed is unchanged
- Inspect logs for explicit GPU-device selection and unsupported-operator fallbacks.
- Confirm that the installed runtime and operating system meet the backend’s requirements.
- Compare warm-run TTFT and decode separately; a GPU may improve only one phase.
- Check power mode and thermal throttling. A thin system may reduce clocks during sustained generation.
Hybrid mode is slower than CPU-only
- Benchmark several layer or expert-offload counts; the default split may be poorly balanced.
- Measure prompt processing and decode independently to identify which phase regressed.
- Watch memory pressure and swapping. Shared-memory contention can erase an apparent GPU advantage.
Results changed after an update
Record runtime, driver, operating-system and model-loader versions with every benchmark. Backend kernels and device-selection defaults are version-sensitive; rerun the same scripted test after upgrades instead of comparing numbers collected under different stacks.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
A practical selection checklist
- Start with the model. Specify parameter count, architecture, quantization and maximum context.
- Check capacity. Reserve memory for the operating system, runtime overhead and KV cache; do not treat total installed memory as available model memory.
- Check backend support. Verify that your chosen runtime can execute the model on the integrated GPU and that logs confirm actual use.
- Measure both phases. Record TTFT or prompt-processing speed and steady-state decode throughput.
- Test your real workload. Use representative prompts, output lengths and concurrency.
- Choose for reliability. If the measured iGPU gain is small or unstable, CPU-only may deliver a better experience. If the gain is repeatable and the model fits with headroom, use the iGPU.
The best local-LLM platform is therefore the one that fits your target model, supports it reliably and meets your latency and throughput requirements. Memory configuration comes before a nominal bandwidth figure; backend verification comes before assuming acceleration; and a two-phase benchmark comes before declaring a CPU or integrated GPU the winner.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
