To run a large language model (LLM) on hardware you control, first choose the task and a model that fits your available memory; then select a runtime suited to your operating system, hardware and intended use. A desktop app is usually the simplest way to try local chat. A command-line runner or serving engine offers more control when you need an API or a configurable deployment. No single model size or backend is right for every machine.
Start with the work you want the model to do
Decide what you want to run before choosing software or buying hardware. Drafting, rewriting, summarizing and question answering in a desktop chat app have different operational demands from serving requests through an API, connecting tools, or handling multiple users. Those needs influence how much memory you need, which runtime is suitable, and whether ease of setup or configuration control matters more.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
GMKtec EVO-X2 AI Mini PC Ryzen Al Max+ 395 Superchip 128GB LPDDR5X 2TB SSD | $3,649.99 | Buy on Amazon |
As an Amazon Associate I earn from qualifying purchases.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →- Interactive, single-user use: Start with a desktop application and a model that fits the machine. This is a practical route for trying everyday prompts without first configuring a server.
- Local command-line use or an API: Choose a runner that can download and run a model and, if needed, expose an API. Check its supported model formats and hardware backends.
- Configurable serving: Consider a serving engine when you need more control over deployment or request handling. Confirm its operating-system and hardware requirements before installing it.
Backend choice depends on more than the model name: operating system, model format, GPU architecture and memory, API requirements, and throughput target all matter. NVIDIA’s local-AI guidance identifies those factors as part of choosing an inference backend.
Check memory fit before choosing a model
Available memory is a central constraint, but a model’s parameter count alone cannot predict whether it will fit or feel responsive. Memory use depends on the model and its format, quantization, context length, and runtime overhead. The context includes the prompt, conversation history, tool output and retrieved material the model can consider; longer context consumes additional memory.
#1 Best Overall
- EVOLUTION RYZEN AI MAX+ 395 MINI PC - GMKtec EVO-X2 is the next evolution in AI mini PC Ryzen Strix Halo series. Thanks to AMD Simultaneous Multithreading (SMT) the core-count is effectively doubled, to 32 threads. Ryzen AI Max+ 395 has 64 MB of L3 cache and can boost up to 5.1 GHz, depending on the workload. The Ryzen AI Max+ 395 is currently rated as the "most powerful x86 APU" on the market for AI computing.
- AI NPU with XDNA 2 ARCHITECTURE - Powered by 16 “Zen 5” CPU cores, 50+ peak AI TOPS XDNA 2 NPU and a truly massive integrated GPU driven by 40 AMD RDNA 3.5 CUs, the Ryzen AI MAX+ 395 is a transformative upgrade and delivers a significant performance boost over the competition. The Ryzen AI Max+ 395 excels in consumer AI workloads like the llama.cpp-powered application: LM Studio. Shaping up to be the must-have app for client LLM workloads, LM Studio allows users to locally run the latest language model without any technical knowledge required and unleash their creativity and productivity.
- AMD RADEON 8090S iGPU GAMING PC - The AMD Radeon RX 8060S offers all 40 CUs with up to 2.9 GHz graphics clock and uses the new RDNA 3.5 architecture. The powerful iGPU is positioned between an RTX 4060 and 4070 laptop GPU and therefore enables gaming in FHD at maximum details in most demanding games. The 8060S can also utilize the full 128GB pool, which is perfect for running LLMs such as Deepseek 70B Q8, which runs comfortably on this machine.
- EIGHT CHANNEL LPDDR5X - LPDDR5X is a new ground breaking memory small form factor installed on-board. With blazing speeds up to to 8000MT/s, it runs 1.5x faster than the DDR5 SODIMMs; 90% better performance over DDR5 SODIMMs in video conferencing and photo editing; 30% better performance in productivity apps; 12% better performance in digital content workloads.
- QUAD SCREEN 8K DISPLAY SUPPORT - EVO-X2 AI Mini PC support 4-screen 4K/8K output via HDMI 2.1 (8K@60Hz), DisplayPort 1.4 (4K@60Hz), and dual USB 4 40Gbps Transfer speed (supporting PD3.0/DP1.4/DATA). Ideal for gaming, video editing, and multitasking, it provides expansive and crisp multi-display support.
Quantization stores model weights in a lower-precision format to reduce memory requirements. More aggressive quantization can affect output quality. NVIDIA’s RTX guide describes NVFP4 or Q4_K_M as a balance of throughput, accuracy and memory for its ecosystem; treat that as NVIDIA guidance, not a universal or independent benchmark.
RTX starting points in NVIDIA’s guide
NVIDIA’s RTX guide, accessed October 7, 2026, suggests the following starting models by GPU memory. These are recommendations for its ecosystem, not compatibility guarantees. Actual fit and performance depend on the model format, quantization, context, runtime and workload; model names and supported formats can change.
| RTX GPU memory | NVIDIA’s suggested starting model | How to interpret it |
|---|---|---|
| 6–8 GB | Qwen 3.5 4B | A suggested starting point, not a guarantee that every format or context setting will fit. |
| 12–16 GB | Qwen 3.5 9B or Gemma 4 12B | Check the specific model format, quantization and context requirements. |
| 24 GB or more | Qwen 3.6 27B | Still verify the runtime’s memory use and the context you intend to run. |
| DGX Spark | Qwen 3.6 35B | A separate NVIDIA recommendation for this system; it is not a general GPU-memory tier. |
If a model exceeds available GPU memory, some runtimes can divide work between CPU and GPU. That can make a larger model runnable, but it does not make hybrid inference as fast as keeping the model fully on the GPU. Before buying a graphics card, choose the workload and model, then check that the card’s VRAM can accommodate the model, quantization and context you plan to use.
NIM memory estimates are product-specific
NVIDIA NIM 1.7.0 gives rough memory estimates for example model deployments. NVIDIA says actual use can vary, and the estimates do not apply to trtllm_buildable profiles. These are NIM-specific guidelines, not universal minimums for quantized models running in consumer desktop software.
| Example in NVIDIA NIM 1.7.0 | Rough memory estimate |
|---|---|
| Llama 8B | About 15 GB |
| Llama 70B | About 131 GB |
| Mistral 7B Instruct v0.3 | About 14 GB |
| Mixtral 8x7B Instruct v0.1 | About 88 GB |
Pick a runtime for your setup and use case
Desktop apps, general-purpose local runners and serving engines solve different problems. A simple chat interface can get you experimenting quickly; an API or serving engine may be a better fit when you need deployment controls. Check hardware support and operating-system requirements for the specific software version you intend to install.
| Route | Best suited to | What to know |
|---|---|---|
| LM Studio, Ollama or llama.cpp as a simple chat path | Trying local drafting, rewriting, summarizing or Q&A | NVIDIA lists these as simple options for installing a chat app and downloading a model that fits the GPU. Their features and setup differ, so check the current version and model-format support. |
| llama.cpp command line or API server | Running models directly, experimenting from the command line or exposing an API | Its quick start documents downloading and running a model and launching an OpenAI-compatible API server. It supports multiple hardware backends and quantization options. |
| vLLM | More configurable serving | NVIDIA’s RTX guide points advanced RTX and DGX users to vLLM and says its path requires Linux. Its current CPU documentation also describes Docker-based CPU serving. The documentation calls its Apple Silicon CPU support experimental; its Metal GPU route uses a community-maintained plugin. |
| NVIDIA NIM | A separately packaged NVIDIA deployment path | Self-hosting requirements in the NIM 1.7.0 guide include Linux, a compatible NVIDIA driver, Docker, CUDA and an NVIDIA AI Enterprise license. These requirements and terms apply to that NIM version, not to local model software generally. |
Running locally does not settle every privacy question
“Local” describes where model inference runs, but it does not by itself tell you what happens to every input or connection. NVIDIA says prompts, files and local context can remain on the machine in local workflows. Check separately whether the app or a connected agent uses web lookup, external integrations or telemetry, and whether an API server is exposed beyond the device. The data path depends on how those features and network access are configured.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Choose a hardware route that matches what you have
NVIDIA GPUs
NVIDIA’s RTX recommendations give starting model tiers for its GPUs, while its broader backend guidance emphasizes GPU architecture, memory, model format and workload. Use the suggested tiers as a shortlist, then verify that the specific runtime and model format support your card. A graphics card with sufficient VRAM may be appropriate when the selected model and workload call for it; there is no single card recommendation established here.
Apple Silicon and other non-NVIDIA hardware
llama.cpp documents Apple Silicon as a first-class target using ARM NEON, Accelerate and Metal. It also documents CUDA for NVIDIA, HIP for AMD, Vulkan, SYCL and other backends, as well as CPU/GPU hybrid inference. Support varies by backend and runtime, so confirm compatibility for the software version, operating system and model you plan to use.
Ollama’s March 30, 2026 post described its Apple Silicon implementation using MLX as a preview and recommended more than 32 GB of unified memory for its showcased Qwen3.5-35B-A3B setup. Its reported test was run on March 29, 2026, using that particular model and quantization configurations. That recommendation and test do not establish a general memory requirement or speed result for other models and Macs.
CPU-only use
CPU inference is another route, especially when the existing computer has no supported GPU or you want to test a small workload without adding hardware. vLLM documents CPU serving through Docker. CPU support and performance depend on the hardware and software path; do not assume it will match GPU inference. For Apple Silicon with vLLM specifically, the documented CPU support is experimental, and the Metal GPU path relies on a community-maintained plugin.
A practical way to get started
- Write down the workload. Decide whether you need private, interactive chat, command-line use, an API, or a configurable serving setup. Note any need for long context, tool use or multiple users.
- Inventory the machine. Record the operating system, available GPU and its memory, system or unified memory, and any hardware backend supported by the runtime you are considering.
- Shortlist a model and format. Use vendor recommendations only as starting points. Check the exact model version, format, quantization and license, and allow memory for the context length and runtime overhead.
- Choose the simplest suitable runtime. For initial chat, try a desktop option listed by NVIDIA. For command-line use or an API, consider llama.cpp. For more configurable serving, check vLLM’s operating-system and hardware requirements. Use NIM only if its product-specific prerequisites and license fit your deployment.
- Run a representative trial. Test the prompts and context length you will actually use. Watch for memory pressure, model-loading failures, slow generation or degraded output quality. If it does not fit or perform acceptably, try a smaller model or a less demanding context or quantization before deciding that new hardware is necessary.
- Check connections and exposure. Review integrations, web access, telemetry and API binding settings. A server reachable from other devices has different exposure from a chat app used only on the local machine.
- Benchmark the actual task before committing. Compare prompt processing and tokens per second on the same machine, model, quantization and context settings. A speed result from another model or setup is not a reliable prediction for yours.
What to verify before installing or upgrading
- Model license: Local execution does not remove the model’s own license terms; check its permitted uses.
- Runtime compatibility: Confirm supported operating system, model format, GPU architecture and backend for the current software release.
- Memory headroom: Account for weights, context and runtime overhead rather than treating parameter count or a vendor tier as a fit guarantee.
- Deployment terms: Check any product-specific prerequisites, drivers, containers and licenses. NVIDIA NIM’s stated requirements are specific to NIM 1.7.0.
- Performance expectations: Measure the model and workload on your own machine. The figures described above are not a matched cross-platform comparison.
Model releases, runtime support and recommended formats move quickly. NVIDIA’s RTX model guidance and the llama.cpp and vLLM documentation cited here were accessed October 7, 2026; check current compatibility before purchasing hardware or building a deployment around a particular release.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
