To run a GGUF model locally, choose a compatible model file and an inference runtime: use llama.cpp to load the file directly, Ollama to import it into Ollama with a Modelfile, or vLLM if you already use vLLM and are comfortable with its experimental GGUF plugin. For most people who want straightforward local-file inference, llama.cpp or Ollama is the more practical starting point.
GGUF is a model-file format, not a program that runs models. It stores model tensors and standardized metadata; you still need a runtime that supports the model’s architecture and file configuration. The instructions below reflect official project documentation accessed October 7, 2026; compatibility and installation details can change.
As an Amazon Associate I earn from qualifying purchases.
Which runtime should you choose?
| Runtime | How it accepts GGUF | Best fit | Important caveat |
|---|---|---|---|
| llama.cpp | Load a local file directly with llama-cli or llama-server. |
Direct command-line inference or a local HTTP server, with CPU, GPU, and hybrid backend options. | Installation or build steps vary by operating system and desired backend. |
| Ollama | Point a Modelfile at the local GGUF, then create an Ollama model. | People who want to bring a local GGUF into Ollama’s model workflow. | Ollama does not quantize the model during import, and its import instructions do not establish universal compatibility with every GGUF. |
| vLLM | Install the separate vllm-gguf-plugin, then serve a GGUF with a tokenizer. |
Readers already working with vLLM who specifically need its serving workflow. | vLLM describes GGUF support as highly experimental and under-optimized, with possible incompatibilities. |
There is no fair, standardized speed comparison among these three runtimes in the documentation cited here, so a blanket speed ranking would be misleading. Choose based on your workflow, model compatibility, and hardware rather than an assumed performance order.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesHow do you choose a GGUF model and check your hardware?
Check the exact model and file before downloading
Use the Hugging Face Hub’s GGUF filter to find candidate files. Its GGUF documentation describes a viewer that can expose model metadata and tensor information. Before downloading, read the specific model card and verify:
#1 Best Overall
- Built for Local AI Development: AMD Ryzen AI Halo is designed for local AI development and inference, featuring 128GB unified memory and support for up to 200B parameter models to build and run intensive AI workloads locally.
- 128GB Unified Memory: Features 128GB LPDDR5x unified memory at 8000 MT/s with 256 GB/s memory bandwidth, providing a shared memory pool across the CPU, GPU, and NPU to support larger AI models.
- AMD Ryzen AI Max+ 395 Processor: Features 16 cores, 32 threads, and Zen 5 architecture, paired with AMD Radeon 8060S integrated graphics featuring 40 RDNA 3.5 compute units and an AMD XDNA 2 NPU with up to 50 TOPS.
- Linux AI Developer Platform: Purpose-built for Linux-based AI development with full AMD ROCm software support and preloaded tools, models, and workflows optimized for local AI development.
- Compact, Connected Design: Includes a 2TB M.2 SSD, 10GbE LAN, Wi-Fi 7, Bluetooth 5.4, USB-C connectivity, and HDMI 2.1b.
- The model architecture and whether your intended runtime supports it.
- The exact file and quantization label you intend to use.
- Any tokenizer or chat-template instructions from the model publisher.
- The model’s license and applicable use terms.
Quantization labels describe different ways model weights are represented, but the label alone does not tell you which file will provide the best quality or speed for your workload. Check the model publisher’s guidance and choose for your own balance of memory use and output quality.
Estimate fit using your workload, not a universal RAM table
No universal minimum RAM or VRAM figure applies to all GGUF models and runtimes. Fit depends on the selected model and quantization, context length, runtime, backend, and how much memory is available to inference. A model’s file size is useful when comparing downloads, but it is not a complete guarantee of how much total system memory the running workload will need.
Compare the model’s requirements and file details with your available system memory and GPU memory, the context you expect to use, and the throughput you need. llama.cpp documents CPU inference as well as acceleration backends including Metal, CUDA, HIP, Vulkan, and SYCL, and describes CPU-plus-GPU inference that can partially accelerate models exceeding available VRAM. Those are project capabilities, not guarantees that every model, build, driver, or operating system will use a particular backend successfully.
How do you run a GGUF file with llama.cpp?
llama.cpp is the direct route when you already have a compatible local GGUF file. The project README documents both interactive CLI inference and a local server. First install or build llama.cpp using instructions appropriate to your operating system and the backend you want; the project’s installation options differ by platform and acceleration method.
Run an interactive prompt
From a terminal where llama-cli is available, run:
llama-cli -m model.ggufReplace model.gguf with the path to your file if it is not in the current directory, for example /path/to/model.gguf. If the model contains a supported chat template, conversation mode may activate automatically. If it does not, llama.cpp documents adding -cnv and an appropriate --chat-template; use the template guidance for that model rather than guessing.
Start a local HTTP server
To serve the file on your machine, run:
llama-server -m model.gguf --port 8080The documented example provides a basic web interface at http://localhost:8080 and a chat-completions route at /v1/chat/completions. A server command starting successfully does not by itself guarantee that the chosen model’s conversation template or backend is configured as you need; check the model’s instructions and the server output if responses are malformed or startup fails.
How do you use a GGUF file with Ollama?
Ollama’s documented import flow is to create a Modelfile that points at the GGUF, then run ollama create. This assumes Ollama is already installed and available in your terminal.
-
Create a file named
Modelfilecontaining aFROMline with the local GGUF path:Rank #3
BOSGAME M5 AI PC MAX+ 395, 128GB LPDDR5x 8000MT/S- 【AMD Ryzen AI Max+ 395 Processor】 Features the 16-core, 32-thread Ryzen AI Max+ 395 workstation processor (up to 5.1GHz, 80MB cache) with an integrated NPU. Built for software compiling, 3D rendering, and local AI workflows. This desktop runs 128B models (like GPT-OSS-120B) at over 40 Tokens/s and 235B MoE models at 15 Tokens/s right on your desk.
- 【128GB LPDDR5X RAM & Variable VRAM】 Uses AMD Variable Graphics Memory (VGM) technology to share its 128GB onboard LPDDR5X system memory. This Unified Memory Architecture lets you allocate up to 96GB of memory as dedicated VRAM to run large 4-bit quantized models up to 128B or high-precision FP16 models up to 32B without professional studio GPUs.
- 【Radeon 8060S Graphics & Quad 8K Display】 Integrated Radeon 8060S Graphics (2900MHz) handle CAD modeling, AAA gaming, and 8K media editing. With 1x HDMI 2.1, 1x DP 1.4, and 2x USB4 ports, you can run four independent 8K@60Hz monitors simultaneously, providing an expansive multi-monitor workspace for day traders, video editors, and designers.
- 【40Gbps USB4 & SD 4.0 Card Reader】 Two USB4 Type-C ports deliver 40Gbps data transfer, video output, and power delivery. A front-facing SD 4.0 slot supports high-speed SDXC cards up to 300MB/s, allowing photographers and videographers to move large files quickly without external hubs or dongles.
- 【USB4 Multi-Device Daisy Chaining】 Equipped with dual 40Gbps USB4 ports that support multi-device daisy-chaining and cluster linking. You can link multiple M5 units or external expansion nodes together to scale up your local AI compute power. This hardware configuration helps developers expand processing capabilities for larger language models and distributed computing setups.
FROM /path/to/file.gguf -
In the directory containing the Modelfile, create the Ollama model:
ollama create my-model -
Use the resulting model name,
my-model, with your Ollama workflow.
Importing a model split across files
Keep the original shard filenames together and set the Modelfile’s FROM path to a wildcard that matches every shard, as in Ollama’s example:
FROM /path/to/model-*.ggufDo not rename shards casually: the wildcard must match the files that make up the model. If creation fails, check that the path is correct and that it matches all required shards.
Rank #4
- AMD RYZEN AI MAX+ 395 MINI PC – THE NEXT GENERATION AI WORKSTATION --- GMKtec EVO-X3 introduces the next evolution of desktop AI computing powered by AMD Ryzen AI Max+ 395 processor. Featuring 16 cores and 32 threads, Zen 5 architecture, TSMC 4nm FinFET process, up to 5.1GHz boost frequency, and 64MB L3 cache, EVO-X3 delivers flagship-level performance for AI applications, professional creation, gaming, and demanding multitasking. With up to 126 TOPS AI performance, this compact AI workstation brings powerful local computing to your desktop.
- AMD XDNA 2 NPU – 50 TOPS DEDICATED AI ENGINE FOR LOCAL AI --- Equipped with AMD XDNA 2 architecture NPU delivering up to 50 TOPS AI acceleration, EVO-X3 enables efficient local AI processing for generative AI, AI assistants, image creation, content production, and intelligent workflows. By processing AI tasks directly on-device, it helps reduce cloud dependency, improve response speed, and enhance data privacy. Run advanced AI applications locally with smoother performance and greater control over your data.
- AMD RADEON 8060S GRAPHICS – RDNA 3.5 POWER WITH DESKTOP-CLASS PERFORMANCE --- EVO-X3 features AMD Radeon 8060S Graphics with 40 Compute Units and up to 2900MHz frequency based on advanced RDNA 3.5 architecture. Delivering graphics performance comparable to RTX 4070-class laptop GPUs, it provides smooth 1080P high-quality gaming, accelerated video editing, 3D rendering, and creative workloads. Experience powerful integrated graphics performance without the size and power consumption of a traditional desktop tower.
- EIGHT CHANNEL LPDDR5X - LPDDR5X is a new ground breaking memory small form factor installed on-board. With blazing speeds up to to 8000MT/s, it runs 1.5x faster than the DDR5 SODIMMs; 90% better performance over DDR5 SODIMMs in video conferencing and photo editing; 30% better performance in productivity apps; 12% better performance in digital content workloads.
- 128GB LPDDR5X 8000MT/s MEMORY – MASSIVE BANDWIDTH FOR AI AND CREATIVE WORK --- Equipped with up to 128GB LPDDR5X memory running at 8000MT/s, EVO-X3 provides exceptional bandwidth for large AI models, professional software, content creation, and heavy multitasking. The unified memory architecture allows more flexible resource allocation between CPU and GPU, making it ideal for local AI inference, large model deployment, video production, engineering applications, and advanced creative workflows.
Quantize before importing if needed
Ollama explicitly says it does not quantize GGUF models during import. If the file is not already in the quantization you want, prepare or quantize it with a GGUF tool before creating the Ollama model. An import failure can also indicate unsupported architecture or metadata; the documented workflow does not promise that every GGUF configuration will work.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Can vLLM run GGUF models?
Yes, but vLLM’s GGUF route is explicitly experimental. Its current documentation says support has moved to the out-of-tree vllm-gguf-plugin and warns that the feature is under-optimized and may conflict with other features. This makes it a less straightforward choice than loading a GGUF with llama.cpp or importing one with Ollama.
Install the GGUF plugin
With the documented uv package workflow, install the plugin in the environment used for vLLM:
uv pip install vllm-gguf-pluginServe a local file with the base tokenizer
vLLM’s documentation gives a local-file serving example with a tokenizer specified separately. Its example uses the Qwen3 tokenizer:
vllm serve /path/to/model.gguf --tokenizer Qwen/Qwen3-0.6BUse the base model’s appropriate tokenizer rather than assuming the GGUF’s tokenizer metadata will be converted reliably. vLLM recommends this because GGUF tokenizer conversion can be slow and unstable, particularly for models with large vocabularies. The tokenizer shown above is an example, not a universal match for other models.
The vLLM documentation also shows serving a Hub repository with a quantization and an optional tensor-parallel setting for multi-GPU use. For some models whose metadata cannot be converted into a Hugging Face configuration, it documents a manual --hf-config-path option. Follow the current vLLM GGUF instructions for the exact repository syntax and model-specific configuration you need; experimental support can be incompatible with other vLLM features.
Quick Recap
What should you try when a GGUF will not load?
- Check the path and file set. Confirm that the runtime can read the file. For a split Ollama model, preserve the shard names and make sure the wildcard matches every shard.
- Verify architecture and metadata support. A GGUF extension does not mean every model architecture or metadata configuration is supported by every runtime. Check the model card and the selected runtime’s current compatibility guidance.
- Check the tokenizer and chat template. Use the model publisher’s tokenizer and conversation guidance. With llama.cpp, a missing or unsuitable chat template may require
-cnvand a matching--chat-template. With vLLM, specify the base model tokenizer as recommended. - Reassess memory and context. A model that cannot be loaded with the available resources may require a different quantization, a shorter context, or a backend configuration that your build supports. Do not assume a file-size estimate alone establishes runtime memory needs.
- Separate runtime problems from model compatibility. Installation and acceleration depend on platform, build, and backend. If the command or import fails, check the current platform-specific runtime instructions as well as the exact model’s compatibility notes.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Free tools Windows power users keep installed
One-click scans. No signup required.
