Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Start with the 8GB Raspberry Pi 5 if you want to experiment with local large language models (LLMs). Arm explicitly says that the 8GB model is preferred for exploring an LLM. A 4GB board can still run useful assistants, but it should be limited to small, quantized models—roughly 1B to 3B parameters—with a moderate context window and enough memory left for Raspberry Pi OS. Your model’s download size is not the same as its running-memory requirement: the context window, temporary buffers and operating system add overhead. For faster local LLM or vision-language-model (VLM) inference, Raspberry Pi’s AI HAT+ 2 adds 8GB of onboard memory and supports approximately six-billion-parameter LLM/VLM workloads. The original AI HAT+ is intended for vision and does not support LLMs.
How much RAM does a Raspberry Pi 5 need for a local AI model?
An 8GB Raspberry Pi 5 is the sensible starting point for local LLM work. It gives the model room for its weights, runtime buffers and context while leaving memory for the operating system and your application. On a 4GB board, stay with a small, quantized model and avoid unusually long prompts.
As an Amazon Associate I earn from qualifying purchases.
These are planning ranges, not hard compatibility limits. A model can fail below its nominal size if the context is large or another service is consuming memory, while a carefully configured model can sometimes run with less headroom.
| Pi 5 memory | Practical starting range | Use case | What to watch |
|---|---|---|---|
| 4GB | About 1B–3B parameters, quantized | Short chat, simple classification, lightweight automation | Keep context modest; close other applications before loading the model |
| 8GB | About 3B–7B parameters, quantized | More capable chat, coding experiments and longer prompts | Leave headroom for the OS, context window and runtime buffers |
| 8GB Pi 5 plus AI HAT+ 2 | Hailo-supported models up to approximately six billion parameters | Accelerated local LLM and VLM inference | Use the Hailo model list; support depends on the currently installed package and model |
Parameter count is only a first filter. Two models with the same number of parameters can have different memory use because of quantization, vocabulary, architecture and context settings. Treat the table as a selection checkpoint, then verify the model’s quantization and runtime behavior before downloading it.
#1 Best Overall
- HIGH PERFORMANCE: Features 26 TOPS (Trillion Operations Per Second) AI acceleration capability through the Hailo AI Accelerator for advanced machine learning applications
- COMPATIBILITY: Specifically designed for the Raspberry Pi 5, connecting via PCIe interface for optimal data transfer and processing speeds
- COMPACT DESIGN: Measures 65mm x 56.5mm, offering a space-efficient solution while maintaining full functionality as an AI acceleration add-on board
- TEMPERATURE RANGE: Operates reliably in temperatures from 0°C to +50°C (32°F to 122°F), ensuring stable performance in various environments
- SEAMLESS INTEGRATION: Functions as a HAT (Hardware Attached on Top) add-on board, providing plug-and-play compatibility with Raspberry Pi ecosystem
What is the difference between model storage and runtime RAM?
The file you download is the model’s storage footprint, not a promise that it will fit in RAM. Raspberry Pi Foundation starter guidance lists model files ranging from 0.6GB for a 300-million-parameter model to 28GB for a 70-billion-parameter model. Those figures describe disk storage; loading a model also requires temporary buffers and memory for the context window.
| Example model size in the starter table | Listed storage footprint | How to interpret it |
|---|---|---|
| 300M parameters | 0.6GB | A small download, but still needs runtime overhead beyond the file |
| Intermediate entries | Varies by model | Check the exact quantized file rather than estimating from parameter count alone |
| 70B parameters | 28GB | A large storage requirement and not a realistic CPU-only target for a standard Pi 5 |
Plan on at least a 32GB microSD card, the minimum named in Raspberry Pi Foundation project guidance. A 32GB card leaves less usable space than its decimal capacity suggests after formatting and the operating system, so a larger card or SSD is more comfortable when you intend to keep several models. Put model files on an SSD or a high-endurance microSD card when possible; repeated large downloads and model swaps create more wear than ordinary desktop use.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →How does quantization make a model fit?
Quantization stores model weights with fewer bits, reducing both disk use and the memory needed while the model is loaded. Ollama documents quantization levels including q4_K_M. A four-bit variant is usually much easier for a Pi to accommodate than the same model stored at a higher precision, although it can produce somewhat different answers from the full-precision version.
Do not select a quantization solely by its label. Check the actual file size, the model’s stated context limit and the backend’s requirements. A quantized model can still exceed available memory when a long context is allocated or when another model remains loaded.
Why context length matters
The context window holds the prompt and the conversation history, and its cache grows as the exchange gets longer. Increasing context can therefore turn a model that initially loads into one that fails after a few messages. Start conservatively, then raise the context only when you have measured free memory and have a reason to retain more history.
What to do when loading exceeds memory
llama.cpp advises lowering the context setting (the -c option) or unloading another model when loading exceeds available memory. The same principle applies to other runtimes: shorten the prompt, reduce the configured context, use a smaller quantization or stop competing services before trying a larger model.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Which model should I choose for a Pi 5?
Choose by task first, then by memory. Raspberry Pi Foundation starter guidance names gemma:2b, phi and tinyllama as approachable starting points. They are small enough to make experimentation practical, but the exact tag and quantization still determine the download size and memory demand.
| Your task | Model direction | Pi 5 selection guidance | Important check |
|---|---|---|---|
| General assistant or local chat | Small general-purpose model | Start with a 1B–3B quantized model on 4GB; try a 3B–7B quantized model on 8GB | Keep enough memory for the context and your interface |
| Programming help | Coder-tuned model | Prefer a smaller coder model that fits reliably over a larger general model that constantly swaps or fails | Test representative code prompts at your intended context length |
| Camera, image or document understanding | Hailo-supported VLM | Use the AI HAT+ 2 path and select from the models exposed by the installed Hailo package | Do not assume every Ollama model is available on the Hailo accelerator |
| Learning and proof of concept | gemma:2b, phi or tinyllama |
These are named in Raspberry Pi’s starter guidance and provide manageable first downloads | Confirm the current tag and quantization before pulling |
Compare every candidate on six points: RAM tier, parameter count, quantization, context window, storage footprint and execution path. CPU execution is the simplest and most portable. An NPU-assisted model can respond faster or leave the CPU available for other work, but it must be supported by the accelerator’s software stack.
What do I need before installing a model?
Use a 64-bit installation of Raspberry Pi OS. The documented CPU and Hailo paths depend on a 64-bit userland, so check the operating-system architecture before downloading multi-gigabyte files.
- Hardware: Raspberry Pi 5, adequate cooling and a stable power supply; choose the 8GB board when local LLM experimentation is the goal.
- Storage: At least a 32GB microSD card according to Raspberry Pi Foundation project guidance; an SSD is preferable for a collection of large models.
- Network: A connection fast enough for the initial model download. The model runs locally after installation, but the first pull can be several gigabytes.
- Software: Fully updated 64-bit Raspberry Pi OS, plus either a CPU runtime such as Ollama or llama.cpp, or the matching Hailo software for an AI HAT+ 2.
- Memory discipline: Close desktop applications and stop unused model servers while testing so that free RAM reflects the model’s real budget.
How do I install a local model with Ollama on the Pi 5’s CPU?
Ollama is the shortest route from a 64-bit Pi 5 to a working local chat model. The Raspberry Pi Foundation project uses Ollama and starts with gemma:2b; use a smaller or differently tagged model if your memory budget requires it.
- Update Raspberry Pi OS. Open a terminal and apply the normal package and firmware updates before installing a runtime. Reboot if the update requests it.
- Install Ollama. Run
curl -fsSL https://ollama.com/install.sh | sh. The script installs the current Ollama package for the system. - Pull and start a small model. Run
ollama run gemma:2b. Ollama downloads the model if it is not already present and then opens an interactive prompt. - Measure the first conversation. Try short prompts, then gradually increase prompt length. If the process is killed or refuses to load, reduce the model size or quantization and lower the context configured by the client.
- Move model storage when needed. Keep the model directory on an SSD or high-endurance card if the microSD card is short on space or you expect frequent model changes. Preserve enough free space for the operating system and future updates.
The installation command is a convenience script, so review your normal system-administration practices before running it. Keep the runtime bound to the interfaces you intend to expose; a local assistant does not need to be reachable from the public internet.
How should I tune an Ollama model on 4GB?
Use a 1B–3B quantized model, keep conversations short and avoid running a desktop workload beside it. If a model loads but becomes unstable as the chat grows, the context cache—not just the model file—may be exhausting memory. Start a new conversation or lower the client’s context setting before trying a larger model.
How do I run a GGUF model with llama.cpp?
llama.cpp gives you direct control over the GGUF file, context and model-loading behavior. This is useful when you want to compare quantizations or serve more than one model from a directory without automatically loading all of them.
- Prepare the models directory. Create a directory on your SSD or fast storage and place the chosen GGUF files there. Keep one clearly named file per model and record its quantization so you can relate disk usage to your memory test.
- Build or install llama.cpp for the Pi. Use the project’s supported build process for your 64-bit Raspberry Pi OS environment. Confirm that the resulting server binary starts before adding models.
- Start the router without a single-model argument. A conservative pattern is
llama-server --models-dir /home/pi/models --no-models-autoload --jinja --host 127.0.0.1 -c 32768 -ngl 999, where supported. The important choices are the model directory, disabled automatic loading, Jinja chat-template support, a loopback-only host, a bounded context and maximum available offload layers. - Select one model for a test request. Point your local client at the server and load only the model you intend to measure. Do not leave several large models resident while diagnosing a memory failure.
- Adjust after the first load. If loading fails, lower
-cor unload another model. If it loads comfortably, increase context in small increments and watch whether response latency or memory pressure changes.
The -ngl 999 setting is useful where the build supports layer offload, but it does not create memory that the Pi does not have. The CPU still needs enough RAM for weights, buffers and context, and an unsupported option should be removed rather than forced.
Free tools Windows power users keep installed
One-click scans. No signup required.
When should I add an AI HAT+ 2?
Choose the AI HAT+ 2 when you need accelerated local LLM or VLM inference and are willing to use the Hailo software path. Raspberry Pi describes the board as having 8GB of onboard memory and supporting approximately six-billion-parameter LLM/VLM models. It is the relevant HAT for generative AI; the original AI HAT+ is for vision workloads and does not support LLMs.
The HAT does not make every CPU model compatible. Hailo support is tied to the installed runtime, model-zoo package and the models exposed by that software. Check the live list on your device instead of relying on a model name from an older tutorial.
Rank #2
- ⚡ PoE HAT for Raspberry Pi 5 CM5: PoE HAT F is a Power over Ethernet expansion board for Raspberry Pi 5 and CM5, supporting network connection and power input through one Ethernet cable.
- 🔌 802.3af/at PoE+ Support: This PoE+ HAT supports IEEE 802.3af/at network standard and works with compatible PoE power sourcing equipment for compact wired deployment projects.
- 🧊 Active Cooling Fan and Metal Heatsink: The PoE HAT with cooling fan includes a metal heatsink and high-speed active fan, helping improve heat dissipation and operating stability during long-term use.
- 🔋 5V and 12V Output Headers: Onboard 5V and 12V header outputs provide power options for external peripherals, with up to 25W total output under suitable PoE input and cooling conditions.
- 🧩 40-pin GPIO Stackable Header: Standard 40-pin GPIO stackable header fits Raspberry Pi 5 and CM5 expansion, allowing users to connect compatible HATs and custom project interfaces.
How do I install and use a Hailo model with the AI HAT+ 2?
Use a 64-bit Raspberry Pi OS installation and complete the Hailo setup before attempting a chat request. Package names and supported models can change, so follow the version-matched Hailo instructions supplied for your board and software release.
- Update the Pi. Update Raspberry Pi OS packages and firmware, then reboot if required. A mismatched kernel, firmware or accelerator package can prevent the device from being detected.
- Install Hailo dependencies. Install the Hailo runtime components required by the AI HAT+ 2 and verify that the accelerator is visible before adding generative-AI software.
- Install the Hailo GenAI Model Zoo package. Use the package version that matches your Hailo runtime. This supplies the model definitions and integrations used by the local GenAI service.
- Start the local service. Launch
hailo-ollama. It provides an Ollama-compatible local API backed by the Hailo accelerator. - List models exposed by the service. Request
http://localhost:8000/hailo/v1/listfrom the Pi. Treat this response as the authoritative list for that installation; names and versions are not permanent. - Pull a listed model. Use the local API or its compatible client to pull one of the models returned by the list endpoint. Raspberry Pi’s official example includes
qwen2:1.5b, but use the name your current endpoint reports. - Send a chat request. Point your client at the local service and verify that inference is using the Hailo path. Compare response time and CPU load with a CPU-only run using the same prompt.
If the endpoint returns no models, do not substitute a random Ollama tag. Check that the GenAI Model Zoo package matches the runtime, restart hailo-ollama, and query the list again. A model absent from the list is not established as supported by that installation.
How can I tell whether a model really fits?
Use a repeatable loading test instead of judging by download size. Start with no other model loaded, record free memory, load the candidate and send prompts that resemble your real use. Test both a short prompt and the longest context you expect to keep.
- Record the model file size, quantization and advertised context limit.
- Boot into the same services you will normally run, but close unrelated desktop applications.
- Load the model with a conservative context setting.
- Send a short prompt, then a longer multi-turn prompt. Watch for process termination, swap activity, severe latency or an error while allocating the context.
- Increase context only if the first two tests leave comfortable headroom.
- Repeat after a reboot so that cached files and leftover processes do not hide the model’s true requirement.
A successful one-line reply is not proof of a good fit. A model that leaves no room for the operating system can become unreliable during a longer conversation, when a camera frame is added or when your application opens another service.
What are the common installation and memory failures?
The model will not load
The usual causes are an oversized parameter count, a higher-precision file, an excessive context or another model already resident. Choose a smaller quantized file, lower the context option and unload competing models before changing hardware.
The model loads, then the process is killed
This generally indicates that total working memory rose after the initial load. Reduce context, shorten prompts and stop background applications. On a 4GB Pi, moving from a 3B model to a 1B–2B model may be more effective than trying increasingly aggressive context settings.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
The microSD card fills during installation
The 32GB minimum is a starting point, not a guarantee that several large models will fit beside Raspberry Pi OS. Remove unused model files or move the model directory to an SSD, and retain free space for updates.
Hailo models are missing
Confirm that the board is an AI HAT+ 2 rather than the vision-only AI HAT+, that the operating system is 64-bit and that the Hailo runtime and GenAI Model Zoo package match. Then restart hailo-ollama and query http://localhost:8000/hailo/v1/list again.
Responses are unexpectedly slow
CPU inference is naturally constrained by the Pi’s processor, and a larger model or context increases work per token. Reduce the model size or context for interactive use, or move to an AI HAT+ 2 model that is explicitly supported by the Hailo stack.
Which installation path is right for my project?
| Priority | Recommended path | Why | Trade-off |
|---|---|---|---|
| Fastest first experiment | Ollama on the CPU | Simple install and a direct command such as ollama run gemma:2b |
CPU speed and system RAM limit model size |
| Fine control over files and context | llama.cpp on the CPU | Direct GGUF selection, explicit context control and a model directory | More setup and more decisions about server flags |
| Camera or document understanding | AI HAT+ 2 with Hailo | Designed for supported local LLM/VLM acceleration and includes 8GB onboard memory | Requires matched Hailo software and only the currently exposed model set |
| Vision-only acceleration | AI HAT+ | Appropriate for supported vision workloads | It does not support LLMs, so it is not a substitute for AI HAT+ 2 in this guide |
For most first-time users, an 8GB Pi 5, a 32GB-or-larger card and a small quantized CPU model provide the lowest-friction learning path. Add the AI HAT+ 2 when latency, VLM input or CPU availability matters enough to justify the Hailo-specific setup.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallHow do I keep a Pi-based AI installation reliable?
- Keep one known-good model. Test new quantizations separately so a failed download or incompatible file does not remove your working baseline.
- Pin your expectations to the current model list. Hailo-supported names can change with the package release; query the local endpoint after upgrades.
- Reserve memory for the application. A chatbot, web interface, camera pipeline or database shares RAM with the model.
- Use realistic prompts during testing. Include the longest normal conversation and largest document or image input, not just “hello.”
- Prefer storage headroom. Keep model files off a nearly full boot card and maintain a recovery image or backup before major runtime changes.
- Limit network exposure. Keep Ollama, llama.cpp and Hailo services on loopback unless you have deliberately configured authentication and firewall rules for a private network.
The reliable choice is the model that leaves measurable room for your complete workload, not the largest file that happens to start once. Begin with the 8GB Pi 5 and a small quantized model, establish a repeatable test, then scale the model, context or accelerator one change at a time.
Frequently Asked Questions
Will buying a larger microSD card make a model run if it does not fit in RAM?
No. A larger card solves storage capacity, not the memory needed while the model is loaded. It is useful for keeping multiple quantized files, but you still need a model and context that fit the Pi’s available RAM.
Can I run a 70-billion-parameter model locally on a standard Pi 5?
The starter table lists about 28GB just for a 70B model file, before runtime overhead and context memory. That makes it an impractical target for ordinary Pi 5 CPU experimentation; choose a much smaller quantized model instead.
Does the AI HAT+ 2 increase the Raspberry Pi 5’s system RAM?
It provides 8GB of onboard memory for its supported accelerated LLM/VLM workloads, but it does not turn every CPU model into a Hailo-compatible model. Your application still needs the matching Hailo runtime and a model exposed by the local service.
Why did a model work with a short prompt but fail after a long conversation?
The conversation consumes context-cache memory in addition to the model weights. Start a new conversation, reduce the configured context or use a smaller quantization so the cache and operating system retain adequate headroom.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
