Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallYes. A Raspberry Pi 5 can run a small language model entirely locally. The simplest route is Ollama running on the Pi’s CPU with a compact model such as Gemma 2B, Phi, or TinyLlama. You need realistic expectations: model size determines storage, memory use and response speed, and larger models can become impractical on a particular Pi 5 configuration.
For supported hardware acceleration, add the Raspberry Pi AI HAT+ 2. It uses a Hailo-10H neural-processing unit and has its own memory, but it is optional for CPU inference. The similarly named AI HAT+ is a different product and is not listed by Raspberry Pi as supporting local large-language-model workloads.
As an Amazon Associate I earn from qualifying purchases.
Choose the right way to run a model
There are two supported approaches. Start with CPU inference if your goal is to learn, prototype a local assistant or test a model with the hardware you already own. Choose the AI HAT+ 2 route when you specifically want the documented Hailo accelerator and are prepared to install its separate software stack.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →| Approach | What it uses | Best for | Main limitation |
|---|---|---|---|
| Ollama on Raspberry Pi 5 | Pi 5 CPU and system memory | Lowest-complexity experimentation with small models | Speed and model size depend on the Pi’s memory, storage and cooling |
| AI HAT+ 2 with Hailo Ollama | Pi 5 plus Hailo-10H NPU and 8 GB onboard memory | Supported accelerated local LLM and VLM workloads | Requires the add-on, its software package and Hailo-compatible models |
Raspberry Pi has not published a directly comparable benchmark between these two routes. Do not treat the accelerator’s theoretical specification as a guaranteed tokens-per-second result for every model.
#1 Best Overall
- Hailo-10H AI accelerator delivering 40 TOPS (INT4) inferencing performance.
- Performance for computer vision models comparable to the Raspbery Pi AI HAT+ (26 TOPS).
- Runs generative AI models efficiently using 8GB on-board RAM.
- Fully integrated into Raspbery Pi’s camera software stack.
- Conforms to Raspbery Pi HAT+ specification.
What you need before installing anything
- A Raspberry Pi 5 with a working Raspberry Pi OS installation and network access for downloading software and model files.
- A microSD card with at least 32 GB. This is the minimum storage recommendation in Raspberry Pi’s educational Ollama guide; larger models require more space.
- A suitable power supply and a way to keep the board cool. Raspberry Pi recommends the Raspberry Pi Active Cooler, particularly for benchmarks and sustained intensive workloads.
- For the accelerated route, a Raspberry Pi AI HAT+ 2 fitted to the Pi 5. The HAT’s included heatsink should be installed, especially when running sustained workloads.
Storage capacity is not the same as usable model capacity. The operating system, package caches, logs and downloaded model files all share the card. Leave headroom rather than filling a 32 GB card with one large download.
Route 1: run a model with Ollama on the Pi 5 CPU
Raspberry Pi’s beginner project guide describes Ollama on Raspberry Pi 4 and 5 and recommends starting with three small model choices: gemma:2b, phi and tinyllama. These tags can change as model libraries are updated, so check the current tag shown by Ollama when you install.
Install Ollama
- Install Raspberry Pi OS and update it using your normal system-maintenance procedure.
- Open Raspberry Pi’s current “Run a large language model on your Raspberry Pi” project guide and use the installation command displayed there. Installation instructions are living software documentation and may change; do not rely on a copied command from an old tutorial.
- After installation, confirm that the
ollamacommand is available in the terminal before downloading a model.
The guide does not imply that every model in Ollama’s library will run comfortably on every Pi 5 memory configuration. Begin with one of the small examples and monitor free storage and memory before trying anything larger.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Download and start a starter model
Use Ollama’s run command with one of the guide’s starter tags:
ollama run gemma:2b
ollama run phi
ollama run tinyllamaRun one model at a time while you are establishing a baseline. The first run downloads model data; later runs can reuse the local copy. If a tag has been renamed or removed, use the current model name shown by Ollama rather than guessing a replacement.
Use the terminal session effectively
- Ask short prompts first. They make it easier to distinguish model startup time from generation time.
- Keep prompts and expected answers modest. A small model may produce useful summaries, classifications or short explanations but will not match the reasoning quality of a desktop-class model.
- Stop the model when you are finished if you need to reclaim memory for another task.
- Watch available storage before downloading a second model. Larger models consume more disk space and generally need more memory and take longer to run.
What CPU inference feels like
Response speed depends on the model file, quantization, prompt length, number of CPU threads, operating-system build, memory pressure and temperature. A model can appear responsive for a short prompt and become much slower when the context grows. Treat the Pi as a local, private inference device for appropriately sized models, not as a replacement for a high-end GPU server.
Route 2: use the Raspberry Pi AI HAT+ 2 and Hailo Ollama
Raspberry Pi’s AI software documentation describes a separate accelerated setup for a Pi 5 with the AI HAT+ 2. The documented software path uses the Hailo GenAI Model Zoo Debian package, version 5.1.1, and the hailo-ollama server. Models come from Hailo’s GenAI Model Zoo rather than being assumed to work merely because they appear in a general Ollama library.
Rank #2
- HIGH PERFORMANCE: Features 26 TOPS (Trillion Operations Per Second) AI acceleration capability through the Hailo AI Accelerator for advanced machine learning applications
- COMPATIBILITY: Specifically designed for the Raspberry Pi 5, connecting via PCIe interface for optimal data transfer and processing speeds
- COMPACT DESIGN: Measures 65mm x 56.5mm, offering a space-efficient solution while maintaining full functionality as an AI acceleration add-on board
- TEMPERATURE RANGE: Operates reliably in temperatures from 0°C to +50°C (32°F to 122°F), ensuring stable performance in various environments
- SEAMLESS INTEGRATION: Functions as a HAT (Hardware Attached on Top) add-on board, providing plug-and-play compatibility with Raspberry Pi ecosystem
Understand the hardware distinction
The AI HAT+ 2 contains a Hailo-10H NPU rated at 40 TOPS INT4 and 8 GB of onboard memory. Raspberry Pi’s hardware documentation describes support for local LLMs and VLMs up to approximately six billion parameters. “Approximately” matters: supported model formats, runtime compatibility and available memory still determine what will run.
Do not substitute the older AI HAT+ for the AI HAT+ 2. Raspberry Pi’s comparison lists the AI HAT+ as not supporting LLM or VLM use, while the AI HAT+ 2 is the model documented for this workload.
Install the Hailo software stack
- Mount the AI HAT+ 2 on the Raspberry Pi 5 and fit the included heatsink.
- Install the Hailo GenAI Model Zoo Debian package version 5.1.1, following the package and repository instructions in Raspberry Pi’s current AI software documentation.
- Start the documented Hailo Ollama server with
hailo-ollama. - List the models exposed by that server. Use the listing command in the current documentation so you see the models available for your installed package and hardware.
- Pull a compatible model. Raspberry Pi’s example is
qwen2:1.5b; use the exact tag returned by the server’s model list.
hailo-ollama
ollama pull qwen2:1.5bThe commands above show the documented server name and example model tag. Package subcommands and model-management syntax can change, so use the version-matched instructions for the remaining list, pull and prompt operations.
Send a prompt through the local API
The Hailo Ollama server exposes a REST API on the Pi. After starting the server and pulling a model, submit a prompt to the local endpoint described in Raspberry Pi’s AI software documentation. A local API lets another application on your network use the Pi without moving the model to a cloud service. Open WebUI is optional; Raspberry Pi’s documented WebUI setup uses Docker, so treat it as an additional interface rather than a requirement for inference.
Recommended Free Tools
CPU route or AI HAT+ 2: a practical decision framework
Choose CPU inference when
- You already own a Pi 5 and want the least expensive experiment.
- You are learning how local model serving works.
- Your workload consists of short prompts, simple classification, lightweight summarization or occasional text generation.
- You do not need a graphical interface or a continuously running service.
Choose the AI HAT+ 2 when
- You want the officially documented Hailo acceleration path.
- You need a supported NPU-backed runtime and are willing to manage its package and model ecosystem.
- You plan sustained workloads where additional hardware and active cooling are justified.
- You need local LLM or VLM support within the roughly six-billion-parameter class described by Raspberry Pi’s hardware documentation.
The HAT does not turn every model into a fast model. Compatibility, quantization, prompt length and software versions remain important, and the available evidence does not establish a universal speedup over CPU inference.
What Raspberry Pi’s benchmark actually shows
Raspberry Pi published a benchmark using an 8 GB Raspberry Pi 5, four CPU threads, 1,024 prefill tokens and 256 decode tokens. The results below apply to that exact test setup and should not be generalized to another model, runtime, memory size, operating-system build or thermal condition.
| Model and runtime | Prefill speed | Decode speed | Peak memory | Test conditions |
|---|---|---|---|---|
| Gemma 4 E2B with LiteRT-LM and QAT | 99 tokens/sec | 9 tokens/sec | 1,432 MB | 8 GB Pi 5, four CPU threads, 1,024-token prefill and 256-token decode |
| Gemma 4 E2B with llama.cpp and Q4_0 | 24 tokens/sec | 4 tokens/sec | 4,406 MB | 8 GB Pi 5, four CPU threads, 1,024-token prefill and 256-token decode |
Prefill measures processing of the input context; decode measures generation of new output tokens. Decode speed is usually the figure you notice while reading a response, whereas prefill affects how long the system takes to absorb a long prompt. The different quantization and runtime choices also change memory use, so comparing only one column can mislead.
Rank #3
- ⚡ PoE HAT for Raspberry Pi 5 CM5: PoE HAT F is a Power over Ethernet expansion board for Raspberry Pi 5 and CM5, supporting network connection and power input through one Ethernet cable.
- 🔌 802.3af/at PoE+ Support: This PoE+ HAT supports IEEE 802.3af/at network standard and works with compatible PoE power sourcing equipment for compact wired deployment projects.
- 🧊 Active Cooling Fan and Metal Heatsink: The PoE HAT with cooling fan includes a metal heatsink and high-speed active fan, helping improve heat dissipation and operating stability during long-term use.
- 🔋 5V and 12V Output Headers: Onboard 5V and 12V header outputs provide power options for external peripherals, with up to 25W total output under suitable PoE input and cooling conditions.
- 🧩 40-pin GPIO Stackable Header: Standard 40-pin GPIO stackable header fits Raspberry Pi 5 and CM5 expansion, allowing users to connect compatible HATs and custom project interfaces.
The same publication includes a comparison involving Gemma 3 270M. That is a different model family and workload from the Gemma 4 E2B results above; it cannot be used to rank all models or runtimes generally.
Cooling, power and sustained operation
A short CPU experiment may complete without extra cooling, but sustained inference keeps the processor busy and can raise temperatures enough to reduce clock speeds. Raspberry Pi recommends the Active Cooler and specifically recommends cooling for benchmarks and intensive workloads. Fit the AI HAT+ 2’s included heatsink when using that board.
- Place the Pi where air can circulate; do not enclose it in an unventilated case during long runs.
- Use a reliable power supply suitable for the Pi 5 and attached hardware.
- When comparing models, keep cooling, thread count and workload length consistent.
- If generation slows after several minutes, check temperature and throttling before blaming the model or runtime.
Storage and memory planning
The 32 GB microSD recommendation is a starting point, not a promise that every model will fit. Model files, operating-system updates and temporary downloads compete for the same capacity. Keep several gigabytes free so package operations do not fail unexpectedly.
Memory pressure can appear as slow startup, swapping, failures while loading a model or instability when other services are active. A larger model is not automatically better on a Pi: the extra parameters can make interaction too slow for the intended task. Start with the smallest model that can answer your questions, then move upward only if quality is inadequate and the Pi has enough memory and storage.
Troubleshooting common failures
The model will not download
- Check network connectivity and free storage.
- Confirm the model tag exactly. The guide’s examples are
gemma:2b,phiandtinyllama, but library tags can change. - Remove incomplete downloads only after confirming which local files are safe to delete.
The model loads but responses are extremely slow
- Try a smaller model or shorter prompt.
- Check whether the Pi is swapping or thermally throttling.
- Compare like with like: the same model file, quantization, runtime, thread count and prompt length.
- Do not use the published benchmark as a guarantee for your configuration.
The AI HAT+ 2 route cannot find a model
- Confirm that the Hailo GenAI Model Zoo package is version 5.1.1 as documented for the Pi 5 setup.
- Make sure
hailo-ollamais running before listing or pulling models. - Use a model returned by the Hailo server’s own list, rather than assuming every general Ollama tag is compatible.
The Pi becomes hot during a long run
Install the recommended Active Cooler for the Pi 5, fit the AI HAT+ 2 heatsink when applicable, improve airflow and reduce workload intensity. Cooling is a reliability and sustained-performance measure; it is not a requirement for every brief CPU test.
Privacy and practical limits
With either route, model execution can remain on the Raspberry Pi instead of sending prompts to a hosted service. That is useful for disconnected environments and data you do not want to upload. Local execution does not guarantee correct answers, however. Small models have limited knowledge and reasoning ability, and they can confidently produce errors. Keep a human review step for decisions where accuracy matters.
A Pi 5 is particularly well suited to learning, embedded prototypes, simple local assistants and automation experiments. It is less suitable for high-throughput generation, large-context workloads or models that require more memory than the board and accelerator can provide.
A sensible first session
- Install the latest Raspberry Pi OS on a Pi 5 and ensure at least 32 GB of storage is available.
- Install Ollama using Raspberry Pi’s current project instructions.
- Run
gemma:2b,phiortinyllamawith a short prompt. - Record startup time, approximate response speed, free storage and temperature.
- Only after the CPU path works, decide whether the workload justifies an AI HAT+ 2.
- If you add the HAT, install the documented Hailo GenAI Model Zoo package version 5.1.1, start
hailo-ollama, choose a model from its list and test through its local REST API.
This progression separates software problems from hardware-acceleration problems and avoids buying an add-on before you know what your workload requires.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errors
