There is no single best small language model for every Raspberry Pi 5. For CPU-only use, Gemma 4 E2B is the clearest current starting point when you have an 8 GB Pi and use the LiteRT-LM build; Raspberry Pi measured 9 decode tokens per second and 1,432 MB peak memory in that configuration. For very light, narrow tasks, Gemma 3 270M uses substantially less memory and reached 22.58 decode tokens per second in the same vendor appendix. If you want an accelerator and supported local LLM workflow rather than experimentation on the CPU, choose the AI HAT+ 2, which adds a Hailo-10H, 8 GB of onboard memory and 40 TOPS INT4 performance. The older AI HAT+ is vision-focused and does not support LLMs.
The right choice depends on your Pi’s RAM, whether your workload is chat, coding, translation or classification, the runtime and quantization you can use, and whether you value the simplest setup or sustained accelerator inference. The figures below are measurements for named builds and settings, not promises for every model, prompt or Raspberry Pi configuration.
As an Amazon Associate I earn from qualifying purchases.
Quick recommendations
| What you need | Best starting model or route | Why it fits |
|---|---|---|
| Small CPU-only experiment on an 8 GB Pi 5 | Gemma 4 E2B with LiteRT-LM | Raspberry Pi’s August 2026 benchmark measured 9 decode tokens/sec with 1,432 MB peak memory under its stated test conditions. |
| Fast, narrow classification or extraction | Gemma 3 270M | Raspberry Pi describes it as suitable for task-specific fine-tuning; its appendix reports 22.58 decode tokens/sec and 680 MB peak memory with LiteRT-LM. |
| Embeddings, retrieval and semantic search | EmbeddingGemma 300M | Raspberry Pi positions it for embeddings, RAG, semantic search and classification rather than ordinary chat. |
| Multilingual generation on the CPU | Gemma 3 1B | Raspberry Pi lists it for multilingual generative tasks while cautioning that larger models can become impractical on a standard Pi. |
| Supported local LLM acceleration | AI HAT+ 2 with a Hailo-compatible model | The add-on provides 8 GB dedicated memory, 40 TOPS INT4 and documented LLM/VLM support for models up to approximately 6 billion parameters. |
| Vision inference without an LLM | AI HAT+ | It is a vision accelerator that uses Pi system RAM; Raspberry Pi says it does not support LLMs. |
These are starting points, not a quality ranking. A smaller model that follows your exact schema can be more useful than a larger conversational model that is faster but unreliable on your data.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →What “best” means on a Pi 5
A Pi 5 model has to fit three constraints at once: available memory, an inference runtime that supports the model’s format, and a response speed appropriate to the task. Parameter count is only a rough signal. Quantization, context length, prompt size, thread count and the runtime’s kernels can change both memory use and latency.
#1 Best Overall
- Hailo-10H AI accelerator delivering 40 TOPS (INT4) inferencing performance.
- Performance for computer vision models comparable to the Raspbery Pi AI HAT+ (26 TOPS).
- Runs generative AI models efficiently using 8GB on-board RAM.
- Fully integrated into Raspbery Pi’s camera software stack.
- Conforms to Raspbery Pi HAT+ specification.
Raspberry Pi’s own tutorial advises caution with models above 5 billion parameters on a standard Pi. That is a warning about the practical ceiling, not a guarantee that every model below 5 billion parameters will be fast, accurate or easy to install. Treat the 5-billion figure as a screening rule, then measure the model you actually intend to deploy.
Separate prefill from decode when judging speed. Prefill is the processing of your prompt and context; decode is the generation of each new token. A model can accept a long prompt quickly but still produce a slow stream, or do the reverse. Peak memory also matters: it determines whether the Pi can keep the operating system, your application and the model resident without swapping.
CPU-only models and the measurements that exist
Gemma 4 E2B: the best general starting point on an 8 GB Pi
Raspberry Pi’s August 11, 2026 benchmark tested Gemma 4 E2B on a Raspberry Pi 5 with 8 GB RAM. With LiteRT-LM using a QAT build, four CPU threads, 1,024 prefill tokens and 256 decode tokens, it reported 99 prefill tokens/sec, 9 decode tokens/sec and 1,432 MB peak memory.
Those numbers make Gemma 4 E2B the most clearly documented general-purpose CPU option in this comparison. Nine generated tokens per second is appropriate for short interactive answers, local assistants and constrained edge applications, but it is not evidence that every prompt or build will sustain that rate. Longer context, different thread settings, thermal conditions and another quantization can change the result.
Gemma 3 270M: the low-memory specialist
Raspberry Pi describes Gemma 3 270M as a model for task-specific fine-tuning, including sentiment analysis and entity extraction. In the same article’s appendix, LiteRT-LM reached 22.58 decode tokens/sec with 680 MB peak memory. The appendix figure is not a full quality evaluation; it shows why a very small model can be attractive when throughput and memory headroom matter more than open-ended knowledge.
Use this class of model for a fixed label set, a short extraction schema or another narrowly defined workflow. Validate it against representative examples before relying on it for decisions. A compact model can be quick while still making systematic errors outside its training or fine-tuning domain.
EmbeddingGemma 300M: for retrieval rather than chat
EmbeddingGemma 300M is aimed at embeddings, retrieval-augmented generation, semantic search and classification. It converts text into vectors so an application can find related documents or examples; it is not the same kind of drop-in conversational generator as Gemma 4 E2B.
Free tools Windows power users keep installed
One-click scans. No signup required.
Choose it when the Pi’s job is to search a local knowledge base, route a request to a known category or supply relevant passages to another model. Measure retrieval quality with your own documents. A fast embedding stage does not compensate for poor chunking, stale source material or an unsuitable similarity threshold.
Gemma 3 1B: a modest multilingual generator
Raspberry Pi lists Gemma 3 1B for multilingual generative tasks. It is a sensible middle option when language coverage matters and a 270M classifier is too narrow. Raspberry Pi’s article does not provide a Pi 5 speed or memory measurement for this model, so do not infer performance from the Gemma 4 E2B or 270M figures. Test the exact runtime, quantization and language mix you plan to use.
Raspberry Pi’s CPU benchmark, side by side
| Model and runtime | Quantization or build | Prefill | Decode | Peak memory | Test conditions |
|---|---|---|---|---|---|
| Gemma 4 E2B with LiteRT-LM | QAT | 99 tokens/sec | 9 tokens/sec | 1,432 MB | Raspberry Pi 5 8 GB; four CPU threads; 1,024 prefill and 256 decode tokens |
| Gemma 4 E2B with llama.cpp | Q4_0 | 24 tokens/sec | 4 tokens/sec | 4,406 MB | The same benchmark article and stated Pi 5 setup |
| Gemma 3 270M with LiteRT-LM | Build not stated in the appendix figure | Not reported | 22.58 tokens/sec | 680 MB | Raspberry Pi article appendix |
The Gemma 4 comparison is a runtime-and-build comparison, not a contest between two different model sizes. LiteRT-LM’s QAT result used less peak memory and produced more decode tokens per second than the cited llama.cpp Q4_0 result under the article’s setup. You cannot transfer that ratio to another model or assume one runtime will always win.
When the AI HAT+ 2 is the better route
Raspberry Pi identifies the AI HAT+ 2 as its Pi 5 add-on for local LLM support. It contains a Hailo-10H accelerator rated at 40 TOPS INT4 and 8 GB of onboard memory. Current Raspberry Pi documentation says compatible models from the Hailo Gen-AI Model Zoo can run through the Hailo Ollama server, with models up to approximately 6 billion parameters fitting the device.
Recommended Free Tools
Rank #2
- HIGH PERFORMANCE: Features 26 TOPS (Trillion Operations Per Second) AI acceleration capability through the Hailo AI Accelerator for advanced machine learning applications
- COMPATIBILITY: Specifically designed for the Raspberry Pi 5, connecting via PCIe interface for optimal data transfer and processing speeds
- COMPACT DESIGN: Measures 65mm x 56.5mm, offering a space-efficient solution while maintaining full functionality as an AI acceleration add-on board
- TEMPERATURE RANGE: Operates reliably in temperatures from 0°C to +50°C (32°F to 122°F), ensuring stable performance in various environments
- SEAMLESS INTEGRATION: Functions as a HAT (Hardware Attached on Top) add-on board, providing plug-and-play compatibility with Raspberry Pi ecosystem
The dedicated memory changes the design problem: the model and accelerator workload do not have to consume the Pi’s system RAM in the same way as a CPU-only deployment. It does not make every model compatible. You still need a model in a supported format, a matching Hailo package and a runtime path that exposes that model.
Models named at the AI HAT+ 2 launch
| Launch-listed model | Stated size | Example use or status |
|---|---|---|
| DeepSeek-R1-Distill | 1.5B | Listed by Raspberry Pi as a launch model. |
| Llama 3.2 | 1B | Listed by Raspberry Pi as a launch model. |
| Qwen2.5-Coder | 1.5B | Demonstrated for a coding task in Raspberry Pi’s launch announcement. |
| Qwen2.5-Instruct | 1.5B | Listed by Raspberry Pi as a launch model. |
| Qwen2 | 1.5B | Demonstrated for questions and translation. |
This January 15, 2026 list is a dated launch snapshot, not a complete current catalog. Check the current Hailo model list and package documentation before selecting a model, because supported entries and software versions can change.
Do not confuse AI HAT+ with AI HAT+ 2
The older AI HAT+ uses Pi system RAM and is described by Raspberry Pi as vision-focused. It does not support LLMs. Buying it for language inference will not provide the documented Hailo LLM path. The AI HAT+ 2 is the model to consider when local language or vision-language inference is the requirement.
CPU versus AI HAT+ 2
| Decision factor | Pi 5 CPU | AI HAT+ 2 |
|---|---|---|
| Memory available to inference | Uses Pi system RAM; the cited test used an 8 GB Pi. | 8 GB onboard memory on the Hailo-10H. |
| Documented acceleration | Four-thread CPU figures are available for selected Gemma builds. | 40 TOPS INT4, with LLM and VLM support. |
| Model selection | Depends on the runtime and model format you can build or install. | Use models supported by the Hailo Gen-AI Model Zoo and Hailo runtime. |
| Setup burden | Fewer hardware parts; runtime compatibility remains your responsibility. | Requires physical installation, Hailo software and a compatible model server. |
| Best fit | Learning, prototyping, narrow local services and low-cost experimentation. | Repeated inference, accelerator-backed services and a supported local LLM stack. |
| Quality expectation | Depends on the chosen model and prompt; speed does not establish quality. | Small edge models still have less knowledge than large cloud models. |
There is no controlled CPU-versus-HAT quality comparison in the available evidence. Select the route for its memory, compatibility and latency characteristics, then evaluate answer quality on the actual prompts or records your application will process.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How to set up an AI HAT+ 2 language model
1. Assemble the Pi for sustained work
Install the AI HAT+ 2 using its supplied mounting hardware and optional heatsink. Raspberry Pi’s physical guidance recommends an Active Cooler on the Pi 5. Cooling is a fitment and sustained-use precaution; the cited documentation does not measure a specific token-speed improvement from the cooler.
2. Use the supported software path
Raspberry Pi’s current AI software documentation uses the Hailo-10H on AI HAT+ 2, compatible models from the Hailo Gen-AI Model Zoo and the Hailo Ollama server. Its installation example names Hailo Gen-AI Model Zoo Debian package 5.1.1 for Raspberry Pi 5. Treat that version as an example tied to the documentation, not a permanent requirement; confirm the package version and installation instructions in the current documentation before you install.
3. Select a model from the live catalog
Start with a model listed for your exact Hailo runtime. The launch announcement’s Qwen2 example is relevant to questions and translation, while Qwen2.5-Coder was shown for coding. A model’s presence in an old launch list does not prove that its current conversion, quantization or server package is unchanged.
4. Expose the model to your application
The Hailo Ollama server provides a REST API. That lets a local application send prompts without embedding accelerator-specific code in every client. Keep the server on a trusted network, require authentication at any reverse proxy you add, and avoid exposing an unauthenticated inference endpoint to the internet.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstall5. Add a web interface only if you need one
Open WebUI is optional. Raspberry Pi’s documentation says it should run in Docker because it is incompatible with Python 3.13 as used on Raspberry Pi OS Trixie. A direct REST client is simpler for an embedded service; a web interface is useful for interactive testing and prompt inspection.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to evaluate a model for your own workload
Start with representative inputs
Build a small test set from the real task: the languages users submit, the document lengths, the labels you need, code conventions or the facts the assistant must retrieve. Include difficult and ambiguous cases, not only clean demonstrations.
Measure the right outputs
- Task quality: accuracy, extraction correctness, translation adequacy or code-test pass rate, depending on the job.
- First-token latency: how long the user waits after sending a prompt.
- Decode speed: generated tokens per second during the response.
- Peak memory: model plus runtime, application and operating-system headroom.
- Stability: behavior during a sustained batch or conversation, not just one prompt.
- Compatibility: whether the exact quantization, tokenizer and context length are supported by your selected runtime.
Keep comparisons fair
Change one variable at a time. Record the Pi RAM tier, operating-system release, runtime version, model build, quantization, thread count, prompt length and output limit. The Gemma benchmark demonstrates why this matters: changing from LiteRT-LM QAT to llama.cpp Q4_0 changed both memory and decode speed while holding the named model and broad Pi setup constant.
Rank #3
- ⚡ PoE HAT for Raspberry Pi 5 CM5: PoE HAT F is a Power over Ethernet expansion board for Raspberry Pi 5 and CM5, supporting network connection and power input through one Ethernet cable.
- 🔌 802.3af/at PoE+ Support: This PoE+ HAT supports IEEE 802.3af/at network standard and works with compatible PoE power sourcing equipment for compact wired deployment projects.
- 🧊 Active Cooling Fan and Metal Heatsink: The PoE HAT with cooling fan includes a metal heatsink and high-speed active fan, helping improve heat dissipation and operating stability during long-term use.
- 🔋 5V and 12V Output Headers: Onboard 5V and 12V header outputs provide power options for external peripherals, with up to 25W total output under suitable PoE input and cooling conditions.
- 🧩 40-pin GPIO Stackable Header: Standard 40-pin GPIO stackable header fits Raspberry Pi 5 and CM5 expansion, allowing users to connect compatible HATs and custom project interfaces.
Practical limits and common mistakes
Assuming parameter count predicts quality
Parameter count does not tell you whether a model understands your domain, follows your output format or handles your language. A 270M model may be excellent after task-specific tuning for entity extraction and unsuitable for open-ended support. Conversely, a 1.5B instruction model may be convenient for a small assistant but still lack the knowledge of a large cloud system.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Running an unsupported format on the HAT
The AI HAT+ 2 is not a universal accelerator for arbitrary model files. Confirm the model’s entry in the current Hailo catalog and the runtime version it requires. If no supported conversion exists, the CPU route may be simpler than trying to force an incompatible model onto the NPU.
Using the AI HAT+ by mistake
Check the product name before purchase and assembly. AI HAT+ and AI HAT+ 2 are different accessories; only the latter is documented here for local LLM workloads.
Ignoring context and memory headroom
Peak memory rises with the model and its active context. Leave room for the operating system, your client, retrieval index and other services. A model that fits in a one-shot benchmark can fail when a production prompt is longer or several requests overlap.
Treating launch examples as a permanent model list
The January 2026 launch models are useful examples, not a promise that the same catalog or package names remain current. Recheck the live Hailo model list and software documentation when you deploy.
Recommendations by project type
Local chat and short assistant replies
Begin with Gemma 4 E2B on an 8 GB Pi using the LiteRT-LM path represented by Raspberry Pi’s benchmark. Keep prompts concise, cap output to what the interface needs and test response quality before adding retrieval or tools. If interaction remains too slow or you need many concurrent requests, evaluate AI HAT+ 2 with a supported small instruction model.
Sentiment, entity extraction and fixed labels
Try Gemma 3 270M first. Its documented use case and low measured memory footprint leave more room for your application. Fine-tune or constrain the output for the exact labels, then evaluate false positives and false negatives on your own data.
Private document search
Use EmbeddingGemma 300M for the vector stage and pair retrieval with a generator only if your application needs natural-language answers. Keep the index local, test retrieval separately from generation and monitor how document updates affect results.
Translation and multilingual prompts
Gemma 3 1B is the CPU-oriented model Raspberry Pi specifically associates with multilingual generation. On the HAT route, Raspberry Pi demonstrated Qwen2 for questions and translation. Compare both on the language pairs, terminology and output length that matter to you; no supplied benchmark establishes a universal winner.
Local coding assistance
Raspberry Pi demonstrated Qwen2.5-Coder on AI HAT+ 2 for a coding task. Use a repository-specific test suite to judge the result rather than relying on fluent-looking code. For CPU-only work, select a model and runtime that fit your memory budget, then keep generated patches small enough to review.
Bottom line
For the simplest CPU experiment, choose Gemma 4 E2B with LiteRT-LM on an 8 GB Raspberry Pi 5 and treat the 9 decode tokens/sec, 1,432 MB peak-memory result as a benchmark reference for that exact setup. Choose Gemma 3 270M when a narrow classifier or extractor benefits from minimal memory. Choose EmbeddingGemma 300M for retrieval, and Gemma 3 1B when multilingual generation is the priority.
For a supported accelerator workflow, use AI HAT+ 2—not the older AI HAT+—and select a model from the current Hailo catalog. Its 8 GB onboard memory and 40 TOPS INT4 specification make it the stronger route for sustained local LLM inference, but compatibility and task quality still have to be verified on your own prompts.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
