Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
You can build a local AI agent without paying a per-request cloud-model API fee by putting FastAPI in front of Ollama and using Chroma for retrieval. That does not make the system cost-free: you supply and operate the hardware, and you still have to secure, monitor, and maintain the service. The useful distinction is that inference and retrieval can stay on infrastructure you control rather than being sent to a hosted model provider.
How do I build a local AI agent with FastAPI and Ollama?
Build it as a small service with clear boundaries: FastAPI accepts and validates requests, an agent layer decides whether to retrieve information or call a tool, a provider module talks to Ollama, and Chroma supplies relevant document passages when needed. Keeping those jobs separate makes it possible to change a model, retrieval strategy, or API contract without entangling the whole application.
As an Amazon Associate I earn from qualifying purchases.
Free tools Windows power users keep installed
One-click scans. No signup required.
Start with a narrow capability rather than an open-ended autonomous agent. For example, define an endpoint that answers questions about a known collection of documents, with no arbitrary tool execution. Then add only the tools the use case needs, with explicit limits on what they may do and how many times the agent may call them.
1. Define the service contract
Use typed request and response models and make the expected behavior explicit. A request might contain a question and an optional collection identifier; the response can contain an answer and, if your application needs them, references to retrieved source records. Decide how to represent invalid input, unavailable models, timeouts, and internal failures before wiring up inference.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
Set a maximum request size and validate fields at the boundary. Treat the user’s question as data, not as an instruction to rewrite the system prompt or bypass your application’s rules. Return structured errors that clients can handle without exposing stack traces, internal prompts, or secrets.
2. Put Ollama behind a provider module
Ollama’s API introduction documents a local API base URL of http://localhost:11434/api and an OpenAI-compatible base URL of http://localhost:11434/v1. Local requests do not need an API key. Make the base URL configurable rather than embedding it in route handlers, and omit cloud credentials entirely from a local-only deployment. Ollama describes its API as a way to build applications with cloud or local models.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Keep model selection and request behavior in configuration: set a model name in model:tag form, choose whether the service streams responses, and set the keep-alive policy for the workload. The Ollama API reference documents chat and completion generation, embeddings, model pull/create/list/delete operations, running-model inspection, and version reporting. It also documents that streaming endpoints can return a single response when stream:false is supplied.
Record the configured model name and its digest or version in operational logs. A tag alone is not a complete reproducibility strategy if that tag can later refer to a different model build; log the identifying information your deployment obtains from Ollama so that changes in output can be investigated.
3. Add retrieval before generation where it helps
For a document-question-answering agent, retrieve candidate passages from Chroma before asking Ollama to generate an answer. The retrieved passages should be treated as untrusted reference material, not as higher-priority instructions. Keep system instructions, user content, and retrieved text clearly separated in the prompt, and tell the model how to handle material that conflicts with the application’s rules.
Do not add retrieval just because the stack includes a vector database. A small, well-scoped task may work better without it. Retrieval is useful when answers need to draw on a collection that is too large or changeable to include in every prompt, or when the application needs to associate answers with source records.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
4. Bound the agent loop
An agent loop should have a stop condition, a maximum number of tool calls, and a timeout for the whole request as well as for individual model or retrieval operations. Validate tool arguments against a schema before execution. Return a controlled failure if the model proposes an unknown tool or produces arguments that do not validate; do not execute arbitrary generated code or shell commands as a shortcut.
Pass only the information a tool needs, and make side effects explicit. For actions that modify data or affect another system, require appropriate authorization and consider a confirmation step. A local model is still capable of producing incorrect, unsafe, or manipulative output; keeping it on your own machine does not make model-generated decisions trustworthy.
Can I run an AI agent without cloud API costs?
Yes. With Ollama serving a local model and Chroma running locally or on infrastructure you operate, inference and retrieval can avoid per-request cloud-model charges. “Zero-cost” is accurate only in the narrow sense of no cloud inference bill: it does not mean zero hardware, electricity, storage, engineering, or operational cost.
Local inference shifts the cost structure. You own or allocate the computer, memory, and, if used, GPU; you also maintain the model files, service, and data. A hosted service may instead charge for usage or capacity while taking on some infrastructure work. Which approach costs less depends on utilization, equipment already available, the model and workload, and the value of keeping data on infrastructure you control. No universal break-even figure follows from the stack alone.
Privacy is also a deployment property, not a label attached to a model. Local processing can reduce the need to send prompts and documents to an external inference provider, but your own logs, backups, access controls, telemetry, and network exposure still matter. If the application calls external tools or services, those calls can transmit data even when generation itself is local.
How do I add ChromaDB retrieval to Ollama?
Use Chroma to store document embeddings alongside metadata, then retrieve relevant records for a query and pass a carefully selected context to the model. Chroma supports dense, sparse, hybrid, full-text, and regex search, metadata filters, and multimodal retrieval; it can run locally or be self-hosted and is Apache 2.0 licensed. The right search mode depends on the data and query, so test it against representative questions instead of assuming vector similarity alone is enough.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Prepare documents for repeatable ingestion
- Normalize the source. Convert input documents into a consistent text representation while preserving useful structure such as headings and page boundaries where possible.
- Assign stable IDs. Use identifiers that remain consistent across re-ingestion so that updates can replace or reconcile existing records instead of silently accumulating duplicates.
- Retain source metadata. Store the source identifier and relevant attributes, such as document type or access scope, so results can be filtered and traced back to their origin.
- Choose and configure embeddings. Select an embedding model deliberately, and keep its identity in configuration. The model used to embed documents must be compatible with the one used for query embeddings.
- Make chunking adjustable. Configure chunk size and overlap for the material and retrieval task. There is no single chunk size established for every dataset; evaluate whether passages preserve enough context without returning oversized or repetitive material.
Test retrieval before judging answer quality
Build a small evaluation set of realistic questions with expected source documents or passages. Check whether the relevant record appears in the returned results, whether metadata filters include and exclude the intended records, and whether duplicate or stale records surface. If retrieval fails, a fluent model answer can conceal the problem by guessing from its general training rather than using the intended source.
Apply authorization filters before retrieved content reaches the model. A prompt telling the model not to disclose a document is not a substitute for preventing unauthorized documents from entering the context in the first place. Keep enough metadata to audit the sources used for an answer, but avoid logging full sensitive content unnecessarily.
What hardware do I need for a 7B local model?
Docker’s 2026 local-model guidance gives a typical planning range of 8–16 GB of RAM for a 7B model. Treat that as an estimate, not a minimum guaranteed to work or a promise of a particular response speed: quantization, context length, concurrency, the runtime, and other processes all change memory use. A GPU can dramatically improve speed, but the cited guidance does not establish one universal GPU requirement.
Plan for the full service, not just the model file. FastAPI, Chroma, the operating system, and concurrent requests also consume resources. Larger contexts and simultaneous users can increase memory pressure; loading multiple models or keeping models resident can change the footprint again. Measure the actual model, settings, and traffic pattern on the machine you intend to deploy.
Capability matters alongside hardware. Docker notes that local models often have smaller context windows than hosted alternatives and that tool-calling support must be checked for each model. Before building an agent around a particular model, verify that it supports the required context size and tool behavior in your chosen runtime; a machine with enough memory cannot supply a capability the model lacks.
How does a local stack compare with a hosted model service?
A local deployment gives you more control over where inference happens and how retrieval data is stored, but it also makes you responsible for operating the stack. Hosted services can reduce infrastructure work, while introducing provider dependence and potentially usage-based charges. The practical trade-offs vary by provider, model, hardware, and workload; the table separates what this stack’s documentation establishes from quantities it does not specify.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
| Decision factor | Local FastAPI, Ollama, and Chroma | Hosted model service |
|---|---|---|
| Inference network path | Ollama documents a local server endpoint; local API requests do not need an API key (Ollama API introduction). | Provider-dependent; not stated in the cited stack documentation. |
| Recurring inference charge | No per-request cloud-model charge when inference is run locally; hardware and operating costs remain. | Provider- and plan-dependent; not stated in the cited stack documentation. |
| Hardware responsibility | You supply and operate sufficient local resources; Docker gives 8–16 GB RAM as a typical planning range for a 7B model (2026 guidance). | Provider-dependent; not stated in the cited stack documentation. |
| Latency and throughput | Depends on the local machine, model, context, and workload; no comparable benchmark is stated. | Depends on the provider and workload; no comparable benchmark is stated. |
| Context length and tool calling | Check the exact model: Docker notes local models often have smaller context windows and that tool-calling support varies by model. | Provider- and model-dependent; not stated in the cited stack documentation. |
| Retrieval control | Chroma can run locally or be self-hosted and supports multiple search modes and metadata filters. | Depends on the retrieval system selected; not stated in the cited stack documentation. |
| Operations | You handle deployment, security, updates, logs, metrics, and backups. | Provider-dependent; the cited stack documentation does not specify the division of operational responsibility. |
What must change before a prototype is production-ready?
A development server and an unauthenticated local endpoint are conveniences for iteration, not a production security boundary. FastAPI frames deployment as making an application available to users through a server with performance and stability, and documents both self-managed and cloud strategies. Treat the move to production as a change in responsibility: your service must handle failures, protect data, and remain observable under real use.
Protect access and limit resource use
- Authentication and authorization: require identity checks at the service boundary and restrict access to collections and tools according to the caller’s permissions.
- Rate and size limits: limit request frequency, input size, retrieved context, and tool-call count to reduce abuse and runaway workloads.
- Cancellation and timeouts: propagate client cancellation where supported and set explicit deadlines for model, retrieval, and whole-request work. A timed-out request should not leave avoidable work running indefinitely.
- Safe errors: return useful status and error data to clients without exposing internal hostnames, prompts, credentials, or stack traces.
Make failures recoverable and visible
Use bounded retries with backoff only for failures that may be transient. Repeating a non-idempotent tool action can duplicate side effects, so retry policy must account for what the operation does. Expose health and readiness checks that distinguish a running web process from a service whose model or retrieval dependency is actually available.
Use structured logs and metrics to track request outcomes, latency, timeouts, model and retrieval failures, and resource pressure. Include the model name and available digest or version information to make behavior changes diagnosable. Do not log secrets or indiscriminately retain full prompts and retrieved documents; observability should not become a second uncontrolled copy of sensitive data.
Control updates and preserve data
Pin the intended model identity and make model updates deliberate. Pulling or changing a model can change response behavior, so test the updated model against representative requests before promoting it. Ollama’s documented API includes operations for pulling, creating, listing, and deleting models, as well as reporting versions and inspecting running models; operational access to those capabilities should be restricted to the deployment process rather than exposed as a public feature.
Back up Chroma’s persisted data and configuration using a procedure appropriate to the way it is deployed, and periodically verify that backups can be restored. Keep source documents or a reproducible ingestion path if the vector collection must be rebuilt. A backup that has never been tested is not evidence that recovery will work.
How should I expose Ollama safely to another machine?
Keep Ollama bound to a protected interface. Its documented local base URL is for local-server use; do not make the unauthenticated local port directly reachable from an untrusted network. If another machine must reach the model, put an authenticated reverse proxy or private network in front of it and restrict which clients can connect.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Expose the FastAPI service as the controlled application boundary rather than giving users direct access to model-management endpoints. Keep credentials and proxy configuration outside source code, allow only the network paths the service needs, and verify access controls from a separate client before treating a deployment as private. “Local” describes where a process runs; it does not automatically mean that the process is isolated.
Frequently Asked Questions
Can I use a local AI agent when my computer is offline?
Inference and retrieval can work without an internet connection once the required model and application data are available locally. Operations that fetch models, update software, or call external tools still require their respective network connections.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesCan more than one person use a local Ollama agent at the same time?
A shared service can accept requests from multiple clients, but actual capacity depends on the model, machine, and concurrency settings. Load-test the intended workload and enforce queueing or concurrency limits rather than assuming a single-user prototype will scale automatically.
Does running Chroma locally encrypt my documents?
The fact that Chroma can run locally does not establish that stored files are encrypted. Choose and verify storage, filesystem, and backup protections that meet your data requirements.
Can I switch embedding models after building a Chroma collection?
Changing the embedding model can make existing vectors incompatible with new query vectors. Plan to rebuild or re-embed the collection with the new model, then validate retrieval before routing production queries to it.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
