Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
LiteLLM lets a homelab expose one OpenAI-compatible endpoint for cloud and local language models. Run a small CLI process or container on port 4000, point existing OpenAI SDK clients at that URL, and keep provider-specific credentials in the gateway. For ordered “waterfall” behavior, assign several deployments to one logical model name so LiteLLM retries the preferred deployment and falls back when it is unavailable. Add Docker Compose and PostgreSQL when you need the Admin UI, virtual keys, spend logs, or budget management; add Redis only when routing state and rate limits must be shared by multiple LiteLLM instances.
What does LiteLLM do in a homelab?
LiteLLM is an open-source library and self-hosted AI gateway. Its proxy presents one OpenAI-format API while translating requests to more than 100 providers, so applications can keep the same client code when you change models or vendors. The gateway also includes retries, fallbacks, access controls, spend tracking, load balancing, guardrails, and logging integrations.
As an Amazon Associate I earn from qualifying purchases.
The LiteLLM product site describes the gateway as MIT-licensed, free to self-host, and capable of running in a fully air-gapped environment. It currently advertises support for “140+ LLM providers and 1,892 models”; that is a vendor-maintained figure viewed on October 26, 2026, not a permanent compatibility guarantee. The documentation uses the more conservative “100+ providers” wording.
For a homelab, the practical result is a stable URL such as http://your-server:4000/v1. Your scripts, chat front ends, or automation tools call that URL with the normal OpenAI client format while LiteLLM selects a cloud deployment, an Ollama-hosted model, or a fallback.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
Which LiteLLM deployment should I choose?
Choose the smallest architecture that provides the controls you actually need. A single configuration-file process is easiest to maintain; Compose with PostgreSQL is the middle ground for identity and accounting; replicated deployments add Redis and orchestration only when availability or scale justifies them.
| Deployment path | What it provides | Data services | Best fit |
|---|---|---|---|
| CLI or one container with YAML | OpenAI-compatible endpoint, provider routing, retries and fallbacks | No database required | Personal or small homelab use |
| Docker Compose quickstart | Admin UI at /ui, virtual keys, persisted model configuration, spend logs and budget-related management |
PostgreSQL | Several users, applications or providers needing central management |
| Helm or Terraform production pattern | Replicated stateless gateway, load balancing, shared rate limits and router state | PostgreSQL, Redis and a secret manager | Multi-instance or highly available operation |
How do I run a basic LiteLLM proxy?
The no-database path needs a YAML model configuration, provider credentials in environment variables, and a listener on port 4000. You can run the CLI directly or mount the same file into a container.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →1. Create the model configuration
Give every deployment a provider-specific model identifier and give fallback deployments the same logical model_name. The shared name is what your client sends; LiteLLM chooses among the deployments behind it.
model_list:
- model_name: homelab-chat
litellm_params:
model: openai/<cloud-model-id>
api_key: os.environ/OPENAI_API_KEY
- model_name: homelab-chat
litellm_params:
model: anthropic/<anthropic-model-id>
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: homelab-chat
litellm_params:
model: ollama/<local-model-name>
api_base: http://<ollama-host>:11434Replace the angle-bracketed identifiers with models available in your accounts or on your local server. Keep the YAML file readable only by the account that runs LiteLLM, because it defines how requests are routed.
2. Export provider credentials
Set only the credentials needed by the deployments you enabled. Environment variables keep keys out of the YAML file and make rotation possible without editing configuration.
export OPENAI_API_KEY='your-key'
export ANTHROPIC_API_KEY='your-key'3. Start the proxy
With the LiteLLM package installed, start the process with the configuration file and port 4000:
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
litellm --config litellm_config.yaml --port 4000For Docker, use the current official LiteLLM image, publish host port 4000 to container port 4000, mount litellm_config.yaml read-only, and pass the provider variables with -e. The container command uses the same --config and --port 4000 arguments.
4. Point an OpenAI client at LiteLLM
Most OpenAI-compatible clients need only a base-URL change. Keep the client’s normal request and model name, but use the logical name from your YAML:
from openai import OpenAI
client = OpenAI(
api_key="local-gateway-key",
base_url="http://your-server:4000/v1",
)
response = client.chat.completions.create(
model="homelab-chat",
messages=[{"role": "user", "content": "Hello"}],
)In a no-database installation, the key you send must match the authentication configuration you selected; virtual keys are not available until a database is connected.
When do I need Docker Compose and PostgreSQL?
Use the Compose quickstart when you need the web administration workflow rather than a static route file. The documented stack starts LiteLLM alongside PostgreSQL, where models, keys, and spend logs are stored.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteStart the quickstart stack
- Download the current
docker-compose.quickstart.ymlfrom LiteLLM’s official quickstart documentation and keep it in a dedicated directory. - Generate a long random value for
LITELLM_MASTER_KEYand another forLITELLM_SALT_KEY. Store both outside source control, such as in a protected environment file or homelab secret store. - Export those variables, then start the stack with
docker compose -f docker-compose.quickstart.yml up -d. - Open the proxy’s
/uipath in a browser, add provider models, create virtual keys, and send a test request from the Playground.
LITELLM_MASTER_KEY is the gateway’s root credential. Anyone who obtains it has full administrative access, so do not paste it into a repository, image, issue, or shell history that is shared. If it leaks, rotate it and restart the services, then replace any clients that used the old value.
LITELLM_SALT_KEY encrypts provider API keys saved through the UI. Preserve it as carefully as the database backup: changing the salt makes existing stored credentials unreadable until you enter them again. A lost or replaced salt therefore creates a credential-recovery outage even when PostgreSQL itself is healthy.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Can LiteLLM enforce budgets without a database?
No. The official quickstart states that global budgets are not enforced without a database, and virtual keys fail when no database is connected. A single container is therefore suitable for endpoint unification and routing, but not for LiteLLM-managed spend caps.
If budgets, teams, or per-application keys matter, run the PostgreSQL-backed Compose path. If you deliberately stay stateless, enforce limits at each upstream provider and monitor usage there; do not assume a YAML-only proxy will stop a global quota.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How does waterfall routing work in LiteLLM?
“Waterfall” is a useful description of ordered fallback, not a distinct LiteLLM feature name confirmed by the documentation. Put multiple deployments behind one logical model name, order the preferred deployment first, and configure retry/fallback behavior so a failed or unavailable deployment hands the request to the next one.
Build an ordered deployment group
- Choose a stable logical name such as
homelab-chat. - Add the primary deployment first, followed by alternatives from another provider, region, or local host.
- Configure retries and fallback rules appropriate to the failure classes you want to recover from. A transient provider error can be retried; an authentication error usually needs credential repair rather than repeated calls.
- Exercise the group by temporarily making the preferred deployment unavailable and confirm that the client still receives a response from the next deployment.
Because every deployment shares the same logical name, application code does not change when the route changes. Keep model capabilities aligned, however: a local fallback may have a smaller context window, different tool support, or different output quality than the preferred cloud model.
Which Router strategy should I use?
LiteLLM Router supports several selection strategies. The right choice depends on whether you value simple distribution, current capacity, latency, cost, or custom policy.
| Strategy | How it selects a deployment | Operational note |
|---|---|---|
simple-shuffle |
Shuffles eligible deployments for straightforward distribution | The routing guide recommends it as the production default for performance |
| Weighted pick | Uses configured weights to send more traffic to selected deployments | Useful for gradual migrations or unequal capacity |
| Rate-limit-aware | Prefers deployments with available request or token capacity | Uses deployment RPM and TPM limits and pre-call checks |
| Latency-based | Chooses according to observed response latency | Usage-based strategies rely on Redis and add latency |
| Least-busy | Favors the deployment carrying less active work | Shared state is important when several gateway instances run |
| Lowest-cost | Selects the deployment with the lowest configured cost | Cost data and model capabilities still need validation |
| Custom | Applies operator-defined selection logic | Use when business or hardware rules do not fit built-in strategies |
Deployment-level RPM (requests per minute) and TPM (tokens per minute) limits let Router reject a deployment before a call would exceed its allowance. That pre-call filtering prevents avoidable provider errors and makes the fallback chain more predictable.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Do I need Redis in a single-instance homelab?
No. A single LiteLLM process can route from its local configuration without Redis. Redis becomes important when you run more than one LiteLLM instance and need rate-limit counters, router state, or caching to be consistent across them.
Usage-based strategies such as latency-aware or least-busy routing also depend on shared state and can add request latency. Start with simple-shuffle on one host, then introduce Redis when replication creates a real coordination requirement.
How should I secure a homelab gateway?
- Keep the master key and salt key in a secret store or protected environment file; never commit them to Git.
- Restrict port 4000 to trusted networks or place it behind an HTTPS reverse proxy when clients connect over an untrusted segment.
- Use virtual keys for applications and people instead of distributing the root credential.
- Back up PostgreSQL and the salt key together. A database dump without the matching salt cannot restore encrypted provider credentials.
- Rotate provider credentials independently and remove deployments that no longer have a valid upstream key.
- Review spend logs and provider-side usage because gateway accounting does not replace upstream billing controls.
What changes when I scale beyond one machine?
The documented production architecture separates a stateless LiteLLM service from its shared data services. An HTTPS load balancer sends traffic to replicated gateway instances; PostgreSQL stores keys, teams, users, spend records, and configuration; Redis carries rate-limit and router state; and a secret manager protects the master and provider keys.
LiteLLM documents Helm deployments for EKS, GKE, and AKS and Terraform modules for AWS and GCP. The reference design can use a monolithic image or a microservices layout that separates gateway, backend, and UI. For a homelab, move to this pattern only after one host and then a Compose stack have become operational bottlenecks.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →What are the common failure modes?
The proxy starts but requests fail authentication
Check which credential mode the installation uses. A no-database setup does not create virtual keys; a Compose setup expects the key issued by the UI, while the master key is reserved for administration.
Fallback never occurs
Verify that deployments use the same logical model_name, that the fallback rule covers the actual error, and that the second provider’s credentials and endpoint are valid. Test the secondary deployment directly before testing the chain.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Requests exceed provider limits
Set accurate RPM and TPM values for each deployment so Router can filter calls before they exceed limits. If several instances share traffic, add Redis so counters are not isolated per process.
Models or keys disappear after a restart
A plain configuration-file process has no UI database to persist changes. Use PostgreSQL-backed Compose for model and key persistence, and confirm that the database volume is retained when containers are recreated.
Recommended Free Tools
Stored provider keys cannot be decrypted
Restore the original LITELLM_SALT_KEY. If it was changed, re-enter every affected provider credential through the UI; the old encrypted values cannot be read with the new salt.
Which homelab setup is the sensible starting point?
Start with one YAML-configured process when your goal is simply to give existing clients one endpoint and a fallback chain. Move to Docker Compose with PostgreSQL as soon as virtual keys, spend records, persisted model settings, or budgets become requirements. Add Redis and replicas only for shared routing state or availability across instances; that is the point where the gateway becomes a distributed system rather than a single home service.
Frequently Asked Questions
Can LiteLLM operate completely offline?
The gateway software can run in a fully air-gapped environment, but cloud-provider deployments still require network access to those providers. An offline installation can serve only models reachable on the local network, such as a model server running beside the proxy.
How do I rotate a leaked LiteLLM master key?
Stop or isolate the proxy, replace the master-key value in the protected environment, restart every LiteLLM instance, and issue new application credentials. Treat the old key as compromised even if logs show no suspicious request.
Free tools Windows power users keep installed
One-click scans. No signup required.
Can one LiteLLM endpoint serve separate teams?
Yes, the Admin UI and database-backed deployment support virtual keys and team or user records. Create separate keys for each application or group so usage and permissions can be audited without sharing the administrative credential.
What should I test before trusting a fallback chain?
Send a normal request through the logical model name, then deliberately disable the preferred deployment and repeat it. Confirm that the response comes from the intended secondary model, that authentication errors are not being retried pointlessly, and that provider-side usage reflects only the calls you expect.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
