A self-hosted AI inference gateway gives your applications one controlled endpoint for model requests, while letting you manage provider credentials, model access, and usage policies centrally. A practical way to learn the deployment flow is LiteLLM’s Docker quickstart; for production, add database-backed spend tracking, shared services for multiple replicas, and network and secret controls.
Choose a deployment shape that fits your stage
For a first deployment, LiteLLM’s Docker quickstart demonstrates the main request path: a gateway listens on port 4000, connects to a model, and issues virtual keys that applications use in place of provider keys. Its sample uses Docker Compose and PostgreSQL-backed management. Review the Compose file, keep its secrets out of source control, and pin a release tag for a reproducible deployment rather than relying on a moving latest tag. The quickstart’s steps and limitations are documented in LiteLLM’s Docker quickstart.
As an Amazon Associate I earn from qualifying purchases.
Free tools Windows power users keep installed
One-click scans. No signup required.
A single-machine Compose setup is useful for learning and smaller deployments, but it is not equivalent to a production multi-instance architecture. The right next step depends on whether you need durable spend-based controls, multiple gateway replicas, and operational ownership of the surrounding services.
| Deployment shape | Best fit | Database and replicas | Operational ownership |
|---|---|---|---|
| Single-machine Docker quickstart | Learning the request path or running a limited deployment on one host. | The documented quickstart uses PostgreSQL; it does not describe a multi-replica topology. | You operate the host, Docker Compose configuration, secrets, network exposure, and upgrades. |
| Production monolithic service | Production deployments that favor a simpler service layout. | Use PostgreSQL for persistent gateway data; add Redis for shared multi-instance behavior when deploying replicas. | You operate or arrange the database, Redis where needed, ingress, secret storage, migrations, and monitoring. |
| Production microservices | Deployments that need gateway traffic to scale independently from management APIs and the UI. | Use the production guide’s shared PostgreSQL and Redis services for the multi-instance setup. | You also operate the separate gateway, management backend, and UI components and their routing. |
LiteLLM’s production guide describes monolithic and microservices layouts, Helm deployment to Kubernetes on EKS, GKE, or AKS, and official Terraform modules for AWS and GCP. Its example production architecture includes a load balancer, PostgreSQL, Redis, and managed secret storage. See LiteLLM’s production deployment guide for the supported deployment paths and topology details.
#1 Best Overall
Start with the Docker quickstart request path
- Prepare the Compose deployment. Follow the official quickstart’s Docker setup. Its sample generates master and salt keys before starting Docker Compose. Treat those as secrets: do not commit them to a repository or expose them in client code.
- Start the gateway and database-backed workflow. The gateway is available on port 4000 in the documented quickstart. Keep the database connection in place if you plan to enforce spend budgets.
- Connect a model. Configure a model the gateway can route to, and keep its provider credential on the server rather than distributing it to applications.
- Issue a virtual key for a caller. Set its permitted model access and applicable usage controls, and associate it with the appropriate user or team when you need spend attribution at those levels.
- Point an application at the gateway. Use the issued virtual key and the gateway’s OpenAI-compatible client interface for the request, instead of placing the provider’s raw key in that application.
- Verify the controls. Confirm that an allowed model request succeeds and a disallowed model request is rejected. If you set a budget, test the database-backed enforcement path with the deployed release rather than checking only that a value appears in configuration.
The exact configuration fields and behavior can vary by release. Check the selected LiteLLM release’s documentation and test its permissions and limits before relying on them in production.
Set up production services for shared state
When the gateway runs as multiple instances, shared services are not interchangeable with local state on each replica. LiteLLM’s production guide assigns distinct roles to PostgreSQL, Redis, and the schema migration job.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →- PostgreSQL stores keys, teams, users, spend logs, and configuration used by the gateway’s management and budget workflows.
- Redis supports rate limiting, router state, and caching shared across instances.
- A migration job applies schema changes during upgrades. In the documented migration-job pattern, proxy instances should not each independently run schema updates.
- A load balancer routes client traffic to stateless gateway replicas, allowing capacity to scale without treating each replica as a separate source of truth.
For Kubernetes deployments, the production guide describes Helm-based deployment and autoscaling options. It also describes metrics endpoints; include them in monitoring for latency, throughput, errors, and resource use. If using tokens-per-second signals, account for the guide’s note that long-running streamed responses are counted when the response completes, not continuously as tokens arrive. These details are in the production deployment documentation.
Give each caller a narrowly scoped credential
Issue a distinct virtual key for each application, user, or team that needs independent access or accounting. Avoid distributing provider credentials or a gateway administrator key to ordinary applications. Configure each virtual key’s allowed models and attach the relevant user or team relationship and limits. LiteLLM documents spend attribution at key, user, and team levels when those identifiers are attached in its virtual keys guide.
Model access and management permissions are different
Do not assume a key has low privilege simply because it was created for an application. LiteLLM evaluates model access at the key, while management-route permissions such as key, user, or team administration depend on the owning user’s role. The documentation warns that a key associated with a proxy administrator can have broad management power; use the documented allowed_routes control where relevant to restrict routes, including for admin-owned keys. Confirm the exact behavior for your release and key ownership setup.
Use a separate identity design when end-user identity matters
Gateway virtual keys are useful for service-level access and usage accounting, but they are not automatically a complete end-user identity system. For AWS environments specifically, Amazon Web Services’ Generative AI inference architecture and best practices on AWS describes identity-based authentication using short-lived credentials and signed requests, and mapping end-user OAuth2/OIDC identities to roles. Treat that as an AWS-specific architecture option, not a universal gateway configuration.
Distinguish token rate limits from spend budgets
TPM and RPM limits govern throughput or request rate. A max_budget is a spend limit over a configured period; setting a budget does not by itself impose TPM or RPM limits. Choose the control that matches the risk: use rate limits to constrain request volume or token throughput, and a budget to constrain recorded spend.
Budgets need a database to be enforced
LiteLLM states, “Budgets require a database.” Its budget checks compare limits with spend read from the database. In DB-less mode, global budget checks fail open, so the proxy can continue serving beyond a configured amount; key and team virtual-key budgets are also unavailable. Do not treat a budget value in a database-less deployment as a hard cap. The implementation detail is covered in the quickstart documentation.
Choose the scope and test the enforcement path
Decide whether a policy belongs to an individual key, a user, a team, or the proxy as a whole, and whether team-level constraints should apply when a team key is used. Set the intended reset period as part of the policy. Verify the result using realistic requests against the exact deployed release: the documentation notes that some routes without token pricing enforce against spend already recorded rather than a reserved cost estimate. A configured monetary budget should not be described as a guaranteed stop at a precise amount without validating the relevant request and pricing behavior.
Recommended Free Tools
Protect the endpoint, provider credentials, and operations
Limit network reachability and encrypt traffic
Put the gateway behind HTTPS/TLS and restrict ingress to clients that need it. For internal company systems, prefer private or internal load balancers where practical. If the endpoint must be public, use IP restrictions where possible and appropriate edge protections. AWS’s access-control guidance puts the rationale plainly: “Network isolation complements API keys and identity-based authentication so that even leaked credentials can’t reach an endpoint directly.” Authentication does not replace network isolation.
Keep secrets server-side and rotate credentials
Store provider credentials and gateway administrator secrets on the server, using environment-backed configuration or managed secret storage appropriate to the deployment. Never embed them in browser code or application bundles. Scope keys to the models and actions they need, and rotate short-lived credentials. AWS also recommends keeping API keys out of client-side code and using private network exposure where possible in its inference security guidance.
Monitor the service as a production dependency
Track latency, throughput, errors, and resource use, and establish an upgrade process that includes database migrations. For Kubernetes, account for the documented metrics and autoscaling behavior, including completion-based counting of long-running streams when using tokens-per-second signals. Pinning a release, validating migrations, and testing key permissions and budgets before rollout make upgrades more predictable.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
