TensorZero

TensorZero is a self-hosted, open-source stack that combines an LLM gateway with observability, evaluation, optimization, and experimentation. It is best suited to technically capable teams that want provider routing and SQL-level control of inference data. Its broad feature set costs nothing, but choosing it also means accepting an archived project with no ongoing maintenance.

Overview

TensorZero places a gateway and web UI in front of multiple model providers. A unified API supports Anthropic, AWS Bedrock, Azure, Google AI Studio, OpenAI, OpenRouter, vLLM, xAI, Together AI, and other OpenAI-compatible endpoints. Routing, retries, fallbacks, prompt templates, schemas, and virtual API keys cover the operational work of sending requests across those services.

The gateway writes inference and feedback data to a database you configure. Teams can query that data directly with SQL, using it for observability, evaluation, optimization, and experiment analysis without handing the dataset to a hosted vendor. That control suits organizations with database and infrastructure expertise; it also makes deployment, backups, access control, and upgrades their responsibility.

TensorZero reports sub-millisecond P99 latency overhead under extreme load, a useful target for latency-sensitive services. The project’s status is the more consequential qualification: the official site says it is no longer maintained, and GitHub marks the repository archived and read-only.

Key features

Routing and workflows

Fallback routing and retries help keep requests moving when a provider or model fails. Traffic can be split for A/B tests, and multi-step workflows support applications that need more than one model call. Feedback can be attached to an individual inference or an episode, giving teams a way to connect an outcome with a request sequence rather than treating every call in isolation.

Data and integrations

Because inference and feedback records stay in your own database, SQL users can build analysis around their existing systems. Integrations cover major hosted providers, vLLM, OpenAI-compatible APIs, the OpenAI SDK, OpenTelemetry, and clients for major programming languages. This is a practical fit for a poly-provider architecture, provided your team is willing to operate the surrounding stack.

Deployment and security

The gateway and UI can run with Docker, Docker Compose, Kubernetes and Helm, or a source build. API-key authentication can protect gateway endpoints, while status and health checks remain accessible. TensorZero collects pseudonymous aggregate usage analytics, says those analytics exclude application data and inference inputs or outputs, and provides a setting to disable collection.

Pricing

TensorZero LLMOps platform — free. The plan is 100% self-hosted and open source, with no paid tier described. It includes the gateway, routing, fallbacks, budget controls, virtual API keys, and the self-managed deployment model. The Apache 2.0 license is favorable for organizations that need to run and adapt an open-source component.

This price suits developers, research groups, and small teams that can supply their own compute, database, operations, and support process. There is no hosted convenience layer or published commercial maintenance commitment to offset that work. The absence of a revenue model, together with the archived status, is a material consideration for production buyers.

Platforms

TensorZero is available through an API and web UI, runs on Linux, and is designed for self-hosted deployment. Docker, Docker Compose, Kubernetes with Helm, and source builds provide deployment routes. It is not presented as a conventional desktop application.

Who it's for

Choose TensorZero if you need one API across several LLM providers, want fallbacks and controlled traffic experiments, and have the skills to operate a gateway and database yourself. It is particularly appropriate for teams that require direct SQL access to inference data or want to keep application and observability data within their own infrastructure.

Look elsewhere when vendor-backed updates, a maintained release stream, or a managed service is a requirement. The archived repository means future provider changes, security fixes, and compatibility work cannot be assumed.

Pros and cons

Pros

• Broad gateway scope: routing, retries, fallbacks, schemas, prompt templates, and A/B traffic controls address both reliability and experimentation.
• Provider flexibility: documented integrations span major commercial providers, vLLM, OpenRouter, Together AI, and OpenAI-compatible endpoints.
• Data ownership: inference and feedback records remain in a database you choose and can query with SQL.
• Open-source economics: the self-hosted plan costs nothing and uses the Apache 2.0 license.

Cons

• Archived and unmaintained: the project’s current status creates a serious long-term risk for production dependencies.
• Operational burden: deployment, database administration, security configuration, and upkeep belong to the user.
• Self-hosting only: teams seeking a managed gateway or vendor-operated support model will need another product.

Alternatives

Bifrost is a better fit if you want an Apache 2.0 open-source gateway with self-hosted npx, Docker, Kubernetes, or Go-binary deployment and a freemium path that includes a free trial.

Routerly suits teams seeking a free self-hosted gateway with no markup on API calls under its AGPL-3.0 plan.

YoloRouter is worth considering when a single binary and SQLite out of the box are preferable, with PostgreSQL optional.

Envoy AI Gateway is an alternative for a free, self-managed open-source gateway focused on local CLI and Kubernetes deployment.

Kong Gateway is the stronger choice when you want a commercial platform with a 30-day free trial, no gateway limits during the trial, and 30 days of analytics retention.

SGLang is a free open-source inference framework rather than a like-for-like operations stack.

Eden AI fits teams that prefer a hosted API gateway with no subscription and provider-priced calls plus a 5.5% platform fee on purchased credits.

Helicone is an alternative for teams wanting an open-source observability service with a free Hobby plan offering 10,000 requests.

For a wider comparison, browse LLM Gateway Software.

Verdict

TensorZero is a compelling zero-price toolkit for an engineering team that wants self-hosted provider routing, experimentation, and SQL-controlled inference data under Apache 2.0. The decisive reason to look elsewhere is its archived, unmaintained status: choose it only when your organization can assume responsibility for the stack’s future.