OpenZiti LLM Gateway
What OpenZiti LLM Gateway is
OpenZiti LLM Gateway is a free, open-source API proxy for routing requests to OpenAI, Anthropic, and other OpenAI-compatible backends. Clients connect through an OpenAI-compatible API, while the gateway translates requests for Anthropic’s Messages API. Optional semantic routing can choose a model using keyword heuristics, embedding similarity, and an LLM classifier when no model is specified. Chat completions support Server-Sent Events streaming. Weighted round-robin routing, health checks, and automatic failover distribute requests across inference endpoints. Virtual API keys can be limited to models with glob patterns. Optional Prometheus metrics report requests, latency, tokens, routing decisions, provider errors, and endpoint health; tracing can record request metadata and truncated content. Deployment is a single Go binary configured by one YAML file, with no database, message queue, or sidecar. It supports Linux, macOS, Windows, APIs, and self-hosting under Apache 2.0.
Who it suits
It suits teams needing a self-hosted gateway for multiple language-model backends, routing, failover, usage visibility, and virtual credentials. It is also aimed at connectivity across NAT or network boundaries.
What it does well
- Free Apache 2.0 open source
- OpenAI-compatible client API
- Optional semantic routing
- Automatic failover and health checks
- Single-binary self-hosted deployment
What to know first
- Health and metrics endpoints remain unauthenticated
- Tracing can log truncated message content
- Semantic routing is optional
Getting started
- Open the OpenZiti LLM Gateway GitHub repository
- Run the single Go binary
- Create its YAML configuration file
- Connect OpenAI-compatible clients to the gateway API
- Configure providers such as OpenAI, Anthropic, Ollama, vLLM, llama-server, or SGLang
Verdict
OpenZiti LLM Gateway provides routing, translation, failover, metrics, and virtual keys in a self-hosted proxy. Its single-binary design and open-source license suit teams that can operate their own gateway.
Questions
Is OpenZiti LLM Gateway free?
Yes. It is free and open source under the Apache 2.0 license.
Which providers can it route to?
Documented backends include OpenAI, Anthropic, Ollama, vLLM, llama-server, SGLang, and other OpenAI-compatible services.
Does it support Anthropic APIs?
Yes. It translates between the OpenAI request format and Anthropic’s Messages API.
Can it fail over between inference endpoints?
Yes. Weighted round-robin routing, health checks, and automatic failover are supported.
What platforms are supported?
The listed platforms are API, Linux, macOS, Windows, and self-hosted environments.
Facts from OpenZiti LLM Gateway's own pages
- Anthropic support
- The gateway translates between OpenAI format and Anthropic’s Messages API. · github.com, 9 Oct 2026
- Client authentication
- Optional virtual API keys can be restricted to models using glob patterns; the health and metrics endpoints remain unauthenticated. · github.com, 10 Oct 2026
- Deployment
- The project describes deployment as one Go binary and one YAML file, with no database, message queue, or sidecar required. · github.com, 10 Oct 2026
- Headquarters
- Charlotte, North Carolina, United States · github.com, 28 Sept 2026
- Integrations
- Documented backends include OpenAI, Anthropic, Ollama, vLLM, llama-server, and SGLang; clients use an OpenAI-compatible API. · github.com, 10 Oct 2026
- Intended users
- The README describes routing across local inference servers and remote providers, including GPU servers behind NAT and clients using OpenAI-compatible libraries. · github.com, 10 Oct 2026
- License
- The repository identifies the project as Apache 2.0 licensed. · github.com, 10 Oct 2026
- Load balancing
- It supports weighted round-robin across inference endpoints, health checks, and failover. · github.com, 10 Oct 2026
- Metrics
- When enabled, its Prometheus endpoint reports request, latency, token, routing, provider-error, and endpoint-health metrics. · github.com, 10 Oct 2026
- Network security
- The project describes using zrok over OpenZiti to connect gateway and backend across NAT or network boundaries without firewall rules or port forwarding. · github.com, 9 Oct 2026
- Observability
- When enabled, Prometheus-exported metrics include request counts, latency, token counts, routing decisions, provider errors, and endpoint health. · github.com, 9 Oct 2026
- Platforms
- The README says prebuilt binaries are available for Linux, macOS, and Windows, and also documents installation with Go or from source. · github.com, 10 Oct 2026
- Provider integrations
- The project documents routing to OpenAI, Anthropic, Ollama, vLLM, llama-server, SGLang, and other OpenAI-compatible backends. · github.com, 9 Oct 2026
- Provider translation
- It translates between the OpenAI request format and Anthropic’s Messages API. · github.com, 10 Oct 2026
- Purpose
- It is an OpenAI-compatible API proxy that routes requests to OpenAI, Anthropic, and compatible backends such as Ollama and vLLM. · github.com, 10 Oct 2026
- Routing
- Optional semantic routing uses keyword heuristics, embedding similarity, and an LLM classifier to choose a model when the request omits one. · github.com, 9 Oct 2026
- Semantic routing
- An optional cascade of keyword heuristics, embedding similarity, and an LLM classifier can select a model when the request omits one. · github.com, 10 Oct 2026
- Streaming
- It supports server-sent event streaming for its providers. · github.com, 10 Oct 2026
- Tracing
- Optional tracing logs request metadata and truncated message content, with a configurable maximum content length. · github.com, 9 Oct 2026
- Zero-trust networking
- With zrok over OpenZiti, the gateway can expose itself or reach backends without firewall rules or port forwarding. · github.com, 10 Oct 2026
Where OpenZiti LLM Gateway stands on Freedom251
- LLM gateways#30 of 33
- LLM Gateway Software#31 of 34