Inference Gateway

Free plan APILinuxMacSelf-hostedWebWindows

What Inference Gateway is

Inference Gateway is an open-source, cloud-native proxy that gives applications one OpenAI-compatible API for hosted and local language-model providers. Its listed providers include OpenAI, Anthropic, Groq, Cohere, Ollama, Ollama Cloud, DeepSeek, Cloudflare, Google, Mistral, MiniMax, Moonshot, NVIDIA, llama.cpp, Z-AI, and ElevenLabs. Applications can switch providers through configuration without redeploying, while model aliases can route to pools of upstream deployments. MCP support connects to multiple servers, discovers tools, and executes calls through the gateway. Agent-to-Agent features support capability discovery, delegation, and streamed results; YAML definitions can generate Go, Rust, or TypeScript projects through a CLI. A Kubernetes Operator manages gateways, agents, MCP servers, and chat orchestrators. Documentation covers OIDC JWT validation, Prometheus metrics, OpenTelemetry, structured JSON logs, and Grafana, Loki, and Jaeger examples. Docker, Kubernetes, desktop, API, and self-hosted deployments are documented. SDKs are available for Python, TypeScript, Go, and Rust. The project uses Apache-2.0.

Who it suits

It suits developers and platform teams that need one API across multiple hosted or local LLM providers. Self-hosting and air-gapped deployment fit organizations requiring deployment control.

What it does well

One OpenAI-compatible API
Provider switching without redeployment
MCP and agent coordination
Kubernetes Operator support
Apache-2.0 license

What to know first

Desktop releases are unsigned
No fallback routing
Approximately 10.8 MB static binary

Limits

Inference Gateway has a free plan under the Apache-2.0 license. Fallback routing is not available, and desktop releases are not signed with Apple Developer or Windows code-signing certificates.

Getting started

Open the Inference Gateway documentation
Choose a Linux, macOS or Windows desktop asset, or use the Docker image
Deploy the gateway with Docker or Kubernetes, including a self-hosted or air-gapped environment
Configure supported LLM providers and model aliases
Set up OIDC authentication if identity-provider JWT validation is needed
Connect MCP servers or define agents in YAML
Use the Python, TypeScript, Go or Rust SDK with the OpenAI-compatible API

Verdict

Inference Gateway centralizes provider access, routing, MCP tools, and agent workflows behind one API. It is free, open source, and deployable through Docker, Kubernetes, or self-hosted environments.

Questions

Is Inference Gateway free?

Yes. It is open source under the Apache-2.0 license and has a free plan.

Which providers are supported?

The listed providers include OpenAI, Anthropic, Groq, Cohere, Ollama, Ollama Cloud, DeepSeek, Cloudflare, Google, Mistral, MiniMax, Moonshot, NVIDIA, llama.cpp, Z-AI and ElevenLabs.

Can it be self-hosted?

Yes. Documentation describes on-premises, cloud and air-gapped self-hosting, with Docker and Kubernetes deployment.

What platforms are available?

Platforms listed are API, Linux, macOS, self-hosted, web and Windows. Desktop assets are provided for Linux, macOS and Windows on amd64 and arm64.

Does it support MCP and agents?

Yes. It discovers and executes tools through connected MCP servers, and supports Agent-to-Agent discovery, delegation and streamed results.

Which SDKs are official?

Official SDKs are available for Python, TypeScript, Go and Rust, with Server-Sent Events streaming support.

Facts from Inference Gateway's own pages

Agent coordination: Its Agent-to-Agent support lets agents discover capabilities, delegate tasks, and stream results within a conversation. · docs.inference-gateway.com, 10 Oct 2026
Agent workflows: The product supports Agent-to-Agent coordination and describes defining agents in YAML with a CLI that generates Go, Rust, or TypeScript projects. · docs.inference-gateway.com, 9 Oct 2026
Authentication: Authentication supports OIDC identity providers, with JWT validation against the issuer’s JWKS. · docs.inference-gateway.com, 10 Oct 2026
Deployment: The getting-started guide documents Docker, Docker Compose examples, and Kubernetes deployment. · docs.inference-gateway.com, 10 Oct 2026
Desktop platforms: The desktop app page lists Linux, macOS, and Windows assets for amd64 and arm64; it also says releases are not signed with Apple Developer or Windows code-signing certificates. · docs.inference-gateway.com, 9 Oct 2026
Footprint: The gateway is described as an approximately 10.8 MB static binary designed to scale horizontally with Kubernetes HPA. · docs.inference-gateway.com, 10 Oct 2026
Kubernetes: A Kubernetes Operator manages gateways, agents, MCP servers, and chat orchestrators declaratively as cluster resources. · docs.inference-gateway.com, 10 Oct 2026
License: The documentation states that Inference Gateway is released under the Apache-2.0 license. · docs.inference-gateway.com, 9 Oct 2026
License and support: The project is released under Apache-2.0, with GitHub Discussions for questions and an issue tracker for bugs and feature requests. · docs.inference-gateway.com, 10 Oct 2026
MCP integration: It can discover tools from connected MCP servers and execute tool calls server-side. · docs.inference-gateway.com, 10 Oct 2026
Observability: Built-in observability includes Prometheus metrics, OTLP tracing, structured JSON logs, and reference Grafana dashboards. · docs.inference-gateway.com, 10 Oct 2026
Privacy: The project states that it has no analytics or telemetry phoning home and can be self-hosted on-premises, in the cloud, or air-gapped. · docs.inference-gateway.com, 10 Oct 2026
Product: Inference Gateway is an open-source, cloud-native proxy that presents supported hosted and local LLM providers through one OpenAI-compatible API. · docs.inference-gateway.com, 9 Oct 2026
Provider coverage: The documentation lists providers including OpenAI, Anthropic, Groq, Cohere, Ollama, DeepSeek, Cloudflare, Google, Mistral, MiniMax, Moonshot, Nvidia, and llama.cpp. · docs.inference-gateway.com, 10 Oct 2026
Provider switching: The gateway lets applications switch providers by changing configuration, without an application redeploy, and can route model aliases to pools of upstream deployments. · docs.inference-gateway.com, 9 Oct 2026
Purpose: Inference Gateway is an open-source, cloud-native proxy that provides one OpenAI-compatible API for multiple hosted and local LLM providers. · docs.inference-gateway.com, 10 Oct 2026
Routing: The gateway supports model routing so a stable model alias can point to a pool of upstream deployments. · docs.inference-gateway.com, 10 Oct 2026
SDKs: Official SDKs are available for Python, TypeScript, Go, and Rust, with streaming support over Server-Sent Events. · docs.inference-gateway.com, 10 Oct 2026