Free tools Windows power users keep installed
One-click scans. No signup required.
To build an AI agent gateway in Rust, use Axum for the HTTP boundary and Tokio to run each agent turn: accept a request, call a model through a provider adapter, execute authorized tool calls, and stream a stable sequence of events back to the client. Server-Sent Events (SSE) suit one-way progress updates; bounded channels, timeouts, and disconnect cancellation keep the long-lived work manageable.
The example below is a reference architecture, not a universal deployment recipe. Choose the transport, state backend, and tool policy to fit your clients and operating requirements.
As an Amazon Associate I earn from qualifying purchases.
Define the gateway contract before connecting a model
Start by deciding what clients send and what they receive. For example, a gateway might accept turns at POST /v1/agent/stream and return an event stream. That path is a design choice, not a standard. Axum provides routing, request extractors, typed JSON handling, and response construction for this HTTP boundary. Axum project documentation describes it as an HTTP routing and request-handling library focused on ergonomics and modularity.
Define a provider-neutral event model before writing a provider adapter. Events might represent text deltas, tool-call requests, tool results, completion, and structured errors. The client then depends on your gateway contract rather than on the upstream model’s stream format. Keep provider-specific request and response translation at the adapter boundary; a crate that translates between Anthropic Messages and an OpenAI-compatible upstream illustrates this pattern, but does not establish that the protocols have complete feature parity. Adapter example
#1 Best Overall
Separate the HTTP layer, orchestration, streaming, and state
A useful starting layout keeps routing and request validation apart from the agent loop and persistence. Split provider adapters further if you expect to support more than one upstream.
gateway: routes, authentication, request validation, and response construction.agent: turn orchestration, model calls, tool authorization, and limits.streaming: internal event types and conversion to the client transport.state: conversation storage and any cross-instance coordination.
Share reusable resources through application state: an HTTP client, provider configuration, and a state backend as needed. Do not let API keys appear in logs, errors, traces, or debug output. A wrapper that redacts its Debug representation can reduce accidental exposure, but it does not replace careful logging and secret management.
Run each turn as a bounded async pipeline
The request handler can create a bounded Tokio mpsc channel and start an asynchronous task to run the turn. The task calls the model, parses provider stream chunks, converts them into gateway events, and sends those events to the HTTP response. If the model requests a tool, the orchestrator checks authorization, runs the tool, passes its result back to the model, and continues until the turn ends or fails.
Rank #2
- Validate and authenticate. Check the incoming request and establish the client’s identity and permitted capabilities. An upstream API key authenticates the gateway to the model provider; it does not authenticate the gateway’s clients.
- Start the turn. Create a bounded event channel and launch orchestration with the request context, provider adapter, tool policy, and state access it needs.
- Translate model output. Convert provider-specific chunks into the gateway’s internal event types before sending them to the client.
- Authorize and execute tools. Treat each tool call as a permission boundary. Enforce a defined allowlist, argument validation, execution deadline, and concurrency policy rather than treating a model’s request as permission.
- Finish or report failure. Send a completion or structured error event, then release turn resources. Define how partial output and tool failures appear in the client contract.
Tokio’s channel and task primitives do not define a safe policy for tool execution. Specify which tools may run, whether they may run concurrently, and how many calls or resources a turn may consume. For tools with external side effects, authorization and cancellation behavior deserve particular care.
Choose SSE for server-pushed turns; use another transport when directionality requires it
SSE is a straightforward fit when the client submits a turn and the server streams progress or generated content over HTTP. Map the channel receiver into an SSE response, with event names and payloads matching the gateway contract. A client can submit a separate request for another turn.
SSE keeps an HTTP connection open, so configure proxy and load-balancer idle timeouts to accommodate the stream and decide what happens when a client disconnects. If the client needs ongoing messages in both directions on the same connection, evaluate WebSockets or another bidirectional transport instead.
Rank #3
A bounded channel makes a slow reader visible as backpressure: when the buffer fills, sends wait or fail according to the API and policy you choose. This prevents an unbounded queue from silently accumulating events, but it does not by itself stop upstream generation or tool work. Detect a dropped receiver, propagate cancellation to provider requests and child tasks, and set explicit deadlines for the overall request, each tool, and upstream calls.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Pick state and fan-out based on deployment shape
For a single process, in-memory conversation state and Tokio channels may be sufficient. Process-local channels do not provide coordination across instances, and in-memory state is not shared with another process.
For a multi-instance deployment, shared storage can support cross-instance conversation access and event fan-out. The cited implementation uses Redis for conversation state with TTL and pub/sub. Redis adds an operational dependency; adopt it when cross-instance access or coordination is a real requirement, not simply because the gateway uses Tokio.
Keep liveness and readiness distinct. A process can be alive while unable to serve requests because a required state backend is unavailable. Make readiness reflect dependencies the gateway actually needs, and define whether a degraded instance should accept new turns.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Put security, limits, and observability in the design
Before exposing the service beyond a trusted local environment, add client authentication and deliberate authorization. Keep provider credentials in environment-based or managed secret storage rather than source code, and redact secrets from structured logs, error messages, debug formatting, and traces.
- Limit concurrent turns, channel capacity, upstream response time, tool duration, and tool privileges.
- Return structured errors for provider failures, tool failures, timeouts, and client cancellation.
- Ensure cancellation reaches provider requests and child tool tasks so disconnects do not leave orphaned work.
- Use tracing around each turn; track time to first token, stream throughput, tool latency, upstream round trips, and failures.
- Use middleware for concerns such as timeouts, tracing, and authorization. Axum integrates with Tower and tower-http for this service layer.
Check crate versions and feature flags against the Axum release you select. Axum’s official README describes its released branch as 0.8.x, its main branch as work toward 0.9, and lists Rust 1.80 as its MSRV; older tutorial prerequisites should not be assumed to match a current dependency set. Axum README
Interpret published performance figures as setup-specific
A 2026 SitePoint tutorial reports measurements from a 4-core, 8 GB machine using a mock LLM server. It reports gateway-internal P50 time-to-first-byte of about 0.8 ms, P99 latency of about 4.5 ms at 1,000 concurrent connections, a maximum of about 12,000 sustained SSE connections, and about 18 MB of memory at 1,000 connections. The tutorial says internal timings exclude end-to-end network hops and notes that hardware, OS, and kernel tuning affect results. These are that tutorial’s reported measurements, not independently reproduced benchmarks or general performance guarantees. Reproduce them on target hardware before using them for capacity planning or comparisons. SitePoint tutorial
When a custom gateway is not the right scope
A custom Axum service gives you control over the request contract and orchestration, but it also leaves authentication, authorization, rate limits, TLS, observability, and protocol support to your design. If your needs include broader provider routing or MCP and A2A-related features, compare the required capabilities with an existing gateway’s documented scope. The agentgateway 1.6.x documentation covers those areas and distinguishes native, translated, estimated, provider-dependent, and unavailable support in its matrix; “OpenAI-compatible” alone is not proof of feature parity. agentgateway documentation
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →
