There is no single best LLM router, because “routing” names two different jobs. One spreads requests across equivalent deployments or providers for reliability and speed. The other picks a different model for each prompt to trade answer quality against cost. LiteLLM is the best-documented option for the first job and also ships an auto-routing feature for the second. RouteLLM is the clearest open framework for the second job. OpenRouter is the main managed alternative. Portkey and Bifrost round out a five-tool shortlist, but the public evidence on them is thinner.
This article therefore gives you a shortlist and a way to choose, not a ranked podium. Every performance number below is labeled with who produced it and under what conditions, because the most-quoted figures in this space come from the projects themselves.
![Computer Networking Bible: [3 in 1] The Complete Crash Course to Effectively Design, Implement and Manage Networks. Including Sections on Security, Performance and Scalability](https://m.media-amazon.com/images/I/412FG7F+RjL._SL160_.jpg)
As an Amazon Associate I earn from qualifying purchases.
Two problems that both get called “LLM routing”
Before comparing tools, decide which problem you have. A tool that is excellent at one need not be good at the other.
Deployment routing: where does this request go?
You have the same model (or interchangeable models) behind several API keys, regions or providers. The router balances load, respects rate limits, retries on errors, falls back when a deployment fails, and cools down unhealthy endpoints. The quality of the answer does not change by design; availability, throughput and latency do.
#1 Best Overall
Model selection: which model should answer this prompt?
You have models of different strength and price. The router looks at the prompt, or a proxy for its difficulty, and sends easy ones to a cheaper model and hard ones to a stronger one. Here quality can change on every request, so the router has to be evaluated on answer quality, not just uptime.
Most production stacks eventually need both: a gateway that handles keys, retries and failover, and optionally a selection policy on top.
The shortlist at a glance
| Tool | Main job | Where it runs | Strength of public evidence |
|---|---|---|---|
| LiteLLM (Router and Auto Router) | Deployment load balancing, retries and fallbacks; plus tiered model selection | Your infrastructure | Detailed official docs; vendor-published benchmark and case figures |
| RouteLLM (LMSYS) | Serving and evaluating trained routers that choose between a cheaper and a stronger model | Your infrastructure (library or OpenAI-compatible server) | Project README with benchmark-based claims |
| OpenRouter | Single OpenAI-compatible API across providers | Managed service | Vendor-authored comparison page only |
| Portkey | AI gateway | Not established here | Appears only as a comparison point in LiteLLM’s benchmark |
| Bifrost | AI gateway | Not established here | Appears only as a comparison point in LiteLLM’s benchmark |
The last two rows are deliberately sparse. The only public figures this article could tie to Portkey and Bifrost are latency numbers published by a competitor, so their feature sets, pricing and routing strategies are not described here. Read their own documentation before including them in a bake-off.
Recommended Free Tools
The five tools, one by one
1. LiteLLM Router
LiteLLM’s official documentation covers load balancing across deployments, retries and fallbacks, cooldowns and timeouts. It documents several selection strategies:
- Weighted / simple shuffle: the docs recommend simple-shuffle for production performance.
- Rate-limit-aware (usage-based): routes with awareness of token and request limits. The docs warn that usage tracking can add latency because it relies on Redis operations.
- Least-busy: favors the deployment with the fewest in-flight calls.
- Latency-based: uses observed response times over a configurable averaging window. A buffer setting widens the pool of eligible deployments so traffic does not pile onto the single fastest endpoint.
- Cost-based: prefers cheaper eligible deployments.
The practical takeaway: the “smartest” strategy is not automatically the best one. The cheapest-overhead strategy (simple shuffle) is the one the maintainers point to for production, and the stateful ones cost you a dependency and some latency.
Auto Router. LiteLLM’s separate Auto Router documentation covers per-request model selection. It describes tier classification, with a choice of heuristic, LLM, JEV, keyword or custom classifiers, plus features such as context escalation and session pinning (keeping a conversation on one model for consistency). The page reports three kinds of results, all publisher-provided for stated configurations:
- “74.5% cheaper at 87.3% of frontier quality” on RouterArena, across 8,399 graded queries.
- “51.1% saved, $12,249 over four months” in a case involving 272,876 production requests and 450+ users.
- Other named test outcomes on the same page.
The RouterArena study date is not stated on the captured page. Treat these as indications of what the feature can do under the configurations described, not as promised savings for your traffic.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstall2. RouteLLM
The LMSYS project describes RouteLLM as “a framework for serving and evaluating LLM routers.” It works as a drop-in replacement for the OpenAI client or as an OpenAI-compatible server, and it ships trained routers that decide between a cheaper, simpler model and a stronger one. A cost threshold controls the quality/cost trade-off, and the README says to calibrate it for your actual query distribution. That caveat matters: a threshold tuned on someone else’s benchmark prompts says little about yours.
The maintainers state: “Trained routers are provided out of the box, which we have shown to reduce costs by up to 85% while maintaining 95% GPT-4 performance on widely-used benchmarks like MT Bench.” That is a project-reported result for its evaluated setup (a strong/weak model pair on named benchmarks), not a general guarantee. The README does not give a year for the claim, so check the original paper for its scope and date before quoting it.
RouteLLM is narrower than a gateway. It decides which model to use; it is not a full key-management, budgeting and failover layer.
3. OpenRouter
OpenRouter offers one OpenAI-compatible API across many providers as a managed service. Its comparison with LiteLLM (published June 19, 2026, updated September 24, 2026) is vendor-authored, so its framing is OpenRouter’s own. According to that page, both products give a single OpenAI-compatible interface; OpenRouter is managed, while LiteLLM runs inside your infrastructure. The page references a 5.5% platform fee. Confirm current pricing on OpenRouter before budgeting, since the fee is OpenRouter’s statement and may change.
Free tools Windows power users keep installed
One-click scans. No signup required.
4. Portkey
Portkey is the second gateway in LiteLLM’s published latency comparison, at 2.29 ms p99 added latency under LiteLLM’s test conditions (see the latency section). Nothing more about Portkey’s routing features, hosting options or pricing is established here, so evaluate it from its own documentation and your own measurements.
5. Bifrost
Bifrost appears in the same LiteLLM benchmark at 4.54 ms p99. As with Portkey, treat that as a competitor-run data point rather than a verdict, and verify features and deployment options directly with the project.
How much latency does a gateway add?
Published gateway figures are usually microbenchmarks of the proxy itself, not of end-to-end LLM calls, which take hundreds of milliseconds to many seconds on the provider side. The one public comparison among these tools is from LiteLLM’s home page:
| Gateway | Added latency (p99) | Other figures published |
|---|---|---|
| LiteLLM (Rust) | 0.66 ms | About 22 MB idle memory; 2,800+ requests/second at about 21% CPU |
| Portkey | 2.29 ms | Not stated |
| Bifrost | 4.54 ms | Not stated |
These numbers come from LiteLLM’s own test, which used identical hardware, a deterministic mock upstream and a single client. They are not independently validated. A mock upstream is the right way to isolate gateway overhead, but a single client does not reproduce concurrent production load, real streaming responses, long prompts, or the effect of a Redis-backed strategy. The labeled “LiteLLM (Rust)” result should not be assumed to describe every LiteLLM deployment mode.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Measuring it yourself
- Run each candidate on identical hardware, with a stub upstream that returns a fixed response after a fixed delay.
- Record client-observed latency at p50, p95 and p99, and subtract the stub’s known delay to get gateway-added overhead.
- Repeat at your real concurrency, with your real prompt and response sizes, and with streaming on if you use it (also track time to first token).
- Switch on the features you will actually use: the routing strategy, logging, budgets, caching. Stateful strategies such as LiteLLM’s usage-based routing add Redis work that a bare proxy test skips.
- Finally, test against real providers and compare overhead with provider variance. If provider latency swings by hundreds of milliseconds, a 2 ms difference between gateways will not matter to users.
Does smart model selection really save money?
The strongest independent evidence is cautionary. LLMRouterBench (January 12, 2026) evaluates more than 400,000 instances across 21 datasets and 33 models. Its authors report:
Best Value
- Strong complementarity between models, so routing has real headroom.
- Many routing methods perform similarly under unified evaluation, and some recent methods, including commercial routers, fail to reliably beat a simple baseline.
- The remaining gap to an oracle router is largely due to model-recall failures, meaning the router does not pick the model that could have answered correctly.
- In their performance-cost setting, up to a 4% average-accuracy gain over the best single model, or up to 31.7% lower cost while matching the best single model.
Those are benchmark-specific results. Read them alongside RouteLLM’s “up to 85%” and LiteLLM’s 74.5% and 51.1% figures: the headline savings are real in their setups, but they depend on how large the price gap between models is, how easy your prompts are, and how quality was graded. The independent benchmark suggests a simple baseline deserves a place in your test, and that a fancier router has to prove it beats it on your data.
A practical way to test routing on your own prompts
- Collect a representative sample of real prompts, with sensitive data removed.
- Run all of them through your strongest model and your cheapest acceptable model. Grade the outputs with a method you trust (human review, task-specific checks, or a calibrated judge model).
- Find what share of prompts the cheap model handles acceptably. That share bounds your possible savings.
- Run candidate routers (a RouteLLM router with a tuned threshold, LiteLLM Auto Router tiers, or a simple rule such as prompt length or task type) on the same set.
- Plot quality against cost per router. Keep the one on the frontier, and include “always use the best model” and “always use the cheap model” as reference points.
- Check multi-turn behavior: if the router can switch models mid-conversation, test whether session pinning or context escalation is needed.
Self-hosted or managed?
This is the other main axis of choice. OpenRouter’s own comparison says that self-hosting LiteLLM can keep data on your network but means running PostgreSQL, Redis and Docker; OpenRouter’s managed service avoids that operational load but adds a platform fee and puts a third party in the request path. That is OpenRouter’s framing, and the trade-off it describes is a fair general one:
| Question | Leans self-hosted (LiteLLM, RouteLLM) | Leans managed (OpenRouter) |
|---|---|---|
| Data must stay on your network | Fits better | Check the provider’s data handling first |
| Small team, no ops capacity | You own upgrades, databases and uptime | Fits better |
| Custom routing policy or classifier | Fits better; RouteLLM and LiteLLM’s custom classifier are open to modification | Limited to what the service exposes |
| Cost structure | Infrastructure and staff time | Platform fee (OpenRouter references 5.5%; verify current terms) |
How to choose
The evidence supports comparing any router on seven points:
Quick Recap
- Task type: deployment failover and load balancing, or per-request model selection.
- Routing signals and transparency: can you see and override why a request went where it did?
- Added p50/p95/p99 overhead under traffic that resembles yours, not only vendor microbenchmarks.
- Quality-cost frontier on your own prompts.
- Failure handling: rate limits, retries, cooldowns and session consistency.
- Data location, security and operational ownership.
- Instrumentation, governance and total cost, including fees and the staff time to run it.
Applied to the shortlist:
- You need reliability across keys and providers, and can run infrastructure: start with LiteLLM Router, begin with simple shuffle, and add stateful strategies only if measurements justify the Redis overhead.
- You mainly want to cut spend by sending easy prompts to a cheaper model: test RouteLLM and LiteLLM Auto Router against a simple rule on your own data.
- You want one API and minimal operations: evaluate OpenRouter, weighing its fee and data handling against your requirements.
- Raw gateway overhead is critical: add Portkey and Bifrost to a latency test of your own, since the only public numbers for them were produced by a competitor.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
