An LLM router is a layer that inspects each incoming prompt and sends it to the model best suited to answer it a frontier model for the hard ones, a cheaper or open-source model for the easy majority. The economic case is blunt: peer-reviewed work from UC Berkeley’s RouteLLM showed routers hitting 85% cost reduction while preserving 95% of GPT-4 quality by sending only the hardest queries to the expensive model. That gap between “what your prompts need” and “what you’re paying for” is the entire reason routing exists and it’s why gateways like OrcaRouter bundle routing across 200+ models behind a single OpenAI-compatible endpoint.

Quick take: Most production traffic is easy simple lookups, short rewrites, classification. Paying frontier prices for all of it is waste. A good router classifies prompt difficulty, sends the easy majority to cheap models, escalates the hard minority, and fails over when a provider goes down. The catch: routing adds latency and complexity, and a bad routing decision is an invisible quality tax until customers complain.

What an LLM router actually does

A naive setup hardwires every request to one model. A router adds a decision step in front of that call: for each prompt it estimates how hard this is, then picks the cheapest model likely to answer it acceptably. The expensive model still handles genuinely difficult work it just stops handling the trivial 70–80% of traffic a smaller model answers just as well.

The savings come from the price spread between tiers. Frontier models like Claude Opus run roughly $5 in / $25 out per million tokens, while efficient models in the same family cost a fraction of that. Route the easy majority down a tier and the bill follows.

How a prompt-aware router decides: score difficulty, route by tier, fail over on error.

How routing decisions get made

Routers differ mainly in how they judge a prompt. Per a 2026 engineering guide on model routing, the common strategies trade speed for accuracy:

Strategy	How it decides	Overhead	Best for
Rule-based	Keyword/regex matching	<1ms	Obvious, high-volume cases
Embedding-based	Vector similarity to known queries	~5ms	Semantic grouping
ML classifier	Trained difficulty estimator	50–100ms	Nuanced cost-quality calls
Cascade	Try cheap model first, escalate if a confidence check fails	Variable	Maximizing cheap-model use

The cascade pattern is often the most cost-effective: answer with the small model, then escalate only when a verification step flags low confidence. RouteLLM’s matrix-factorization router took this further — on MT Bench it reached 95% of GPT-4 performance while calling GPT-4 for only 14% of queries.

How good is routing, really?

This is where independent benchmarking matters, because vendors quote their best-case numbers. RouterArena, the first open platform for comparing routers (published September 2025), evaluated routers across 8,400 queries spanning 9 domains and 44 categories at multiple difficulty levels. Its headline finding is the honest one: no single router is universally optimal there’s a real accuracy-vs-cost trade-off.

On RouterArena, the highest accuracy came from a GPT-5-only router at 71% on the long-context task, but it ranked poorly overall because restricting the pool to one expensive family blew up cost. The platform’s broader lesson: commercial routers don’t necessarily beat open-source ones they often just buy a little accuracy at a lot of cost.

Treat any vendor’s single accuracy or savings figure as a ceiling, not a guarantee. OrcaRouter, for instance, advertises roughly 75.5% RouterArena accuracy and up to ~40% cost reduction plausible and in range, but a vendor claim you should validate on your own traffic before trusting it.

Reported routing savings vary widely with traffic mix and model tiers. The 40% figure is a vendor claim. Source: RouteLLM / LMSYS, Digital Applied, OrcaRouter (vendor-reported).

What you actually save

Savings track your traffic mix, not a marketing slide. The same engineering guide models it cleanly: a 70/30 split (cheap/frontier) cuts roughly 67% of the bill; an 80/20 split toward a very cheap model approaches 79%; a frontier-heavy 30/70 split saves only ~29%. RouteLLM’s benchmark-specific 85% sits at the optimistic end and the authors are explicit that it’s not a universal guarantee.

Failover: the other half of the job

Routing isn’t only about cost it’s about staying up. Provider outages are routine: in December 2025 alone, major LLM providers logged dozens of incidents including multiple 30-minute-plus outages. And adding a single dependency at 99.3% uptime drops a whole app to ~99.25% availability. A router with failover detects errors (429s, 5xxs, timeouts) and retries on a healthy model or provider degraded answer instead of a hard failure.

When routing helps and when it doesn’t

• Helps: high-volume mixed-difficulty traffic; multi-provider reliability needs; teams whose easy-query share is large.
• Skip it: low volume where the savings don’t cover the added latency and ops; safety-critical work where you want one validated model every time.
• The hidden risk: a wrong route is a silent quality regression invisible until complaints land days later. Guard it with a pre-deploy eval gate on representative prompts.

The bottom line

An LLM router is leverage on a real inefficiency: you’re almost certainly overpaying to run easy prompts through frontier models. Done well, routing keeps the hard 15–30% on the best model and pushes the rest down a tier, cutting cost by anywhere from ~30% to ~85% depending on your mix — while failover keeps you online when a provider blinks. Done badly, it trades pennies for silent quality loss. The buyer’s move is to demand independent benchmarks, validate any vendor figure on your own traffic, and gate quality before you ship.

Frequently asked questions

What is an LLM router?

Software that inspects each prompt and sends it to the model best suited to answer it cheap models for easy queries, frontier models for hard ones to cut cost while preserving quality.

How much can an LLM router save?

It depends on your traffic. Studies show ~30% to ~85%; the 85% figure is a benchmark best case, not a guarantee.

How does a router decide which model to use?

Via rule-based matching, embeddings, a trained difficulty classifier, or a cascade that tries a cheap model first and escalates on low confidence.

Does routing hurt quality?

It can. A wrong route is a silent regression. Mitigate it with a pre-deploy evaluation gate on representative prompts.

Is there an independent benchmark for routers?

Yes RouterArena compares routers on accuracy, cost, latency, and robustness, and finds no single router is universally best.