Model Routing

Definition

Model routing is the practice of sending each incoming request to the cheapest model that can still answer it well. A small, fast model handles the easy majority of queries; the minority that are genuinely hard are escalated to a large frontier model. Because most real traffic is easy, the blended cost and latency fall sharply while overall answer quality barely moves.

The deciding component is the router itself: a lightweight classifier that reads the request and predicts which model should handle it. That is the whole trick and also the whole risk. The router is not free and not perfect — it has its own accuracy, its own latency, and its own evaluation cost — so a wrong route is not a neutral event. It is either a quality regression (a hard query sent to the weak model) or a cost regression (an easy query sent to the expensive one). Everything interesting about routing is the arithmetic of that trade.

One disambiguation belongs in the first breath, because the two ideas share the word "routing" and are constantly confused. Model routing chooses between separate, independently deployed models, at the level of a whole request. It is different from mixture-of-experts (MoE) routing, where a gating network inside a single model sends each token to a subset of expert sub-networks during that model's forward pass. Model routing spans models you can price, swap and call over an API; MoE happens invisibly inside one set of weights and is trained jointly with them. This page is about the first. The Mixture-of-Experts (MoE) page covers the second, and the section on Key Concepts below returns to why the distinction matters in practice.

How It Works

A router sits in front of two or more models. When a request arrives, the router scores it and picks a destination before any answer-producing model runs. The classifier can be as simple as a keyword or length heuristic, or as involved as a fine-tuned encoder (a small BERT-style model) or an LLM-as-judge prompt that rates difficulty. What matters is that it is cheap and fast relative to the models it is choosing between — if the router costs as much as the frontier model, there is nothing left to save.

The economics are easiest to see on numbers. Suppose the frontier model costs 10 units per request and a small model costs 1 unit — one-tenth the price, a ratio that matches the real gap between a flagship model and a small one on today's price lists. Suppose too that a small model can genuinely answer 70% of your traffic. Send everything to the frontier model and you pay 10 per request. Route perfectly instead — easy to small, hard to large — and you pay:

• 70% of requests at 1 unit = 0.70
• 30% of requests at 10 units = 3.00
• the router's own cost, say 0.05 on every request = 0.05

That is 3.75 units, about 2.67x cheaper than all-frontier, a 62% saving — and, if the router is right, at the same quality, because the 30% that needed the big model still got it. This is the core promise, and it is durable arithmetic: it does not depend on this month's prices, only on the price ratio and the share of easy traffic.

Now make the router realistic. It misroutes. Say it wrongly escalates 10% of easy queries (a false upgrade) and wrongly keeps 10% of hard queries on the small model (a false downgrade). The blended cost rises to 4.11 units — still 2.43x cheaper, a 59% saving — but the two errors do different damage. The false upgrades alone add 0.70 units of pure waste: you paid frontier price for easy questions the small model would have nailed. The false downgrades add almost nothing to the bill and instead show up as 3% of all traffic getting a worse answer than it should have. The bill looks fine; the quality quietly dropped. That asymmetry is the whole reason a router needs its own evaluation and cannot be judged on cost alone.

The published evidence supports the shape of this. In the RouteLLM study (Ong et al., ICLR 2025), routers trained on human preference data "reduce costs by over 2 times without sacrificing response quality" on public benchmarks — a figure the paper's abstract states directly and which its results attribute to sending only the genuinely hard fraction of queries to the strong model. The exact multiple depends on the query mix and the price gap, which is exactly what the arithmetic above predicts.

Types

There are two established strategies, and the difference is when the router commits.

• Predictive routers decide up front. The classifier reads the request and picks a model before any answer is generated, as in the RouteLLM framework. This adds only the router's small latency, but the decision is made blind — the router never sees how the cheap model would have done.

• Cascades decide after trying. The request goes to the cheap model first; a scorer inspects that answer's confidence or quality, and only escalates to a larger model if it falls short. FrugalGPT (Chen et al., 2023) is the canonical cascade. Cascades route on evidence rather than a prediction, so they can be more accurate, but a query that escalates has now paid for both models and waited for both — the worst case is slower and more expensive than going straight to the frontier model.

Most production systems are one of these or a blend: a predictive router for the obvious cases and a cascade for the ambiguous middle. The choice is a latency-versus-accuracy trade, not a matter of taste.

Real-World Applications

• Unified model systems. OpenAI's GPT-5, launched in 2025, is described in its own system card as "a unified system with a smart and fast model that answers most questions, a deeper reasoning model for harder problems, and a real-time router that quickly decides which model to use based on conversation type, complexity, tool needs, and explicit intent." The router is continuously trained on production signals — when users switch models, preference rates, measured correctness — which is model routing operating as a product feature rather than an infrastructure bolt-on. It is the reason a single chat endpoint can feel fast on trivia and deliberate on a proof.

• Open routing frameworks. RouteLLM (from LMSYS / UC Berkeley) ships trained routers that decide between a strong and a weak model, and is the source of the "over 2x cost reduction" result above. Teams drop it in front of their own model pair rather than training a classifier from scratch.

• Commercial routing services. A layer of providers — OpenRouter, Martian, Not Diamond and others — route each request across many third-party models, optimising for cost, latency or quality per call. Here the router is the entire product: the customer sends a prompt and the service decides which vendor's model earns it.

Across all three, the same decision recurs: name the fraction of traffic that truly needs the expensive model, and stop paying for it on the fraction that does not.

Key Concepts

Model routing is not mixture-of-experts. This is worth stating twice because the confusion is so common. MoE is a routing mechanism inside one model: a gating network sends each token to a few of many expert feed-forward blocks, all of which live in the same checkpoint and were trained together, so that a large-parameter model activates only a slice of itself per token. Model routing is a routing mechanism between models: a separate classifier sends each whole request to one of several distinct, independently deployed models with their own prices and endpoints. MoE optimises the compute of a single forward pass; model routing optimises which forward pass, in which model, you pay for at all. A page that mixes them up will make wrong claims about cost, because MoE's savings are already baked into a model's per-token price, whereas routing's savings are yours to capture at deployment time.

The router is a classifier, so all of classification's problems are yours. It has precision and recall on the "needs the big model" label; it can be miscalibrated; and its accuracy is only as good as the traffic it was trained on. A router tuned on last quarter's query mix degrades as that mix drifts — a form of distribution shift — which is why learned routers like GPT-5's are retrained on live signals.

Blended cost is the number to track, not per-call cost. The frontier model's price never changes; what changes is how often you invoke it. Routing is entirely a bet on the escalation rate, and the worked example above is how you size that bet before building anything.

Challenges

• The wrong route is invisible in the wrong metric. Aggregate latency and total spend can both look healthy while a steady 3% of hard queries get downgraded to bad answers. Only a quality evaluation on the routed traffic — not on either model in isolation — catches a false-downgrade problem, and most teams do not run one.

• The router adds latency to every request, including the ones it sends to the fast model. If the router itself takes 200 ms, it has erased part of the reason you wanted the small model. A router must be dramatically cheaper and faster than the cheapest model it routes to, or it is a tax rather than an optimisation.

• Cold start for new models. Add a model to the pool and the router has no data on which queries it handles well. Until it is retrained or evaluated against the newcomer, it either ignores the new model or routes to it blindly.

• Escalation-rate drift silently changes the bill. If your traffic gets harder over time, the same router escalates more often and the blended cost creeps back toward all-frontier — with no code change and no alert, because nothing broke. The saving assumed a query mix that no longer holds.

• Evaluating the router is a recursive cost. To know whether a route was right you often have to run the query on both models and compare, which is exactly the expense routing exists to avoid. Good router evaluation is sampled and offline, not run in the serving path.

Future Trends

The frontier is routers that improve from their own production traffic rather than a fixed training set — the GPT-5 router's continual retraining on user switches and preference rates is an early instance, and it turns routing from a static rule into a system that tracks the query mix as it drifts. A second direction is routing to specialised models rather than just a big-versus-small pair: a code request to a code model, a long-context request to a long-context model, a request needing deliberation to a reasoning model. As the catalog of narrow, cheap, strong models grows, the router's job shifts from "how hard is this?" to "which specialist owns this?".

The countervailing trend is worth naming: OpenAI has said it intends to fold GPT-5's separate fast and thinking models "into a single model" in the near future. If a single model can allocate its own effort per request internally, the external router disappears into the model — which would make request-level routing look, in hindsight, like a stepping stone toward the kind of internal, learned effort-allocation that MoE already does at the token level.

Code Example

The blended-cost calculation is the entire business case for a router, and it is short enough to run before writing any serving code. Costs here are relative units — the small model costs 1, the frontier model costs 10 — so the result depends only on the price ratio and the traffic mix, not on any price that will be stale next month.

# Costs are in relative units: the small model costs 1, the frontier model costs 10.
c_small  = 1.0    # cost per query on the cheap model
c_large  = 10.0   # cost per query on the frontier model (10x pricier)
c_router = 0.05   # the router's own classification cost, charged on EVERY query
p_easy   = 0.70   # share of traffic a small model can actually answer well

baseline = c_large                      # send everything to the frontier model

# A perfect router: easy -> small, hard -> large
perfect = p_easy*c_small + (1-p_easy)*c_large + c_router

# A real router misroutes. fu = easy queries wrongly escalated (cost regression);
# fd = hard queries wrongly kept on the small model (quality regression).
fu, fd = 0.10, 0.10
easy_small = p_easy*(1-fu)*c_small
easy_large = p_easy*fu*c_large
hard_large = (1-p_easy)*(1-fd)*c_large
hard_small = (1-p_easy)*fd*c_small
real = easy_small + easy_large + hard_large + hard_small + c_router
bad_answers = (1-p_easy)*fd     # fraction of ALL traffic that got a downgraded answer

print(f"baseline (all-frontier):     {baseline:.2f}")
print(f"perfect router:              {perfect:.2f}  ->  {baseline/perfect:.2f}x cheaper, {(1-perfect/baseline)*100:.0f}% saved")
print(f"real router (10% each error):{real:.2f}  ->  {baseline/real:.2f}x cheaper, {(1-real/baseline)*100:.0f}% saved")
print(f"  of which false-upgrade cost:{easy_large:.2f} (paid 10x for easy queries)")
print(f"  downgraded (bad) answers:   {bad_answers*100:.0f}% of all traffic")

Running it prints:

baseline (all-frontier):     10.00
perfect router:              3.75  ->  2.67x cheaper, 62% saved
real router (10% each error):4.11  ->  2.43x cheaper, 59% saved
  of which false-upgrade cost:0.70 (paid 10x for easy queries)
  downgraded (bad) answers:   3% of all traffic

Two lessons fall straight out of the numbers. First, the saving is real and large even with a mediocre 10%-error router — routing is forgiving on cost. Second, the damage from errors is lopsided: the false upgrades cost 0.70 units of visible money, while the false downgrades cost almost nothing on the bill and instead hide as 3% worse answers. Change p_easy and the price ratio to your own traffic and this tells you, in one run, whether a router is worth building at all.