LEARN / MODEL ROUTING

Cut AI costs with model routing

PUBLISHED 2026-08-04 · BY Llorenç Ballester — Chief Burn Officer

Sending every request to the flagship model is operationally simple, in the same way heating a sandwich with an industrial furnace is simple. Model routing separates routine, measurable work from the smaller set of requests that genuinely need the expensive model.

Model selection is a cost-control system

Providers ship models with different capability, latency, context, and pricing. The spread is large enough that picking the model changes a high-volume endpoint’s economics more than trimming a few hundred prompt tokens: GPT-5 Mini is $0.25/MTok input, GPT-5.6 is $5 — 20x on the same vendor, before you even look at DeepSeek or a Flash tier. Exact rates are live data, so compare the models you can actually call on the board, then cost from your measured input and output distributions.

A lower per-token rate does not guarantee a lower cost per successful task. The cheaper model may generate longer answers, fail validation, trigger retries, or produce errors a human repairs. Routing works when the model is cheaper after quality and recovery costs — not merely cheaper in the pricing table.

Route tasks with objective success conditions first

The safest first candidates have bounded outputs and mechanical validation: assigning one label from a fixed taxonomy, extracting fields into JSON Schema, detecting language, normalizing text, ranking a small candidate set, short summaries with required facts, rewriting a search query. Each has a result that code or an eval set can inspect without consulting the model’s feelings about its own performance.

Keep ambiguous architecture calls, unfamiliar debugging, long-horizon planning, sensitive decisions, and difficult tool use on the stronger model until evidence supports moving them. Route by task properties — input length, required tools, schema complexity, domain risk, prior failures, confidence signals — not by account tier. A premium customer can request lowercase conversion; a free user can submit a race condition.

Decide the cutoff with an escalation policy

Start deterministic. Map known endpoints or task labels to a default model, then escalate when a validator detects invalid JSON, missing fields, unsupported tool calls, low confidence, contradictory evidence, excessive latency, or a task outside the cheap model’s evaluated range. The basic path is request → task classification → routed model → validator → accept or escalate.

Set the cutoff from measured false-accept and escalation costs. A strict validator sends more work to the flagship but rarely accepts bad output; a permissive one lowers immediate spend but can ship cheaper wrong answers — a metric finance enjoys until customers discover it. Store the route, validator result, escalation reason, model versions, prompt version, and final outcome. Without those fields the router is an opinion generator attached to a load balancer.

Worked example: routing support-ticket classification

100,000 tickets/month, 1,500 input tokens and 200 output tokens each. All-flagship on GPT-5.6 ($5 in / $30 out — verify on the board) is 150 MTok × $5 + 20 MTok × $30 = ~$1,350/mo.

Now route 80,000 routine tickets to GPT-5 Mini ($0.25 / $2) and send 20,000 straight to the flagship. Cheap usage: 120 MTok × $0.25 + 16 MTok × $2 = ~$62. Direct flagship: 30 MTok × $5 + 4 MTok × $30 = ~$270. If 5% of cheap attempts escalate, that adds 4,000 flagship calls ≈ ~$54. Routed total: ~$386/mo — about 71% off, escalations included. The saving survives the recovery cost; a routing pitch that ignores escalations does not count.

The router is infrastructure — version it and watch it

A router is code that decides money on every request, so treat it like a paid dependency, not a config toggle. Version the prompt per model (the same prompt rarely behaves identically across tiers), pin model versions, and gate changes behind the same evals you use for the models themselves. Alert on the two signals that quietly erase the saving: a rising escalation rate (the cheap model degrading, or task mix drifting) and a rising false-accept rate (validators going soft).

Resist the urge to build a learned classifier router before a deterministic one has proven the taxonomy. Adding an ML router before you know what success means creates another service whose main output is meetings. Start with rules keyed on endpoint and task label; graduate to something cleverer only when the rules demonstrably leave money on the table.

Measure quality before and after routing

Build an eval set from real production inputs — especially the malformed, ambiguous, multilingual, adversarial, and unusually long ones. For classification, track precision, recall, per-class confusion, and the costly error type. For extraction, measure schema validity, field exactness, missing values, and unsupported additions. For summaries, test required-fact retention and fabricated claims. For code, run tests, type checks, linters, and security checks — not another model asked whether the first answer “looks good.”

Report cost per accepted result, cost per correct result, escalation rate, false-accept rate, latency, and manual-correction rate. Evaluate the routed system, not each model alone: cheap model + validator + occasional escalation can beat a single-model policy economically, but only if validators catch the failures that matter. Ship a small traffic slice first, and confirm the per-model token mix in your usage export — the how-to is in reading your AI usage export — or run a sample through the auditor. Expand only when the quality threshold holds.

When routing is not the largest lever

Routing barely moves the bill when almost every request genuinely needs the strongest model, volume is low, or spend is dominated by oversized context, repeated retrieval, tool calls, or long generated output. It is also premature when there is no stable task taxonomy or eval set — a router without those is just latency with opinions.

Check model-level spend and task volume first. If one routine endpoint drives most requests, routing is promising. If cost comes from a handful of long research or coding sessions, reducing context, caching, or limiting agent steps may matter more. Routing is a strong lever because model rates differ — not a universal law requiring every request to pass through an elaborate classifier.

Frequently asked questions