Model Routing

A company that serves AI models does not run one model. It runs several: a small, fast, cheap one, a mid-range one, and a large one that is the most capable and the most expensive to run. Every request that arrives has to be answered by one of them.

Model routing is the decision about which model gets which request. The idea is to match the size of the model to the size of the task. A short summary goes to a small model, and a hard design problem goes to the large one.

Routing can be done in two ways. You can do it yourself by picking a model before you send a task. Or the service can do it for you, automatically and out of sight. This lesson covers both, and it explains why the companies that serve models care about routing as much as their users do. After this lesson you can explain what a router does, and you can choose a suitable model for a task when the choice is yours.

1. A morning of requests, all sent to one model

Take the requests one person might send in a morning:

Suppose every one of them goes to the most capable model the provider runs, the biggest one. That works. Every request gets a good answer.

But three of the five never needed that model. Summarising a thread, rewriting a paragraph, and copying dates out of an invoice are light tasks. The other two are different. A schema is the plan for how an application stores its data, and designing one means weighing many choices. Finding out why a build fails (a build is the step that turns a program's source code into something that runs) means investigating several possible causes. Those two benefit from the strongest model.

Sending all five to the same place treats them as if they were equally hard.

2. What a summary asks of a model

To see why some tasks are light, look at one from start to finish.

What goes in:

Following our call, the delivery slot
has moved from Tuesday to Thursday.
The crew will be on site from nine.
Access is by the rear gate. The front
is closed for resurfacing this month.

What comes back:

Delivery moved to Thursday, 9am.
Use the rear gate.

That is the whole task. The model reads the text, keeps what matters, and drops the rest. There is no plan to make. There are no tools to call. There is no branching, meaning no point where the work could go in different directions depending on what is found. A small model does this well.

3. The same request at two different prices

The summary above takes roughly the same number of tokens whichever model handles it: the same text goes in, and a summary of about the same length comes out. What differs between models is the price of each token.

A worked case with made-up prices. Suppose the request is 800 input tokens and 100 output tokens.

                         input price      output price     cost of this request
                         (per million)    (per million)

most capable model       $15              $75              $0.0195
lighter model            $1               $5               $0.0013

The first cost is 800 x 15 / 1,000,000 plus 100 x 75 / 1,000,000, which is $0.012 + $0.0075. The second is 800 x 1 / 1,000,000 plus 100 x 5 / 1,000,000, which is $0.0008 + $0.0005.

The work is identical, and for a task this light the summary from the lighter model is about as good. The lighter model costs one fifteenth as much. Nothing about the task changed. Only the rate it was billed at changed.

A workplace comparison: a company would not assign its most expensive specialist to copy two lines out of an email. The specialist would do it correctly, and the company would pay specialist rates for clerical work. The comparison is accurate about cost. It stops being accurate about attention: a human specialist is one person who cannot be in two places, while a provider can run many copies of a model at once.

One request costing two cents instead of a tenth of a cent looks unimportant. A business that sends a million such requests a month pays about $19,500 with the large model and about $1,300 with the lighter one.

4. A subscription still counts usage

On a flat monthly subscription to a chat app, there is no bill per request. The money is already spent at the start of the month. Something is still being counted, though. As the lesson on tokens explained, a plan covers a certain amount of usage, called the allowance, and usage is measured in tokens.

Three snapshots of the same allowance, with made-up amounts:

at the start of the month:                  [#...................]   almost nothing used
after one hard question to the top model:   [##############......]   a large share used
the same question to a lighter model:       [####................]   a small share used

Two points follow from this.

First, a single request to the top model can use a real share of the allowance, especially when it includes long documents or a long thinking step.

Second, tokens spent on different models may be counted differently. Some plans give a smaller allowance for the top model, or count its usage more heavily than usage of a lighter model. The rules differ between products and change, so the plan's own description is the place to check.

So routing matters on a subscription too. Light tasks sent to the top model use up the allowance that the heavy tasks will need later in the month.

5. Routing by hand

The simplest kind of routing needs no technology. Look at the task, then pick the model.

task: summarise this thread

  the light one            chosen
  the mid-range one        -
  the most capable one     -

Many chat apps and coding tools have a model picker, a menu that lists the available models. Choosing the light model for the summary, and switching to the most capable one for the schema design, is model routing done by hand, one task at a time.

Some criteria for choosing:

If a light model's answer is not good enough, the task can be sent again to a larger one. The extra cost of that retry is small, because the light model was cheap.

6. Automatic routing

When people say "model routing", they usually mean the automatic kind. Nobody picks anything by hand.

flowchart LR
    A["Your request"] --> R["Router<br/>(usually a model itself)"]
    R --> M1["A cheaper model"]
    R --> M2["Another model"]
    R --> M3["The most capable model"]

Your request first reaches a router. The router is a component that reads the request and decides which model should answer it. The router is usually a model too, a small and fast one, because its only job is to sort.

The request is then passed to one of the available models. Only one of them gets it, and you are not asked which. What decides the outcome is how the request was classified on the way through. Classification means sorting something into a category.

A router adds a small cost of its own, because the sorting step also has to run. The saving comes from the much larger difference in price between the models it chooses from.

7. What the router decides

Your prompt never says which model it wants. The router works that out from the request itself. Written out, its decision looks like this:

request    Summarise this thread for me.
attached   twelve messages, no files

class      text summary
needs      no tools, no long reasoning
fits       the smallest model that can do it
route  ->  the cheaper model

The first two lines are what arrived: the request and what came with it. The next three lines are the router's assessment. It classifies the request as a text summary. It notes what the task needs: no tools and no long reasoning. It concludes that the smallest model that can do the job is enough. The last line is the result: the request goes to the cheaper model.

None of this is decided by you, and none of it is shown to you. It happens before any model starts writing the answer.

8. What you see and what happens behind it

In an app with automatic routing, no model is named to you. At the time of writing, ChatGPT works this way in its default mode: the app decides which of the provider's models handles each message. Depending on the plan, it may also offer a manual picker. Other chat apps and coding tools have similar automatic modes. Products change, so check how the app you use behaves today.

Automatic routing has a consequence for quality. A router can misjudge a request. If it classifies a hard question as a light one, a weaker model answers, and the answer may be worse than expected. When that happens and the app offers a manual choice, selecting a more capable model for that task is the fix.

9. Routing saves the provider money too

Routing is often presented as a way to save the user money. That is half of it. Every answer also costs the provider something to produce.

A larger model has more parameters, so each token needs more computation, more expensive hardware, and more electricity. On a flat subscription, the user pays the same each month whichever model answers, but the provider's cost changes with every request.

So providers prefer not to send everything to the most capable model. Routing light requests to small models lowers their costs and leaves the large model's capacity free for requests that need it. That also keeps waiting times down for everyone.

10. Matching the model to the job

The purpose of model routing is to match the size of the model to the size of the task. When a tool does this automatically, a router classifies each request and picks the model. When a tool does not, the choice is yours to make by hand, using the criteria from section 5.

Interview points

• Model routing means sending each request to a suitable model instead of sending every request to the most capable one.
• The same request uses about the same number of tokens on any model, but the price per token differs widely, so routing light tasks to small models cuts cost without changing the work.
• Light tasks such as summarising, rewriting, and extracting need no planning, no tools, and no branching. Design and debugging tasks benefit from the most capable model.
• On a subscription, usage is still counted against an allowance, a single request to the top model can use a large share of it, and tokens on different models may be weighted differently.
• Manual routing is picking a model per task from a model picker. Automatic routing uses a router, usually a small model, that classifies each request and forwards it.
• With automatic routing the user sees one text box and is not told which model answered.
• A router can misclassify a request, which sends a hard task to a weak model. A manual override or a retry on a larger model addresses this.
• Providers benefit from routing because large models cost more to run per request. Routing lowers their serving cost and keeps capacity on the largest model free.

Summary

• Providers run several models of different size, capability, and cost.
• Sending every request to the largest model wastes money or allowance on tasks that do not need it.
• The same tokens cost less on a lighter model.
• Routing by hand means choosing the model after looking at the task.
• Automatic routing uses a router that classifies the request and picks the model without asking.
• Routing reduces costs for users and for providers.