Mixture of Experts (MoE)

Model descriptions sometimes contain two parameter counts, for example "40 billion parameters, 10 billion active". The second number looks odd at first. A model has a fixed set of parameters, the numbers that training produced, and it is not obvious how some of them can be inactive.

The explanation is a design called mixture of experts, usually shortened to MoE. A model built this way holds all of its parameters, but for any one input it uses only some of them. This lesson explains what that means, what it gains, and what it does not gain.

After this lesson you can read "total" and "active" parameter counts correctly, explain why a mixture of experts model answers faster than an ordinary model of the same size, and explain why it still needs the same amount of memory.

1. The starting point: a large model uses everything

Large language models are called large because of what is inside them: many layers and a very large number of parameters. An earlier lesson showed that "large" is a count of parameters, often in the billions.

In an ordinary model, every input passes through all of it. Each token you send is processed by every layer, and every parameter in every layer takes part in the calculation. This ordinary design is often called a dense model.

That has a direct cost. Each parameter that takes part means some arithmetic for the computer. A model with twice as many parameters needs about twice as much arithmetic for every token it reads or writes. More arithmetic means slower answers and a higher bill for the computers that run the model.

So there is a tension. More parameters usually make a model better, and more parameters make every answer slower and more expensive.

2. What a mixture of experts model is

A mixture of experts model is a specific kind of large language model. It is not a different category of thing, and it is not a small model. It is still large.

Two statements describe it:

"Activated" means "used in the calculation for this input". A parameter that is not activated stays where it is, unchanged. It does no work this time.

The set of activated parameters is not fixed. One input switches on one subset, and a different input switches on another. Over many different inputs, all parts of the model get used.

3. How the model picks a subset

The usual way to build this is to change what a layer contains. In a dense model, a layer has one large block of parameters. In a mixture of experts model, that block is replaced by several separate blocks side by side. Each block is called an expert.

A small extra piece sits in front of the experts. It is called the router (another name is gating network). For each token, the router looks at the data arriving at the layer and picks a few experts, often one or two out of eight or more. Only the picked experts do their calculation. The others are skipped for this token.

flowchart LR
    A["Data for one token<br/>arrives at the layer"] --> R["Router<br/>picks a few experts"]
    R --> E2["Expert 2<br/>runs"]
    R --> E5["Expert 5<br/>runs"]
    R -. "not picked" .-> E1["Experts 1, 3, 4, 6, 7, 8<br/>loaded but idle"]
    E2 --> O["Results are combined<br/>and passed to the next layer"]
    E5 --> O

The diagram shows one layer. Data for one token arrives on the left. The router picks experts 2 and 5. Those two run, and their results are combined and passed on. The other six experts are present in the computer's memory but do nothing for this token. At the next layer there is another router and another set of experts, and the pick is made again.

Three points prevent common misunderstandings:

• The data still passes through every layer. Layers are not skipped. Inside each layer, only some of the parameters are used.
• The choice is made per token and per layer. It is not made once for the whole question.
• Nobody assigns topics to the experts. The name suggests a law expert and a cooking expert, but there is no such list. During training, the router and the experts are adjusted together, and the model works out by itself how to divide the work. The resulting division is often hard for a person to describe.

An everyday picture: a hospital employs many specialists, and all of them are on the payroll and in the building. A patient is sent by the front desk to two of them, not to all. The front desk is the router, the specialists are the experts, and the patient is the token. The picture stops being accurate at the third point above: hospital specialists have named fields chosen by people, while a model's experts do not.

4. Why this makes answers faster

Running a finished model to get an answer is called inference. It is the prediction part of a model's life: text goes in, and the model predicts what comes next. The next lesson covers the term in full.

In a mixture of experts model, the data for a token does not have to travel through all the parameters. It travels through a subset. Fewer parameters take part, so there is less arithmetic per token, and the answer arrives sooner. Less arithmetic also means lower running costs for whoever operates the model.

A worked case with made-up numbers:

Dense model
  total parameters:            40 billion
  used for each token:         40 billion

Mixture of experts model
  total parameters:            40 billion
  used for each token:         10 billion   (the "active" parameters)

Both models hold 40 billion parameters. The dense model does arithmetic with all 40 billion for every token. The mixture of experts model does arithmetic with about 10 billion for every token, so each token needs roughly a quarter of the work.

This is where the two counts from the opening come from. Total parameters is everything the model holds. Active parameters is the amount used for one token. The total says how much the model can store. The active count says how much work each token costs.

5. What it does not give you: lower hardware needs

A tempting conclusion goes like this: less of the model runs, so a weaker machine should be enough. That conclusion is wrong.

The reason is that nobody knows in advance which experts the next token will need. The router decides at the moment the token arrives, and the next token may need different experts. So every expert must already be sitting in memory, ready to run.

An earlier lesson explained that all of a model's parameters have to be loaded into a computer's memory before it can produce anything. A mixture of experts model does not change that rule.

Continue the worked case. Suppose each parameter takes 2 bytes of memory, a common size.

Dense model:               40 billion x 2 bytes = 80 GB of memory
Mixture of experts model:  40 billion x 2 bytes = 80 GB of memory

The memory needed is the same, because memory depends on the total count, not on the active count.

6. Memory is the hard part

Memory is one of the most difficult parts of running a large model, and the parameters are what demand it. The fast memory on the chips that run models is limited and expensive. A model that does not fit cannot run on that machine at all, however quick its arithmetic would be.

So a mixture of experts design helps with one problem and leaves the other untouched:

The whole model is loaded, and only a part of it runs for each token.

This also tells you how to read a model description. If you want to know whether a model fits on a machine, look at the total parameter count. If you want to estimate how fast it answers, the active count is the better guide.

7. Why many current models use this design

A mixture of experts model keeps the benefit of size and reduces its running cost:

• It is still a large model, with all its layers and parameters, so it has the capacity of a large model.
• A request travels through only a subset of those parameters.
• That makes inference quicker.

This is why the design appears in many large language models at the time of writing. A model builder can add more parameters, and so more capacity, without making every token proportionally slower.

There are costs on the builder's side too. The router has to learn to spread work across the experts, which makes training more complicated. And the memory requirement stays as high as the total count demands, so the hardware bill for memory does not shrink.

Interview points

• A mixture of experts model is a large language model in which only a subset of the parameters is used for each input. Nothing is removed from the model.
• Inside a layer, the single block of parameters is replaced by several experts, and a router picks a few of them for each token. The choice is repeated at every such layer.
• Which experts run depends on the input, so different inputs use different parts of the model.
• The experts are not assigned human topics. The division of work is learned during training together with the router.
• Fewer parameters take part per token, so there is less computation, and inference is faster and cheaper than in a dense model with the same total size.
• All parameters must still be loaded into memory, because any expert may be needed for the next token. Memory needs follow the total parameter count.
• Total parameters describe capacity and memory needs. Active parameters describe the computation per token.
• The design does not lower the memory requirement, so it does not allow a machine with less memory. The gain is in speed and running cost only.

Summary

• A mixture of experts model is still a large model with all its layers and parameters.
• For each input, only a subset of the parameters is activated, and the subset depends on the input.
• A router inside the model picks which experts run for each token.
• Using a subset makes inference quicker.
• Every parameter still has to be loaded into memory, whether it is activated or not.
• Memory follows total parameters. Speed follows active parameters.